wetterdienst: Long runtime and wrong number of observations

Describe the bug I experience very long execution times for the following script and get a wrong number of observations:

from wetterdienst.dwd import observations as obs

sites = obs.DWDObservationSites(
    parameter_set=obs.DWDObservationParameterSet.TEMPERATURE_AIR,
    resolution=obs.DWDObservationResolution.HOURLY,
    period=obs.DWDObservationPeriod.RECENT,
    start_date="2020-06-09 12:00:00",
    end_date="2020-06-09 12:00:00",
)
df = sites.all()
ids, lat, lon = map(np.array, [df.STATION_ID, df.LAT, df.LON])
observations = obs.DWDObservationData(
    station_ids=ids,
    parameters=obs.DWDObservationParameter.HOURLY.TEMPERATURE_AIR_200,
    resolution=obs.DWDObservationResolution.HOURLY,
    start_date="2020-06-09 12:00:00",
    end_date="2020-06-09 12:00:00",
)
temp = np.array(observations.collect_safe().VALUE, dtype=float)
head = "id, lat, lon, temp"
np.savetxt("temp_obs.txt", np.array([ids, lat, lon, temp]).T, header=head)

After ~15min I get an Error, because there are 496 stations but only 494 temp values, so numpy can’t save this to a text file.

Expected behavior When truncating the station array, it works fine:

ids, lat, lon = ids[:10], lat[:10], lon[:10]

Desktop (please complete the following information):

OS: Linux
Python-Version 3.6

Am I doing something wrong?

Cheers, Sebastian

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 18 (11 by maintainers)

Most upvoted comments

This is the example in its current state: https://geostat-framework.readthedocs.io/projects/gstools/en/latlon_support/examples/08_geo_coordinates/01_dwd_krige.html#sphx-glr-examples-08-geo-coordinates-01-dwd-krige-py

When there is an update within the next week I can still modify the downloading routines.

Current form of the downloading routine (for the record):

def get_dwd_temperature():
    """Get air temperature from german weather stations from 9.6.20 12:00."""
    from wetterdienst.dwd import observations as obs  # version 0.10.1

    sites = obs.DWDObservationSites(
        parameter_set=obs.DWDObservationParameterSet.TEMPERATURE_AIR,
        resolution=obs.DWDObservationResolution.HOURLY,
        period=obs.DWDObservationPeriod.RECENT,
        start_date="2020-06-09 12:00:00",
        end_date="2020-06-09 12:00:00",
    )
    df0 = sites.all()
    ids, lat, lon = map(np.array, [df0.STATION_ID, df0.LAT, df0.LON])
    observations = obs.DWDObservationData(
        station_ids=ids,
        parameters=obs.DWDObservationParameter.HOURLY.TEMPERATURE_AIR_200,
        resolution=obs.DWDObservationResolution.HOURLY,
        start_date="2020-06-09 12:00:00",
        end_date="2020-06-09 12:00:00",
    )
    df1 = observations.collect_safe()
    temp, ids1 = map(np.array, [df1.VALUE, df1.STATION_ID])
    select = np.isfinite(temp)  # care about missing values
    sorter = np.argsort(ids)  # care about missing stations
    sort = sorter[np.searchsorted(ids, ids1[select], sorter=np.argsort(ids))]
    ids, lat, lon, temp = ids[sort], lat[sort], lon[sort], temp[select]
    head = "id, lat, lon, temp"  # add a header to the file
    np.savetxt("temp_obs.txt", np.array([ids, lat, lon, temp]).T, header=head)

MuellerSeb on Nov 27, 2020

Dear @MuellerSeb ,

I’m still working on another issue atm but will try to make your example work and create a next release until sunday.

Meanwhile as soon as your release is done we would like to also include the interpolation scheme into our example here [1]. BTW for the purpose of speed you may as well want to filter for a state e.g. Saxony (Sachsen) this will cut the stations down by a lot but will still be a meaningful result!

sites = obs.DWDObservationSites(
    parameter_set=obs.DWDObservationParameterSet.TEMPERATURE_AIR,
    resolution=obs.DWDObservationResolution.HOURLY,
    period=obs.DWDObservationPeriod.RECENT,
    start_date="2020-06-09 12:00:00",
    end_date="2020-06-09 12:00:00",
)
df = sites.all()
df = df[df.STATE == "Sachsen"]  # or any other known state of Germany

[1] https://github.com/earthobservations/wetterdienst/blob/master/example/climate_observations.ipynb

gutzbenj on Nov 27, 2020