Determine earliest tempest station data at WF

Owners
#1

Hi, I’ve had my tempest station (still running version 134) and hub (running version 147)
I’m trying to determine:

  1. How to download all my station data from WF to WeeWx (4.4).
  2. Determine earliest epoch timestamp recorded by my station (station ID=40660)
  3. Determine proper way to extract that data into a CSV file on local machine (i assume use the API)
  4. Practice using wee-import to get it correct.

Questions:

  1. I notice that the API for station/device info shows “station Units” to be in US options (i assume that’s what i’d selected during install of ST), but i’m assuming all WF data is in metrics?
  2. When i use the sample API for my device observations, i notice that there are quite a few “Devices” under the “Station” id, does UDP collect data for each device, or aggregate all devices data before the hub releases the UDP call? i.e., do i have to download each device’s data to weewx, or is the data stored from all devices for a station?

And, yes, i’ve poured thru the Weewx Utilities and various forums, but no one seems to have commented or asked questions about this specific topic that i can find.
Thanks!

#2

You’re asking both something new and something relatively painful to do, especially since you’re going first on this one.

First question I’d ask is '*why bother - is your historical data THAT important ?

Second question is "Approximately how many days of old data do you think you need to catch up on ?’

But that said:

  1. The Hub emits metric and I think the app and website let you tweak what is displayed to your tastes, but it would be surprising if the WF servers don’t also store in metric too.

  2. The UDP emits what your sensors measured. What is available on the WF servers can be different, depending on whether you have RainCheck enabled or not. The lightning is always synthetic based on surrounding sites and your measurements. I don’t know personally whether downloaded lightning data via the web interfaces is just your data or not. It would need an experiment to verify either way.

The REST API ‘get observations’ looks metric to me, matching the Hub UDP API notionally. It looks like it works off UTC time if you drill down through Tempest API

You can grab 5 days at a time according to “TIME FILTER - Time range start time epoch seconds UTC. Observation data at a one minute time resolution is available for a time range that is five days or less. You also need to send “time_end”.”

So what you would need to do would be something along the lines of:

  • grab multiple up-to-5-day intervals of data via multiple API calls
  • reformat the JSON responses into something CSV to line up with wee_import’s expectations
  • build a big’honkin’ set of data for wee_import
  • try to import it

If you do multiple passes, don’t worry about duplicate key (dateTime secs since epoch) messages, they’re just warnings. Sqlite3 will just skip records for a timestamp already in the database.

Do a one-hour test to get your formatting and process right…and write it up for the weewx wiki please so the second guy doesn’t have to also blaze the trail on this one…

#3

Vince, once again, thanks!

  1. On historical (hysterical?) data, I’ve only had the station for a month, and ran the weewx simulator for most of that time; so, I’d like to gain all of the reported data back to weewx for review purposes. Plus, I think the ability to fill in missing time periods (as easy as possible) might be handy as Weewx is running on a laptop that sleeps most of the time.
  2. In trying to understand the juxtaposition between WF data and Tempest raw data, I’ve been reviewing the API data and the API/UDP data.
    Reading the API, I thought the 5 day time period was restricted to current-time+past 5 days. If you can spec different 5-day intervals then that makes it easier to loop intervals until done.
  3. UDP data:
    a. I noticed that WF has multiple devices listed under each “Station”.
    b. Each UDP packet has the a preamble of device_status, hub_status or rapid_wind that repeats two, three or four times with either a hub or device record, but only lists station id, not device id.
    So, I’ll embark on trying to assemble database layout w/keys; workout curl or wget downloads of data. I’ll let you know how it goes.
    David
#4

You can grab 5 days at a time according to “ TIME FILTER - Time range start time epoch seconds UTC. Observation data at a one minute time resolution is available for a time range that is five days or less. You also need to send “time_end”. ”

I don’t think it is implemented that way. It only works for the last 5 days.

When I want data between the first of January and the 4th, I do the following request (after authorizing it):

https://swd.weatherflow.com/swd/rest/observations/device/105013?time_start=1609545600&time_end=1609804800&format=csv

the response is:

"device_id","type","bucket_step_minutes","timestamp","wind_lull","wind_avg","wind_gust","wind_dir","wind_interval","pressure","temperature","humidity","lux","uv","solar_radiation","precip","precip_type","strike_distance","strike_count","battery","report_interval","local_daily_precip","precip_final","local_daily_precip_final","precip_analysis_type"
105013,obs_st,5,1609545600,0,0,0,0,6,1009.3,4.1,96,0,0.0,0,0.032193,1,0,0,2.42,5,0.032193,,,0,
105013,obs_st,5,1609545900,0,0,0,0,6,1009.3,4.1,96,0,0.0,0,0.03013,1,0,0,2.43,5,0.081733,,,0,
105013,obs_st,5,1609546200,0,0,0,0,6,1009.3,4.1,96,0,0.0,0,0,0,0,0,2.43,5,0.081733,,,0,
105013,obs_st,5,1609546500,0,0,0,0,6,1009.3,4.1,96,0,0.0,0,0,0,0,0,2.42,5,0.081733,,,0,
105013,obs_st,5,1609546800,0,0,0,0,6,1009.4,4.0,96,0,0.0,0,0,0,0,0,2.42,5,0.081733,,,0,
...

Showing there are 300 seconds between timestamps.

One minute data is available by using the day_offset option in this case 53 days ago. (you need multiple requests for a multiday period) The request would be:

https://swd.weatherflow.com/swd/rest/observations/device/105013?day_offset=53&format=csv

the response is:

105013,obs_st,1,1609545600,0,0.19,1.12,64,6,1009.3,4.1,96,0,0,0,0.01941,1,0,0,2.42,1,0.051603,,,0,
105013,obs_st,1,1609545660,0,0,0,0,6,1009.3,4.1,96,0,0,0,0.014652,1,0,0,2.42,1,0.066255,,,0,
105013,obs_st,1,1609545720,0,0,0,0,6,1009.3,4.1,96,0,0,0,0.013781,1,0,0,2.42,1,0.080036,,,0,
105013,obs_st,1,1609545780,0,0,0,0,6,1009.3,4.1,96,0,0,0,0.001697,1,0,0,2.42,1,0.081733,,,0,
105013,obs_st,1,1609545840,0,0,0,0,6,1009.3,4.1,96,0,0,0,0,0,0,0,2.43,1,0.081733,,,0,
...

@dprell note that you can get csv responses directly from the API

1 Like
#5

Look at the UDP API and you’ll see there are a variety of messages with different intervals, as well as their content. The reply from @sunny give you the howto basically. Note that you have to supply the date+time as seconds since the unix epoch…