Data archive buckets explained

Tempest Owners
1

So how does WeatherFlow store data?

Explained: your AIR & SKY sensor devices are constantly collecting and sending observations to the HUB. The HUB then uses a set of sophisticated algorithms to inspect the data before publishing to the secure WeatherFlow servers. HUB also makes the data available locally via UDP and BLE broadcast for users who wish to collect and store the data at home.

The secure WeatherFlow servers not only store all the 1min data, but the data is also processed and stored as 5min | 30min | 3hour | 1day . Each temporal resolution is it’s own data set, ready to be queried quickly by the app for your display.

Here’s a more detailed summary of stored data by sensor device and parameter:

AIR_dataarchive

SKY_dataarchive

9 Likes
3

@dsj I know this is slightly off-topic, but regarding the data buckets is there any chance of replacing “Closest Ob” in many of the data buckets with “Avg” over the period of that data bucket instead? “Closest Ob” can give a very skewed impression of how parameters have varied throughout the day, if it happens to be anomalous compared to the rest of the day.

I don’t have any particular examples to share, but say, for example, during a cloudy day there might be single break in the cloud that creates in peak in solar radiation for 5 minutes at noon. This peak is far greater than anything else observed that day, however when you zoom out to say the 3 hour bucket, it complete dominates the graph, giving the impression the day has been significantly more sunny than it actually has. Averaging the data over that 3 hour bucket would be a significantly better overview of how the radiation as evolved.

4

I moved your post so it’s no longer off-topic!

That’s a really good question. Reducing data can be tricky and inevitably there is some loss of fidelity. Many parameters don’t change much over a short time period, and in that case “snapping” to the closest observation creates the “truest” line. But you’re right that this is not necessarily true for the solar parameters (UV, Lux, Radiation). Perhaps, like wind, we should display three values (high, low, average)?

1 Like
5

btw, rain rate is missing from that table with the data buckets, or isn’t it stored, but calculated from the accumulated rain?

1 Like
6

Good catch - that’s an oversight. We’re missing solar radiation too. We’ll get that updated shortly!

1 Like
7

Personally I would just display the average. Max/min can be left to the Z4 (1 day) level (which as an aside, I’ve always thought would look better if a line connected the daily averages).

I think this is true for the Z1 and probably Z2 level, but at the Z3 level, I think all parameters (not just solar radiation) can change sufficiently rapidly in a non-monotonic manner between two discrete observations that the average would give a far truer impression than just the discrete observations.

1 Like
8

I just found out that weatherflow servers do store all the 60 second data. This is data I just retrieved from the server. Note the 60 seconds difference in the timestamp (4th number in a row). 1551312000 is computer language for 28 Feb 2019 0:00.
I’m glad they do :smiley: (weatherflow didn’t say it doesn’t store 1 minute Z0 data forever, It might be that in addition to that they store the Z1, Z2, Z3 and Z4 data)
I just got the impression that all the old 1 minute data would be replace by Z4 in the end. Not so!

15472,obs_sky,1,1551312000,0,0,0,1.56,2.05,2.73,41,3.42,1,0,0,,,,,,
15472,obs_sky,1,1551312060,0,0,0,0.89,1.62,2.15,334,3.42,1,0,0,,,,,,
2 Likes
9

Can we get an update on this for the Tempest Weather System?

10

Nothing has changed. Although this post refers to Sky and Air devices, which have now been combined into a single Tempest, the details for each observation remain exactly the same

1 Like
11

Interesting, thanks. Maybe @WFmarketing should update the post body to make this more clear.

12

@peter Do you know if “closest observation” is still used? Or did they make changes based on your suggestions?

13

As far as I can tell, “closest observation” is still being used