Data archive buckets explained


#1

So how does WeatherFlow store data?

Explained: your AIR & SKY sensor devices are constantly collecting and sending observations to the HUB. The HUB then uses a set of sophisticated algorithms to inspect the data before publishing to the secure WeatherFlow servers. HUB also makes the data available locally via UDP and BLE broadcast for users who wish to collect and store the data at home.

The secure WeatherFlow servers not only store all the 1min data, but the data is also processed and stored as 5min | 30min | 3hour | 1day . Each temporal resolution is it’s own data set, ready to be queried quickly by the app for your display.

Here’s a more detailed summary of stored data by sensor device and parameter:


Downloading all the back data
Wind Speed Questions
Change in graphing rain rate
New owner- Pro & con observations
What is wind gust anyway
What is wind gust anyway
Questionable wind speed data (3hr vs. 1min data explained)
Wind Speed Questions
UV Graph readings
Questionable wind speed data (3hr vs. 1min data explained)
WeatherFlow PiConsole
Wind comparison between SKY & Acurite (solved)
REST API: my wishlist to Santa Claus
pinned #2

#3

@dsj I know this is slightly off-topic, but regarding the data buckets is there any chance of replacing “Closest Ob” in many of the data buckets with “Avg” over the period of that data bucket instead? “Closest Ob” can give a very skewed impression of how parameters have varied throughout the day, if it happens to be anomalous compared to the rest of the day.

I don’t have any particular examples to share, but say, for example, during a cloudy day there might be single break in the cloud that creates in peak in solar radiation for 5 minutes at noon. This peak is far greater than anything else observed that day, however when you zoom out to say the 3 hour bucket, it complete dominates the graph, giving the impression the day has been significantly more sunny than it actually has. Averaging the data over that 3 hour bucket would be a significantly better overview of how the radiation as evolved.


Change in graphing rain rate
#4

I moved your post so it’s no longer off-topic!

That’s a really good question. Reducing data can be tricky and inevitably there is some loss of fidelity. Many parameters don’t change much over a short time period, and in that case “snapping” to the closest observation creates the “truest” line. But you’re right that this is not necessarily true for the solar parameters (UV, Lux, Radiation). Perhaps, like wind, we should display three values (high, low, average)?


#5

btw, rain rate is missing from that table with the data buckets, or isn’t it stored, but calculated from the accumulated rain?


Change in graphing rain rate
#6

Good catch - that’s an oversight. We’re missing solar radiation too. We’ll get that updated shortly!


#7

Personally I would just display the average. Max/min can be left to the Z4 (1 day) level (which as an aside, I’ve always thought would look better if a line connected the daily averages).

I think this is true for the Z1 and probably Z2 level, but at the Z3 level, I think all parameters (not just solar radiation) can change sufficiently rapidly in a non-monotonic manner between two discrete observations that the average would give a far truer impression than just the discrete observations.