I work for an electric utility, and we recently purchased and installed 9 Tempest Weather Systems:
We want to monitor and store weather data for our 9 substations, which will later be used in future growth / load studies and other analysis.
Currently, we’re trying to decide if we need to store this data ourselves locally, or whether Tempest’s data storage is adequate.
How long does Tempest store the data in the cloud and how long is it accessible through the API? Is all history kept forever? Is it the history periodically truncated? Are there any limitations I should be aware of? We plan on using the API for various internal apps.
Depending on another company is always risky, but you should probably contact WF officially to get something in writing. FWIW, if it’s me, I’d store everything locally regardless of what they answer.
We do read the forums but not all day long. And as said for these kind of questions you better reach out to the team. This is important and you need a real answer and not assumptions. And it is always wise to have backups … imagine a week with no internet … our servers run, your Tempest works but no way to get them to talk to each other …
This forum is ment for users to help each other mostly and we will help for sure, depending on time we have on hand It is pretty busy behind the scenes. There is a lot more then just Tempest in the company
@eric I see. I guess to me this is a pretty straightforward question that I can’t find an answer to anywhere. Does Tempest store our weather data in the cloud, and if so, for how long? I would think that is a fairly standard question.
When we use the API are we getting data directly off the devices themselves? Or is that data stored in the cloud?
There was a thread here a while back describing the data storage “bucket” logic, and how stored cloud data loses observation frequency as it ages and becomes less relevant. There is no local storage of observations, absent a third-party integration.
I would contact the commercial sales folks at WF, since they work with other utility companies and can probably tailor something to store data to your specific needs…
@vreihen I have a question maybe you can answer. So, we have the Tempest Weather Systems… how often is this device actually logging data? Does it log every minute?
Would that be the sampling interval here:
And related… If I make an API call and set the bucket at 5 minutes, how are these 5 minute values calculated? Is it like an average? Do you know where I can read more about this?
I think you might want to do more reading. How often the Tempest emits data is well documented. You could also run one of the fine UDP listeners and see for yourself.
rapid_wind is once every 3 seconds typically. obs_st is once every 60 secs typically. I’ll skip mentioning events (they happen when they happen) and device status messages as they’re not observations you probably care about.
Note - the Tempest backs off on what it sends and how often when your battery gets too low (until it charges up enough again), but it won’t be ‘faster’ than the above.
re: API calls - again do a trivial test and see for yourself, but it is reasonable to believe that a 5-minute reading is going to be an average over the readings received in that period, likely with min/max readings that should match your UDP measurements.
I do not use the cloud/API data, so i have no insight to offer.
A quick search for “data buckets” revealed this 2018 thread. It is for the older Air/Sky devices, but should mostly still apply to the Tempest (which combined the Air and Sky into a single unit).
I vaguely remember a more recent thread on the buckets and retention, but didn’t save it because I have no interest in the cloud/API stuff and only work with the local UDP API stuff…
Irregardless to whether or not Tempest stores the data. Tempest does integrate with IFTTT and the ability to push your data to IFTTT and then to Google Sheets is a very reasonable alternative to relying on WeatherFlow data. You can keep this data as long as you want. Just use the “New Observation” trigger in IFTTT and write it to a Google Sheets doc. Note… that for some odd reason, IFTTT limits logs to 2000 lines in Sheets so it’ll roll it over and create a new sheet after it hits that threshold. There is a solution to this documented here Google Sheets, IFTTT and the 2000 row limit - a query [EDIT - and Solution]
@mabeatty1978 Thanks… I do quite a bit of Python programming, so I’ll probably go the API route but on a schedule to pull the data daily, or however often we need. And then will most likely store it in a RDBMS.
IFTTT integration could be cool for some other stuff for us though.
I figured this would be an easy question to have answered, but since it’s clearly not, then I’m beginning to believe we shouldn’t trust Tempest with historical data storage.
What I’m noticing as a new Tempest user is that documentation used to be pretty good, but it seems like they haven’t really kept up with new devices. For example, I can’t find any thorough documentation for the Tempest Weather System.
There are a lot of mistaken conclusions in your conclusions there. I think there is so much thorough documentation for the WF systems that it’s hard to find it if you’re coming into the ecosystem new.
Unfortunately the brutal forums software here makes searching painful.
If you’re going to roll your own, just use the published APIs and don’t sweat inconsistencies you notice. Just accept them as-is and code accordingly.
When you see Air+Sky anywhere just think of that as today’s Tempest split originally into two sensorsl. Nothing really changed in the APIs other than a new obs_st type combining obs_air and obs_sky together since it’s a one-device setup now.
If it’s me, personally I’d just sniff the UDP and write to influxdb directly. I might know a guy who wrote a python utility (link) you might use if you have a pi you can put on each LAN with the Hub for a station. Same utility can publish to MQTT if you want to go that way.
Or stick an arduino or something out there for each station, sniff the UDP, and feed MQTT to integrate things and use telegraf/influxdb for your database. Or go weewx and feed from there.
There are a zillion ways to go there without relying on WF servers ever.
@vinceskahan Yeah, you’re probably correct, but IMO that would mean the documentation isn’t organized or maintained properly. Anyway, that’s not really important, but as a new customer, I thought I’d give my 2 cents.
Before I started developing a process to store historical data, I really only wanted to know 2 things:
Does Tempest store data in the cloud, and if so, how long do they retain it?
For the “buckets”, how is the data aggregated? Is it averaged?
The first question was just to see if I needed to go through the effort of developing an internal process to store the data, which is looking like a definite yes. I hate to reinvent the wheel if it’s not needed. I opened a ticket with this question, but it’s just crickets so far.
And the second question was important if I did indeed need to develop something so I could understand how the data is aggregated - which I’m VERY surprised I can’t find this information anywhere.
Thanks for the background history on the devices, that’s helpful, and I’ll take a look at your Python utility and your suggestions.
You’re asking for stuff kinda down in the architectural weeds a bit, so it’s not too surprising it’s hard to find. This stuff has grown a lot over the years. The forums are particularly difficult to search.
Best I can suggest is a google search ala “site: weatherflow.com data buckets” and start reading. The link @vreihen’s provided is the first hit there.
Another link might be Downloading all the back data - but be a little careful about believing ancient references to test API keys etc. as that has changed over the (five) years since that discussion. I would doubt how WF stores and aggregates (or not) data has changed much, in the absence of info saying otherwise.
I think that in the original air+sky days some folks did tests indicating that the bucketed info was the average info over the time period specified but I’ll never be able to find the old threads from several years ago. It might be easier at this point to just do the test.
Grab the readings over a time period from the UDP. Average them up. Grab the same time period via the WF server interfaces. Compare.
To answer your first question… I think it would be prudent for WeatherFlow to go on record and publish a retention policy. I’m sure they would like to keep data indefinitely under the assumption that they can continue to cheaply add more storage as their storage requirements grow. The reality is, it’s hard to say how long they will be capable of retaining data… I’m sure at some point, data would be cleaned up, but if they say 2 years, people will get upset that it isn’t long enough and if they say 15 years, people will get upset in 10 years when they run into a data/performance/storage issue and need to trim it sooner.
To the second question, I do agree that the documentation is poorly organized. But this getting started guide has been all I’ve ever needed to do any of the development work I’ve done.
WeatherFlow Support’s answer to this question via my support ticket:
Tempest data is stored in the cloud indefinitely and is always accessible. As of now, device data history is kept forever. Only internal calibration bias corrections are periodically cleansed after several months of storage. I am not aware of any limitations beyond only being able to access data from your own network of Tempest stations unless you are subscribed to a higher tier allowing access to a wider weather station network accessible through the API.
It is pretty busy behind the scenes. There is a lot more then just Tempest in the company 