WeatherFlow API Changes

@Weather34 mentioned in his thread Weather34 PHP Template that WeatherFlow are making some forthcoming changes to their API. Can @dsj or anyone else in the know please make the details of these changes clear ASAP? Those of us who are currently developing with the API do not want to be caught out when the changes happen, and do not want to invest a lot of time in code that is going to become superfluous.
Thanks.

3 Likes

Hi Peter. Fear not! We don’t actually have details, yet, because nothing’s final yet. I think my direct message to @Weather34 may have come off as more “final” than I intended it to be. I have been chatting with several developers directly for a little while, explaining the issue and collecting feedback. I am certainly hoping @weather34 continues with his project once we do settle on those details, perhaps with a tweak or two.

We were planning to seek input from the developer community soon, well before anything is implemented. So now that the cat is out of the bag, here goes:

From a high level, we want to support all third-party developers - we just need to make sure your apps work together, rather than against, ours. And I’m certain we can find a way to do that!

The main thing we must control (as I’m sure you can appreciate) is the ability for a third-party app to view or download ALL data from ALL public stations. That is currently possible, even using the developer API key. We have been clear (hopefully) in our developer docs, that there would be restrictions on data access. We simply haven’t applied any, yet. That’s what we need to do soon.

Our current thought is to make one small change to the way a normal API key works, as follows:

  • No change: Station owners have unlimited access to ALL data (metadata and observations) from his/her station, whether public or private, either directly or via a third-party application.
  • No change: Third-party applications have access to metadata for ALL public stations
  • Change: Third-party applications have limited (or no) access to observations from public stations

Our goal is (and always has been) to allow users unfettered access to the observation data produced by their own stations (either locally or remotely), and to allow any developer the ability to create applications and integrations that work on behalf of a user. And an app offering unlimited access to all observations would still be possible, it would just require partnering with us in some way.

What do you think?

5 Likes

I would opt like, if data is set public, then data is useable to third parties (maybe fuzzy a bit position ??) BUT by default the station would be private, for dummies not knowing what they do. At least they can’t yell if their data is openly available Once you toggle public (with a good big warning that data will be available to others in blablabla way…

for the last point I might propose something in between

As you know, you (anyone) has to respect the fact that when someone decides to stay private, you can’t use his location at minima …

But maybe you could introduce something like many others do, allow to use the data in an anonymous way. For those that would like to use the data for statistical purposes, you make some data available but without link to whom or where, though for weather data not knowing where can be difficult but still … not to sure what data could be made available to third parties in this way …

1 Like

Hi David. Many thanks for taking the time to explain your thinking so clearly. I apologies if my original post come across a bit demanding. I imagined you were going to be changing the underlying architecture of the API, and all code developed to date would become useless. I’m glad to see that this is not the case!

I appreciated your need to control the ability of third party apps to access ALL data from ALL public stations. You have certainly made it clear that users will always be able to access their own data, but you have also indicated to the community that WeatherFlow as a company intends to leverage the huge amount of data that this weather network will be providing to produce commercial outputs such as hyper-local forecasts (see: Forecast API/Software?). I assume you’ll be using some of these commercial outputs to offset the likely huge development costs you have sunk into development of the weather station, so it seems reasonable to me.

However, I would caution that the community is always going to feel like it ‘owns’ more than just their data. After all, it is the community that is buying the product and going to the effort of installation etc. Just look at the backlash against Weather Underground for removing their free API (I know you have no intention of doing this!!!). Therefore I think there needs to be a halfway-house whereby access to ALL data for a third party developer can be given on a case-by-case basis (as you have already indicated might be the case). For example in this system I would envisage that the maps simply showing the conditions at each station being developed by @GaryFunk and @Weather34 will be given the go ahead, as they are not generating any derived output. In contrast, an app that wants to derive a forecast, which clashes with your commercial apps/output, may not.

Hope this makes sense, and many thanks again for taking the time to explain.

PS. Please don’t make any changes soon!! I am still waiting for my station (I was a very late backer), so am having to relay on data from another station during development.

1 Like

Let me say this about that.

David discussed this with me in May and I gave him my comments at the time. It’s unfortunate that these private discussions were made public before WeatherFlow has decided on the final API changes.

I have been working with WeatherFlow data for just over a year. The current API does not lend itself well to collect data a large number of stations.

Let me use the Smart Weather Station Map as an example.

  • The latest station to be added is Station ID 4259.
  • There are currently 1375 public stations.

To update the data that produces the Map, I have to query each Station ID. That is 4259 calls to the API. The API returns a Status of 0 or 404. If the Status is 0, I make two additional API calls to get the Meta Data on the station. If the API returns a 404 I mark the station as 'invalid."

The problem with Status 404 is it is returned form non existing stations AND private stations. So in order to find a Private Station that is now Public, I have to continually poll all invalid stations to find one that is now public.

At this time, that is 2884 additional calls to find one a single station that may have changed. It’s becoming a burden on the WeatherFlow servers to maintain the Map. This is why the Map is only updated once every 24 hours.

Several have suggested that I build a map of temperatures across the world. This would require an application that constantly hammers the WeatherFlow server for data as weather data can only be read one station at a time. This is not practical with the current API.

Another issue is the stations that have never reported any data or no data in the last 30 days. There are 105 stations that get polled once every 24 hours yet I suspect they will never report any data.

As you can see there will come a time where it will take more hours to update that Map data than there are in a day.

As for getting weather data for templates, it’s no big deal. 99.9% will be for stations that the user controls. Third party applications will work that same way as the phone applications to. They will be authenticated with OAUTH2. It’s easy and super fast.

As David has mentioned, the changes are a few months off and there will be many discussions among developers and WeatherFlow staff.

On a personal note, I want to suggest to David and WeatherFlow that a private section be set up and ONLY developers be invited to participate. It should be moderated only by David, Keith, Corrine and internal WeatherFlow programmers to cut down on the unnecessary comments and noise.

I will take a wait-and-see posture and continue to develop applications for local use. The only issue I foresee is obtaining data that is not available via UDP and requires calling the REST API.

So, to quote Officer Barbrady, “Move along people! Nothing to see here!”.

And above all, DON’T PANIC!

Remember the famous words of Dixie Longate who once said, “Never Wear a Tube Top While Riding a Mechanical Bull.”

5 Likes

all I see the problem is, is with trying to download every station data to produce a temperature map
putting a burden on the servers
just getting data for one station should not be impacted, so not sure why what is available now via 3rd party has been withdrawn, that does that

of course a better way to get all station data is its a list of the each station data all together (i.e in the format [data1,data2,data3] etc

I do suspsect WF will be wanting to roll their own features that use all (public) station data, e.g mesonet data and even forecast data (e.g 12 hours ahead)

I agree that some will want to limit the weather data that is generated by their station. If that respect I suggest that new items be added to the phone and web applications

My thought on this matter is you may consider adding more options to Public Data.

Yes/No

Keep Data Private
(No data is sent to WeatherFlow)

Send Data to WeatherFlow
(Data is sent to WeatherFlow for use with calibration. Data is not used by WeatherFlow in any other way)

Share Data With WeatherFlow
(Data is sent to WeatherFlow and allowed to be used by WeatherFlow)

Share Data With Authorization
(Data is shared with third-parties with some type of authoriczation - to be decided on)

Share Data With Third-Parties
(Data is shared publicly and no authorization is needed)

Question and comments are welcome.

5 Likes

I would lean more towards this type of policy. I feel it allows each user more flexibility and also allows for more innovation in the future.
–Sam

2 Likes

You might want to look at the fitbit api for an example of a pretty good one. I can download my info for a range of days in one call rather than trying a day at a time and quickly hitting their api limit in calls per hour using my api key there.

I could forsee similar apis for getting a range of stations or the like for this case…

It would be the same API, just with added functions. I already have a list of functions I’d like to have access to.

1 Like

Thanks for the feedback, folks. We’ll be digesting it all and tweaking our API policy over the next few weeks. I’m confident we can find something that works for all (users, developers and WeatherFlow). Couple of points for now:

Hmm, this is a good point about the expectation of some users, though I don’t follow the logic. If I buy a web camera and set it up in my back yard, I DO expect to have full access to the data it generates (Nest Cam, are you listening???), but I DON’T expect to have full access to my neighbor’s video camera (unless he shares it with me).

Nothing will be implemented without plenty of heads-up.

Correct. That’s because our initial focus for 3rd-party apps has been on individual users accessing data from individual stations. We do have “admin” (not public yet) functions that make it easier to collect data from many stations at once. We also have functions that provide reduced data (stats or metadata) about multiple stations. We currently use these services on our back-end for reporting and visualization by our customer service and IT staff.

Correct again. Gary has created a map showing multiple stations at once despite not having more efficient multi-station services in the API. As he notes, the hoops he has to jump through to make that work are inefficient for both Gary’s system and ours. Not too big a deal with 100 or even 3000 stations. But as the network grows to 25,000 or 100,000, it won’t be viable.

Yes! Nothing we are planning will limit the ability of you to access data produced by your station. We have an Oauth2 interface that works great (it’s how owners connect to Alexa, Google, Nest, IFTTT and others right now). We don’t require that for public stations at the moment, but we will likely need to do that. We might also have to impose a limit on how long we store data form a given station, but that’s a longer-term issue.

Yes for sure! It’s important to note, if it wasn’t clear, that we’ve ALWAYS intended to build an interface for viewing data from large numbers of stations at once, in a map form or otherwise. In fact, we’ve been doing this for many years (see windalert.com for example).

More granular levels of “private-ness” will likely be necessary. We are considering a few concepts similar to Gary’s proposal. The tricky part is making it simple for the average user.

Thanks we’ll take a fresh look. We did look at fitbit a while back (and many others). Looking at how someone else solved a similar problem is always a good place to start!

5 Likes

David, Get off your computer and go play with your family.

5 Likes

I think we may have crossed wires here! I fully agree that no one should expect to have access to their neighbours data without permission. My logic, however, was more to do with how WeatherFlow as a company leverages the data from the network. If you start using anonymous data from large groups of stations to generate commercial output, then some users may start thinking along the lines: ‘I spent money on a weather station and this allows WeatherFlow to make money and therefore WeatherFlow “owes” me’. I don’t condone this line of thinking, but I think if you just restrict the API to allowing individuals to access only their own data and nothing more, you may run into this. I think you need a way that allows the community to also access the data from the wider network of weather stations - without violating privacy of course!

Hope this makes more sense!

1 Like

Thanks for the clarification, Peter! We are on the same page. The question to be ironed out is specifically “how” access to the wider network is provided. Keep the ideas coming and stay tuned!

3 Likes

@peter - If it makes you feel any better, every company/manufacturer that I know of that has tried to “monetize” consumer-grade weather station data has ultimately given up on their effort. I suspect that the data’s value is lower than the costs to run the servers to collect it, and even Weather Underground (now owned by IBM) seems to be losing interest in PWS data. Maybe WF will have better luck because they are coming into it as a data company with a professionally-installed sensor network, but I personally would not invest my own money in a business plan based on generating revenue from PWS data…

1 Like