UDP is higher than what?
Are joking or just didn’t reading any else but that one part?
No, I am not. UDP and MQTT are the same values. The Hub does not send one value to UDP and a different value to WeatherFlow.
EDIT: Actually, that’s true but not entirely true. The data sent is the same but the WeatherFlow “continuous learning system” may actually be throwing out some data if it thinks it is false rain. If in fact that is the case there may be a deeper issue here.
You need to be specific and give better details and we will help you track down the issue.
I’ve given the specifics several times in this thread (even had to specifically re-explain it to you once before). Earlier on WF staff already said that the UDP data I pull and what’s displayed from their servers will never match because of the AI stuff, but it should be closer once the calibration deployed.
Here maybe these screne shots I just took will make it clear to you (again).
This is part of the issue. First, the sky will not pick up mist and very light rain. It won’t send rain data to the Hub. So it’s not that UDP doesn’t pick it up. Nothing is going to sense it.
Second, you are using an ISY node server to capture and display the rain. It’s not fair yo call that UDP data, even though there is a good chance the values are derived from captured UDP data because the data is manipulated by the node server and we have way to verify if it is actually accurate.
The data that the ISY node serve is using needs to be compared to the actually UDP data and that compared to the data stored at WeatherFlow in the device database.
Now that we are all up to speed, I can discuss this with Bob (I have worked with Bob on his node servers) and we can compare data. I also want to get more information from David as to exactly what happens to the data received by MQTT before it is stored.
I am not doubting you have an issue, I just don’t know where it is. I capture all the UDP data and store it locally and I am not seeing the issue you are seeing.
Gary this is a redundant conversation but if you think you can verify more about what I’m seeing that would be great.
Here’s what WF said last time we talked about this:
For rain in particular, the haptic sensor is unique as it can tell you instantaneous rain intensity which a tipping bucket cannot. Sensitivity tuning limitations wane at the lighter end of the rain intensity spectrum (ie. light drizzle). This is where some of the backend magic comes into play – we will use an array of other high-res precip data in addition to augment haptic’s rain intensity raw values to QC and compile hourly and daily accumulation values. This is accomplished on WeatherFlow network servers and is available via REST API, but is missed by the local-only UDP stream. The raw intensity data and subsequent accumulation values broadcast by the HUB via UDP will continue to improve with in-situ calibration, but the UDP stream will not benefit from the other network intelligence . — which is why you will see a difference in rain accum API vs UDP.
When I get a chance I’m going to be setting a RPI to pull UDP(I’ll probably move over to websocket later) from my WF as I’m probably going away from ISY (or at least off loading a lot of things from it) and I’ll see if what it pulls any different. I’m just not sure when I’ll get around to it, maybe after I finish rebuilding a customer’s 5 axis rotary table between service calls over the several days.
As for light rain I don’t mean mist, I talking about when WF reports rain as light. On a day or portion of a day when it’s light rain both WF and UDP are less than other local stations (at these times the error between the 2 isn’t very high but goes up the heavier the rain is, so maybe maybe the node is doing some bad math
Okay, I’m on the same page as you. As calibration improves the local only will improve. I need to ask David but I am looking also using the web socket data to correct the UDP rain data. I’m not sure that’s a possibility though. Also there is the possibility of pulling down the Archive device data to correct any locally stored data. There are still a lot of changes coming and we just have to wait and see what changes.
I’m going to add a tipping bucket and use that data along side of the Sky data to give a better indication of rain.
The information you provided is good and hopefully we will both see some improvement in rain data. Thanks for taking the time to discuss the issue.
BTW, I had assumed that the UDP data from the node server was correct because during the first rain events it much closely matched WF and drifted appart later. I figured it was due to what WF had been doing for false rain and with all the bird landings I get the server is seeing my heavy rain as false rain, but all the earlier rain events were lighter rain so maybe things are about the same since rain rate seems to effect the difference a lot.
If after the calibration goes live my rain data is still way off I was planning to also get a tipping bucket for accumulated rain and use rate from sky since it’s instantaneous. I mentioned this before (might be in a different thread) one time we had light rain starting in the early morning and my WF didn’t detect it was raining until just before my irrigation rain sensor tripped (adjusted to .25’’) , I was surprised it didn’t sense rain at that level as it was more than mist but still light enough it 4-5hr to accumulate .25’’, if another event like that happened the sky wouldn’t do much good for rain rate.
When talking about the node server, it helps to mention which one. There are currently 3 versions, stand-alone, Polyglot local and Polyglot cloud.
Rain accumulation took a while to get right, but the most recent versions should be accumulating what is reported over UDP correctly. The stand-alone version has the capability to collect data from both UDP packets and websocket connections to the WF servers. However I don’t think it can do both for the same station, but I haven’t used that version in quite a while. The cloud based Polyglot node server also gets it’s data from the WF servers but via the REST interface. It would be possible to run both the local and cloud based Polyglot node servers simultaneously and have the two rain nodes in the ISY to compare.
I’m using the windows version (I assume that’s the stand-alone) I don’t see any options for using websocket so I assume you moved where you’re hosting it because the page I was getting them from has the same version I’ve been running on it (22.214.171.124) as well as the thread on UDI forum. For most of what my ISY uses the data for UDP is a better source since it doesn’t rely on internet connection, ISY on uses the rain data for my weather overlays (well sends it to Blue Iris).
Assuming WF server rain data ends up being more accurate I’ll probably move over to using websocket. I just hope someone writes something for weewx my python coding down to fixing a broken Kodi add on, there’s plenty for python websocket clients so figuring out how make it into a weewx driver (I think that;s what they call them) will be hardest part for me. I decided on weewx because it already has MQTT, something for my rainmachine, a UDP for WF and some nice skins if I choose to ever use them.
If you mean WeeWX for WeatherFlow that was done early last year.
I only saw UDP not websocket
My apologies but that wasn’t clear so again I misunderstood your meaning.
I have to admit being a bit lost in your various comments and what exactly you’re looking for. My understanding is:
- there are periodic UDP broadcasts the WF gear sends onto the LAN
- the Hub also communicates observations to the WF servers. Same data. Different transport.
- the weewx UDP driver simply listens to the UDP broadcasts and populates the weewx db locally.
- the WF app displays what is known on ‘their’ servers, which also can include calculated/derived data not emitted as a sensor reading from the Hub
- so any piece of the observation data (rain, temperature, humidity, etc.) should have the same data in weewx and in the WF app
Now that said, the Hub doesn’t emit ‘derived’ data that WF cooks up on their end based on the raw sensor data from the station. You can’t get something from the Hub that the Hub doesn’t emit.
But I’m not aware that the CL software changes anything for an observed value like rain after the Hub emits the observed value. Does it @dsj ? That would be pretty confusing. I thought the CL stuff just makes the Hub emit more accurate data essentially.
My opinion is you don’t need any weewx websocket driver unless you need data that is only available by querying the WF servers via a websocket.
Yes, that’s the stand alone version and it’s the latest one. In the configuration there’s a checkbox for each sensor called “Remote”. Checking that (and having the actual station ID populated) will use a websocket connection instead of UDP to get the data.
My understanding is that the websocket data and the UDP data are the same. The only exception would be the daily rain accumulation. The hub doesn’t do any accumulation so always reports that as NULL, the node server has to do the accumulation. So if WF is correcting the rain data before adding it to the daily totals, there would be a difference between the WF app and the node server.
The WeatherFlow servers do derive additional metrics and those are available via the REST API. This node server doesn’t use the REST API and doesn’t have access to those derived metrics. It would be possible for the node server to derive the same metrics itself if that was something people were interested in.
The last point contradicts the one prior to it. The app will show that calculated/derived data (which from what I understand does happen with rain data) so if anything changes it won’t match your local UDP data.
I image there will be a lot of magic going on the servers due to the rain sensor type before it’s all said and done and it will be the most accurate. Since one of my primary uses for my WF station is to help my irrigation system not waste water by over/under watering I want to use what ever data is correct but right now non of my data is for rain.
Disagree. And also disagree about any conflict in the two bullet items.
I think you’re confusing their CL algorithms and data processing magic running on their servers…with the resulting individualized ‘calibration’ they push to each unique Hub so that each station’s emitted observations ‘tomorrow’ will be more accurate than it was ‘today’.
For parameters that are directly measured by a sensor (rain, temp, humidity, windspeed, etc.) the Hub transmits a human-friendly representation of that data to the LAN over UDP in JSON format. It also communicates the same information in a WF-server-software-friendly format to the WF servers.
Every time I have looked at rain (specifically) and compared the WF app totals for a period with totals I got from either weewx (listens via UDP) or my alternate grafana dashboard graphing influxdb data (generated by my python listener monitoring the same UDP broadcasts), they all agree exactly. Like ‘really’ exactly to the Nth decimal point. If they’re messing with our data, they’re adding 0.0000000 to it
And regardless, you’re waaaaaaay overoptimizing your rain scenario here anyway.
The WF gear is ‘not’ good at this time regarding precipitation. Like really really not good. If you want to be ‘that’ super-optimized, you probably need to buy other gear from this or another vendor, and you’d need to significantly up your price point.
This station is simply not ready for prime time currently regarding expecting precipitation to be even plus/minus100% accuracy vs. truth in the 7 months I’ve had mine in service, and it’s not getting better. I simply discount any and all precipitation readings from the WF station as bogus data.
But regardless, I’ve not heard anybody from WF say they mess with our raw observation data after it’s received on their servers. If they do, I’d sure like to hear that officially.
My data used to seem to match also so I trusted whatever Bob did with his node server was showing whatever my sky sensed.
You really haven’t heard WF say they mess with our raw observation data after it’s received on their servers, yesterday I just copy and pasted a statement of them saying they do from a post they made in this thread.back in Jan.
I actually started seeing the mismatch between the app and what’s on my local data after WF mentioned in another thread that they were doing some stuff to help with false rain. My assumption was that since I had been getting a lot of bird lands what ever they did to help minimize the false rain from those is what cause a lot of the difference.
I 100% agree that the SKY isn’t ready for prime at least for rain (I guess there’s some issues for wind but I can’t say I’ve noticed anything significant on my end). I didn’t even know there was any issue with he rain detection until after we started getting our stations and hoped it would be sorted by the time my rain season got here.
I don’t need it to be perfectly accurate but as time went on at times it’s so far off it’s of no use. As I said before I’ll see what happens after the calibration if it’s not were I feel I need it I’ll pick something else up for rain.
We’re supposed to get rain next Thur, I’ll try to get one of my RPIs setup with weewx and see if it matches Bob’s node server’s data and WF’s servers.
It’s important to have a ‘wired’ pi if you want to catch all the observations. I’m running on a pi3b+ and it never misses any at all. Getting weewx running is pretty simple with the UDP driver.
Also - remember weewx won’t show the same ‘number’ of observations, but it will show the same ‘totals’.
It shouldn’t matter which model pi as long as it’s wired, but a pi3 or pi3b+ would be best if you have a modern one available. You can alternately run it in Docker, in Vagrant/VirtualBox etc. if needed. Works the same everywhere…but a pi3 that’s wired is plenty good enough.
Well I was probably going to use one of the pi zero w’s I picked to use for occupancy sensing. It would be on my HA vlan/ssid but if that isn’t good enough I could add a wired connection. Docker is another route I’d possibly go, I think I might even have it already set up on my BI server but that one only has 1 NIC (which is busy with video feeds) so I’d what to put it on my actual server server.
I’ve also got a plan in place for a secondary rain gauge which will consist of a RAINEW 111 and pi zero w for a total cost of $70, which will also easily integrate with weewx.