No idea at all. In the past I’ve fiddled with the Saratoga templates and getting weewx to write the right data files out via weewx-wd, but it’s always made my brain a little foggy…maybe I’ll take another pass at it just for the heck of it.
Really nice analysis (omitted for brevity).
I’m not suggesting WF is doing anything ‘wrong’ here, just noting that the weewx reference driver for the VP2 does fill in the blanks for this one item where the sensors emit the data, keeping track of the max over that time period, and that the code part seems pretty simple to similarly add in for WF (if desired).
Incidentally, another link is the weewx hardware guide which has a lot of info on the many types of stations, what the measure, and what each weewx driver calculates as needed per station. Just glancing at the list, only a few vendors provide windGustDir and lots don’t even measure gusts
(and the vantage section saying windGustDir comes from the hardware is a little misleading, but true it does come from the hardware…with the driver is figuring out what to write in the assembled loop and archive packets)
this thread has evolved into something pretty weewx specific re: approach. Any objections if we ask the question on the weewx-development google group since that’s where those guys hang out ?
adding WeatherFlow to the weewx hardware guide would be cool to do. Any objections if I take a pass at it and run it by those guys ?
Sorry to update, but wanted to save you the trouble of looking…looks like Tom’s VP2 driver also calculates dewpoint/heatindex/windchill at a minimum, in addition to getting windGust and windGustDir from the measured wind data.
windGust and windGustDir - from the wind readings over the time period
dewpoint - via calls to wxformulas.py
heatindex/windchill - via calls to wxformulas.py
(pressure trend seems to come from the hardware)
rain rate - done outside the driver by adding up rain over the interval
(sealevel pressure seems to come from the hardware)
not sure about vapor pressure
Again, I don’t know what breaks if any of these are missing…but my (somewhat dated) experience is weewx-wd and the Saratoga templates would be the torture test to know what the worst case might be…those things are really picky.
I have no problem if you want to follow up on weewx-development. My personal feeling is that their published guidance should be followed, which says don’t fill in the blanks if the station doesn’t provide a value. I’m not going to question Tom’s choices in the Vantage driver, since it probably pre-dates the docs and his use of the Davis logger data is a thing of beauty. WF is providing gust speed data every minute in obs_sky, so that is available if something really needs it.
I believe that the stumbling block is that the MeteoTemplate uploader has an option “binding” that takes either “loop” or “archive” as options. Weewx is properly going back and filling in the archive database blanks for the missing windGust and windGustDir fields, apparently based on loop packets. I suspect that these fields are not being passed to MT because the default binding is loop packets and not using the archive data. Since my MT instance is being fed from my Davis VP2 and not the WF hardware, I can’t dig deeper without setting up a test MT instance and feeding it from WF.
Regarding the Hardware Guide, it would be nice to have WF listed there if it meets their criteria. (One of the big items on my to-do list is to implement the wee_config setup routines, which will include building a basic sensor_map.) I know there are plenty of other stations not listed in the Hardware Guide, but they are all listed in the weewx wiki (where there is already a link to the WF UDP driver)…
ok - I’ll take a pass at the hardware guide and do the reviews over in their google group. Thanks.
I just did a little test where I dumped my most recent archive record and looked at the data field-by-field and it looks like weewx ‘is’ calculating out the stuff nicely per their defaults. Cool. No work for you on any of those !!!
Short answer is that it seeeeeems like the prefer_hardware settings really help a lot in making weewx hands-off, and the right things seem to happen if we just stick with the prefer_hardware settings that are the defaults and use rapid_wind for wind measurements.
[StdWXCalculate]
[[Calculations]]
# Derived quantities are calculated by this service. Possible values are:
# hardware - use the value provided by hardware
# software - use the value calculated by weewx
# prefer_hardware - use value provide by hardware if available,
# otherwise use value calculated by weewx
pressure = prefer_hardware
barometer = prefer_hardware
altimeter = prefer_hardware
windchill = prefer_hardware
heatindex = prefer_hardware
dewpoint = prefer_hardware
inDewpoint = prefer_hardware
rainRate = prefer_hardware
I’m also getting windGust and windGustDir automagically in my archive records as well, based on me using rapid_wind for the wind mappings…
I would tend to agree. I don’t run MT but I did get weewx-wd to work as well as the Saratoga templates using WD as the format for data and the Saratoga stuff seemed to work ok, which is usually very painful. To do so you need to run weewx-wd-1.0.3 then overlay a patched version of weewx-wd.py from Gary down in Oz. He had a thread in July HERE with where to get the patched code. Works great.
Running the weewx driver on a pi-zero and noticed gaps in my plots this morning. Looks like the Air disappeared from the saved data for a while after 4AM local time. The Sky logged normally to the weewx database during that period.
The Air was actually online all throughout according to querying the REST api: (url here)
Here’s the weewx data for the period. I tweaked my sqlite3 query a little to convert the dateTime to a more human-friendly output…
Looks like yes, but I can’t say what the reality is:
the Hub/Air/Sky might be not sending observations 100% of the time
the weewx driver might not hear 100% of what the Hub is sending
the pi-zero weewx is on might be too slow affecting the end-to-end result here
I know my UDP listener has drop-outs all the time with the hub specifically. Here’s some matching RSSI plots from a random time today. Note that the air+sky seem to disappear differently, but seeing ‘any’ observation tells me the hub sent ‘something’ it seems…today looks clean from a weewx standpoint but that might be just luck when the driver saw something on the wifi…
#----------- update - corrected errors in my original reply ------
This data is:
from influxdb, collected via telegraf by a pi3 with a SSD disk
the pi3 listens for MQTT data from a ‘different’ pi3
and ‘that’ pi3 gets its MQTT data from my UDP listener running on a pi zero
and ‘that’ zero also runs weewx using your driver
So it’s probably a bit convoluted to say the least. Uncertain if my hacked up setup that feeds influxdb can cause the end-to-end to miss messages or not. I could probably clean things up to have more powerful gear and less gear involved I suppose…
Anywhere you see || is a missing element in the archive record. If it’s right after the decoded timestamp it’s a missing outTemp (air missing), and if it’s right before the dateTime at the end it’s a missing windSpeed (sky missing).
A quick glance shows a lot of missing Air, and just a few missing Sky (but non-zero).
Definitely uncertain that there is a network problem involved in this at this point.
the half-dozen raspi and nodeMCU arduinos that I have on the same 2.4 GHz network send temperature data on 15-sec intervals through the same ‘publish to MQTT’ and ‘influxdb box subscribes to those topics’ mechanism and have 100% success getting all expected observations into influxdb through the various raspi involved.
at this time, I can’t explain why my UDP listener publishing to MQTT is resulting in missed observations in influxdb. I don’t have wireshark captures or independently saved time-series data to know for sure.
Weewx not recording anything for a 5-minute archive period while my listener data did ‘not’ disappear for the same 5-minute archive period makes me ‘lean’ toward it being a weewx driver issue related to how it deals with partial info when assembling the saved archive record, but I’m not absolutely certain there, nor do I have any logfile proof one way or the other.
But I’m willing to tear my existing bunch’o’raspi all down and set up a one-box gigabit-ethernet i5 NUC if it’ll help answer the question. @vreihen what did you think about the following as an approach ?
run weewx only on a hard-wired pi3+
run my listener and influxdb on a hard-wired i5 Intel NUC with 16 GB RAM and fast SSD
verify influxdb shows my listener does or doesn’t capture every observation
verify weewx does or doesn’t have 100% complete archive records
I forgot to mention that I ‘am’ still seeing radio reboot stat counts increment. I was up to 12 reboots on v94 and it’s up to 4 now on v98 already. I didn’t try to correlate when those happened vs. when expected observations were missed, but I do have that data if I get bored enough to go digging there just in case…
[update - got bored enough - no radio reboot and no firmware update corresponds to the possible lost observation data]
I wonder if this isn’t being caused by an archive/buffering issue that overflows the stateless UDP sending buffered data from the time that the radio was down? WF’s web API is fed via TCP AFAIK, so there’s a handshake and packet integrity verification involved on that end…
I didn’t see any correlation at all to radio reboots in a quick check yesterday, but I can post the data when I get home from work this evening. I have the last 30 days of UDP data in influxdb, so it caught the v98 firmware update and the three radio reboots since then up to last evening.
Quick check seems to show WF loses either air or sky for a one-minute measurement once a day perhaps, but I only checked a few days back.
Trying to take the mess’o’pi out of the equation, I put all the mqtt/telegraf/influxdb setup plus my UDP listener on a clean pi3+ this afternoon, so it’s now a one-pi3+ solution listening+saving the data to a time-series db.
I’ll let my listener do its thing for a few days as a first check to see if it catches everything expected from the Hub. If so, I’ll add weewx onto the same pi3+ and see what happens.
After 4-5 hours up, the pi3+ single-computer listen/mqtt/telegraf/influxdb setup is far better at catching everything, but comparing the two sets of results, I ‘do’ see at least one record captured on the multi-pi setup that was missed by the one-pi3+ setup for hub RSSI. Ugh.
I’ll have to do the i5 Intel NUC thing to throw some serious horsepower at the UDP listen+save compute load.
@GaryFunk - what hardware are you using to run your listen+save solution on that you’re saying isn’t missing any records ever basically ?
Just a RPi 3B+ with fast SDCard. I also use NodeJS as it runs in non-blocking mode and can write several packets at once. That’s the main reason I switched from Python to NodeJS.
I also use several scripts so as to distribute the work.
I will do this but it will need wait. I’m currently testing and constantly restarting the archive script.
Edit: I can pick a period from last night until I started testing and give you that number.
The image is form the database showing the number of packets received from each device. In a perfect world it is 1440 per day. But the device clocks are a little fast and sometimes send 1441
Just an update - I checked my python code and it is indeed set to the default of ‘blocking’ so it is reasonable that my listener might miss a UDP message occasionally. Also explains why the percentages got much closer to 100% when I moved things to a much faster unloaded-up pi3+ from the relatively loaded slower zero.
I haven’t checked the weewx driver to see whether it’s blocking or not, nor have I updated my stuff to go explicitly non-blocking. The sun is out here for a couple days and that’s pretty rare for the Seattle area this time of year…
