API/Websocket/App rain discrepancy

@corrineb, I am trying to reconcile a discrepancy in rainfall amounts from yesterday given by an API call today, what is shown in the app history for yesterday, and what was collected live over the websocket yesterday.

Both the history in the App and my Weewx database that was listening to the websocket show a total rainfall of 28.3 mm yesterday, however an API call this morning gives a total of 27.4 mm. I cannot identify what is causing the difference.

The API call I am using is: https://swd.weatherflow.com/swd/rest/observations/device/16392?time_start=1597532400&time_end=1597618800&api_key=20c70eae-e62f-4d3b-b3a4-8586e90f3ac8. We are currently on British Summer Time, so the start and end times are 23:00 UTC 08/15/2020 until 23:00 UTC 08/16/2020.

if you shift your api call by exactly one hour, does it match the app values?

Good question! Let me check…

Edit: Nope, that doesn’t help. Shifting forward by 1 hour gives 27.3 mm. Shifting back (just for completeness) gives 27.8 mm

1 Like

you do have one measurement too much in your query (deduct one second on the end time), but that only would make the difference worse if you had rain in the last minute. So that doesn’t help.

what happens if you plot the difference between the two? Is it one value that is way off?

something went wrong at epoch 1597591020 ( Sun, 16 Aug 2020 15:17:00 UTC). the measured amount of rain in that minute is 0.6177 (equiv of 37.06 mm/hour) but the accumulated value increased by 1.503 an extra increase of 0.885. All the other values are spot on. So, it looks like the accumulated value changed unexpectedly.
Here is the data from the api:
1597590900 3589 0.07 0.933425 0 0.98 3250 3.51 1 29 24.528226 3 3 null null 0
1597590960 4175 0.09 0.841153 0 0.42 1.83 214 3.51 1 34 25.369379 3 3 null null 0
1597591020 5013 0.13 0.617782 0 1.07 3.08 234 3.51 1 41 26.872376 3 3 null null 0
1597591080 5666 0.15 0.275281 0 0.92 2.46 225 3.51 1 47 27.147657 1 3 null null 0
The difference between the values in the column for accumulated rain (the 11th column, starting with 24.528226) is:
0.933 (from the row above, but not shown)
0.841
1.503
0.275
Which match the reported values for rain in the last minute (rain rate) in the 4th column except for the one at epoch 1597591029 where the rain rate reports 0.618 instead of 1.503.
Pretty strange. There isn’t anything more I can find so it is now up to @WFstaff to find the reason why.

2 Likes

Thanks for looking into all the detail @sunny! I am supposed to be working from home, but this difference caught my eye and I wasn’t really going to look into it further until this evening.

Given that everything was matching yesterday (app, weewx, PiConsole; all using the websocket) it seems instinctively (although I could be wrong) that it is the rain rate that has been stored in the API that is wrong rather than the accumulation. The rain rate in the original websocket messages from yesterday must have been correct to match the accumulation given in the API today. Let’s see what the WF team can find out.

That’s also a good point. I should get that fixed in the PiConsole.

@peter I will take a closer look at the data. What index are you using in the obs array for your calculations? I want to make sure to rule out any RainCheck issues.

Thanks @corrineb. I am using index 3 from the SKY API calls. I am just doing a simple sum over all values that are not equal to None.

Thanks for the extra info @peter. I just wanted to make sure that RainCheck was not a factor. I did the same thing that you did and I am noticing the same discrepancy. I did a quick quick check of the 15th and those numbers look good to me. We are not sure of the exact cause of the issue of the 16th, but we will continue to look into it.

1 Like

Thanks @corrineb. It’ll be interesting to hear what you find

@peter We re-ran the bucket generation script, which also recalculates the rain accumulation, for 8/16. We are not sure of the root cause of the issue, but it should be resolved now.

Looks good to me. The accumulated value (index 11) and the sum of all the individual 1-minute rain accumulation values (index 3) now match.

It is interesting though that whatever caused this glitch also caused the 1-minute rain accumulation value in the websocket to be incorrect. I think the error happened at epoch 1597591020 (based on @sunny’s analysis), but I can’t be sure

Hi @corrineb, I’m not sure if whatever caused the issue discussed above has happened again, but today I noticed another discrepancy between the different rainfall totals. This time both the PiConsole and my Weewx database listening to the Websocket recorded a total of 33.1 mm of rain yesterday, but today the app/web history and API calls only give a total of 32.2 mm for yesterday. I can’t be 100% sure when they diverged as I wasn’t watching all day, but I am pretty sure they were agreeing perfectly earlier in the day.

Thanks for the report @peter. We will take a closer look at the data and will let you know what we find.

1 Like

Thanks @corrineb. Sorry the report is a bit vague! Perhaps the websocket kicked out a bad value, but I don’t log these permanently so can’t be completely sure.

@peter We took a look at the 1-minute data and the accumulation totals and they all look correct. Are you using index 11 of the ob returned by the websocket for the accumulation? And then you are looking at the timestamp to know when a new day starts or are you using some other logic?

I am going to start a script that we can run over time to double-check the accumulation values, but I want to understand your process before I kick it off.

So the PiConsole uses index 3 of the ob returned by the websocket and calculates the accumulation by keeping track of the sum of each of the individual 1-minute rain accumulation values. I keep full precision at all times so there is no rounding in the calculation. I then use the local time of the console (rather than the timestamp of the observation) to determine when a new day starts. I know I need to fix this, as it can cause a single observation to be attributed to the wrong day if the observations is taken just before midnight but arrives at the console just after midnight. This doesn’t happen that often though and the impact is small as it is only a single observation. I am pretty sure Weewx takes exactly the same approach.

I don’t want to start a wild goose chase, but it is the fact that both Weewx and the PiConsole (fully independent of each other) gave identical accumulations yesterday, and they are both greater than that given in an API call today. If they had missed websocket messages then the total would have been different, but it would have been less. I can only imagine that a bad value came from the websocket, causing the values to diverge. I will also keep a close eye on this and hopefully we can spot the culprit at some point!

Thanks for the extra info @peter. Do you have any logging on if you lost your websocket connection and reconnected at some point during the day? When a websocket connection is opened, we send back the last observation we have received from the station. This is so you have observation information immediately and you do not need to wait for the station to send in a new observation. This is noted in the websocket message when the source is cache. If you lost your connection and then reconnected there is a chance that you may have double-counted an observation as we may have sent you an observation you have already received over websocket before you disconnected.

Index 11 in the SKY obs array provides accumulation information for the local day. You can use this value at the end of the day to double-check your rain totals to what the API has. Also, using this value instead of doing your own calculations will help if your websocket connection is dropped for a period of time and you happen to miss observations from the station.

1 Like

I can’t see any obvious disconnections in my logs, but it is certainly possible that this may be the issue. I hadn’t quite appreciated that it is possible to get the same observation twice when connection to the Websocket. I probably should have guessed though, given that an observation is always sent when the connection is opened.

This is a good idea and something I will implement. I’m not sure this index existed when I first put the console together, or if I did I just completely missed it. I’ll make the changes and keep and eye on things to see if I see the same behaviour again.

1 Like