Dropped samples on June 15th

I have noticed over the last week or so that my station has been regularly dropping samples. It was particularly bad Monday morning (June 15th), and looking at the data I can see that the samples were being dropped with alarming regularity on the hour during the small hours of the morning:

I see the same in all three devices attached to my station (so it is not device specific). At other times the behaviour is less regular, but I still see drops of 3-4 minutes:

I have checked the log on my router and I can’t see any evidence of the internet going down, and even if it did there is no evidence of the hub backfilling the data. The RSSI values are all around -50. It seems like the hub is sending the observations, but they are not being received by the WF servers. Is there any suggested troubleshooting, or any debugging that can be done on the WF end?

1 Like

I have observed the eXact same phenomena. I don’t always have a chance to look at the Hubs… because they are behind me. I’d have to type side-ways in order to view them continuously…while still working at the keyboard. I’ve notice a greater number of data drop-outs on Hub 4889 which runs for both WFSKY and WFAIR. Per your examples above. . .these last anywhere from 4-6 mins at a time. There are some drop-out on Hub 3503 which runs for Tempest Field Test…but not as frequent as the other Hub. However, those drop-outs on 3503 can also last anywhere from 4-6 mins.

Sometimes…I am lucky enough to turn around and notice that the Green LED on one (or both) of the Hubs has turned Red for a period of time before returning to Green.

Interesting - glad my station is not the only one! Do you see the gaps in your data as well like I do?

What I don’t understand in my case is why the data is not getting backfilled. If the hub cannot connect to the WF servers for whatever reason, it is designed to buffer the observations before sending them to the WF servers once the connection is back up. It seems like either the hub is sending the observations and the issue is at the WF end, or the backfilling is not operating correctly.

Either way, it seems quite suspicions that the hub is dropping offline for almost the same length of time every hour (in my first example).

yes Peter, the gaps in data are identical to yours. Here is one example from earlier this morning…

The gap is from 4:28am CDT to 4:34am CDT.

1 Like

Interesting - it looks almost identical! Hopefully someone from WF will be along to shed some light on this.

1 Like

added Footnote: I got my 1st Gen WF SmartWeatherStation back in October of last year. It didn’t used to be that way. The features identified above only started in earnest. . .about 4 months ago. Before that. . .the gaps were Very Isolated and hardly ever occurred. . .at best.

I would agree with this statement completely. This is a relatively new behaviour that I didn’t use to see (except for the odd drop here and there).

1 Like

I see same kind of gap with my SKY and Tempest on June 15th from 12:43pm to 12:49pm.

1 Like

Seeing gaps with my SKY and Tempest as well… both at the same time…


1 Like

This is indeed a Puzzlement! Hopefully WF Staff will analyze / diagnose the situation to determine why all-of-the-sudden this is increasingly occurring.

“I’m a meteorologist. . .not a software or hardware engineer”. . . .but thinking about his just now. . . this new WFSKY and WFAIR issue only came about in the last several months. . . .coinciding with the switch over from “smartweather.weatherflow.com” server to the “tempestwx.com” server.

1 Like

As another data point with 3 stations running on 1 home network (3 Hubs, 2 with Tempests and 1 with Airs/Skys). Some data drops confirmed at the same time period on 2 stations (1 Tempest, Air/Sky) while the 3rd (FT Tempest) had no data issues in the same period. Perhaps a clue that 1 of 3 stations did not see the issue at the same time.

Drop Tempest


Drop Sky

No Drop FT Tempest

2 Likes

All three of my stations show the same periodic gaps in the data. All devices on a given hub have the same gap. Two stations have one SKY and two AIRs and the field test station has two Tempests. The gaps are not all at the same time between stations.

1 Like

I just looked at my Tempest and see the same thing happening. This morning there is a gap between 2:28 AM and 2:34 AM.

1 Like

I am seeing the same behavior on my Tempest and I had started a topic about it a couple of days ago. This is the topic: WeatherFlow Servers Having Issues or Is It My Tempest?

Now that several of us are seeing the same issue it may get more attention. I’m betting it’s a server issue.

1 Like

This looks very much like a server issue. It could be caused by the WeatherFlow servers or by the provider.

I’m sure @dsj will have an answer soon.

3 Likes


Acknowledged! 4:28am CDT to 4:34am CDT.

1 Like

Just a quick note that I have seen the same behaviour multiple times this afternoon. For the drop between 17:49 and 17:53 BST I was on a video conference so am 100% sure my internet connection did not fail (although that was using a wired connection so it is possible the WiFi went down, although it seems unlikely that the WiFi went down three times). I assume this is now being looked into, so I won’t keep posting every time it happens.

1 Like

Mine was far worse this morning!
image

All devices dropped from 12:10AM to 6:05AM when I noticed they were all offline. Rebooted hub and they all came back online and back-filled from 3:15AM (so I am missing 12:10AM - 3:15AM).

Internet did not drop, WiFi did not drop, no electrical storms at the time :man_shrugging:

Same here…

Thanks for the reports, everyone. We have been investigating and believe there were two issues here, one causing the one-minute gaps and another affecting the higher zoom levels (5-minute and up). Both issues are related to some of the components of the Tempest Cloud not scaling quickly enough to keep up with the growth of the network. Both issues should be resolved later today.

12 Likes