After an internet outage, no back filling

,

Last night our internet was down (external reason), wifi was always up and udp packages arrived ok in weewx etc
But the hub wasn’t able to talk to the WF servers … this morning I pushed a bit and internet came back.
Hub talks via wifi to a router that is linked to internet router, no direct talking between hub and internet router.

Since this morning the hub hasn’t or couldn’t send back the (stored, hopefully) data to servers …

production units from unit 2230

As said I have not rebooted hub or air, I just let it run since and it is sending data to WF servers since the internet came back. I leave as is till some @staff member tells me what to do … reboot, follow some special procedure or whatever to help debug.

I will open a second topic for the problem in graphs zoomed in or not showing other time lapse

46
19

2 Likes

Thanks, @eric. We’ll investigate. Leave everything as-is for now.

2 Likes

@eric, please reboot your hub and let’s see if your data backfills. If not, we’re going to need to add more debugging code to a future firmware build to figure out what’s going on.

1 Like

ok rebooted just now, took quite some time to come to green … let’s wait 10 mins to see if it can send the data

1 Like

10 mins passed and it didn’t back fill

If you want to make a special version for the hub, ok to limit it at first to me, I can simulate what happened, maybe no need to disturb other testers for now as they are testing the sky ?

let me know

2 Likes

Something went wrong, Eric. But we’re not sure what :slight_smile:

We have tested this case many times in-house. And we can see lots of backfilling data coming in, So I don’t think we have a systemic issue.

It could be your hub didn’t detect the outage and marked the data as already sent. It could be the hub didn’t correctly start the backfill process after your internet connection came back online, and then lost track. It could even be the flash on your Hub isn’t working correctly.

We’re going to add more debug info to a future firmware release so we can better troubleshoot these issues when they happen. Stay tuned…

5 Likes

Interestingly, I had the same issue this morning. I was at work, but I believe we had an internet outage, and I now have a gap in my graphs. I will reboot the hub when I get home.

2 Likes

This has happened to me twice upon a power outage and/or internet outage.

2 Likes

Thanks, @dave & @nwwebb - we are still unable to reproduce this issue in-house, but we’ll be adding more debug instrumentation to a future firmware build and hopefully figure out why it’s happening.

2 Likes

I’ll try to simulate this again most probably on Sunday and see if I can find some reason …

maybe @dave and @nwwebb can already share the setup … are your hubs directly to the internet router or is there another router in the middle that might somehow fool the hub ?
Do you use some UDP listening device in the setup ?

@dsj : I guess when the hub sends it data to the (amazon) servers it expects an aka after each packet … if so when can safely assume the hub knows pretty immediately if it is not heard … and activating or at least not deleting the buffer …

and this is for your network guru, when doing a tcpdump you see every x lines an arp request from my internet router

tcpdump, click to open the arrow

yoyos-Mac-Pro:~ yoyo$ sudo tcpdump host 192.168.1.176
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pktap, link-type PKTAP (Packet Tap), capture size 262144 bytes
20:52:24.785693 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 197
20:52:34.780796 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 197
20:52:44.780578 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 197
20:52:45.983855 ARP, Request who-has 192.168.1.176 tell 192.168.1.1, length 46
20:52:52.543159 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 208
20:52:52.646340 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 142
20:52:54.785369 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 197
20:53:04.779420 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 197
20:53:14.778204 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 197
20:53:24.778330 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 197
20:53:34.778071 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 197
20:53:44.778111 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 197
20:53:47.559716 ARP, Request who-has 192.168.1.176 tell 192.168.1.1, length 46
20:53:52.696870 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 142
20:53:54.777947 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 197
20:54:04.778036 IP 192.168.1.176.50222 > broadcasthost.50222: UDP, length 197

know that this router is not master in my network as 192.168.1.2 is the one having the routing table and to which every device reports (wifi included). 192.168.1.1 is set to forward all to 192.168.1.2 (set dmz). not sure this can help solve the mystery.

edit : other thing popping up in my mind, here in Europe all routers are slowly but surely adopting ipv6 also for internal network. Could this mess somehow ?I know that in recent past this could send wacko older devices as they got ipv6 addresses not knowing what to do with it ?

1 Like

Hub is connected to a TP-Link WiFi Powerline extender, which is connected to another PowerLine extender and then hardwired into the Netgear router. I’m upgrading my kit to Ubiquti soon, so all that will be replaced.

I have a Raspberry Pi3B+ running Home Assistant and my custom Weatherflow sensor, which is monitoring the UDP data.

There was no gap in the UDP data when the Internet outage occurred.

Hub is running firmware 37
Air is running firmware 20

I still have not yet rebooted the hub, but I am happy to do so to see if it back fills if required, just let me know.

1 Like

Thanks for the extra info, folks. Please let me know if you have a test case that reliably reproduces the issue for you. Something like “disconnect WiFi router/extender from upstream network, wait 30 minutes, reconnect”.

1 Like

I can reproduce the issue any time you need. My hub never comes back up after internet
outage without being reset, and it has never back filled. Station 3132.

1 Like

David,

Look at 1878 at 05:00 your time. I kill power to my router everyday. It seems I have a small hole everyday at this same time.

The “not coming back up after internet outage” issue is separate from the “no back fill after internet outage” issue, although the former would also cause the latter.

We had an issue with hubs not reconnecting after loss of internet, but i thought it was resolved well before the v37 firmware. Please open a new ticket in #owners:bug-reports so we can track it separately. Thanks!

1 Like

At 14:00 my internet went down and didn’t come back up until I rebooted my cable modem at 15:55. Both hubs had red lights on them but my internal network was working find and the access points were all on. An hour and a half later data still hasn’t backfilled. Should I reboot the hubs? This is on stations 5075 and 5080.

don’t reboot as you loose the data but you have to zoom in to max to see the data. At present the buckets on higher averaged levels are not back calculated. The data is probably there but not on all levels. The script has not yet been altered to take in account an outage … how far it has to go back and redo the maths.

1 Like

I just zoomed in and didn’t see any of the data. I see a couple of other gaps where my modem was rebooted and reprovisioned to hopefully fix the intermittent network issues I’ve been having.

The bummer is that today was the first real significant rain event and the connection died right at the start. Oh well, there will be another one on the way.