I wasn’t sure the first time if it was necessarily due to the firmware update. But since my replacement Tempest did the same thing, it seems like it’s likely correlated.
Both my first tempest and my replacement tempest started with firmware 126. A little while after they were online, they updated to firmware 134. After the firmware update, the tempest restarted itself (as evidenced by the uptime), but after the update it stopped sending observations. In both cases (my first tempest and the replacement tempest), it stayed that way until I manually restarted the Tempest.
I climbed up on the roof, climbed another ladder, took the Tempest off its pole mount, switched it off for 30sec and then back on. Flashing green lights looked good, hit the button and it was green, so I put it back on. Climbed back down, checked my status and I was getting new observations again.
This was probably the 30th or so time in the past couple weeks I’ve had to climb up on the roof, climb a ladder, remove the Tempest and power cycle it to recover it from failed sensors. Which is why I really want the ability to remote reboot it … (posted it in feature request, it was merged with a similar request, and I’ve upvoted that…).
Really thankful they sent me a replacement, and really hoping the new one is better behaved and my ladders can go back in the garage for a while.
Darnit - the replacement looks like it might be doing what my original one was doing. The updates stopped again, and it’s reporting pressure, wind and uv failed. So I went up and pushed the button - red, as expected. Restarted it, hit the button, green, so put it back on the mount.
This was what I kept having to do with the first one, and then the UV sensor permanently failed - no amount of restarting recovered the UV sensor. So support sent me a new one.
I’m hoping this is just an initial startup thing and it’s going to settle down.
Did you have to restart it again? It is 21:48 and it has an uptime of only 43 minutes. One station I manage seemed to do the same thing, stop sending data, after the hub received firmware v144, and it has been fine ever since.
@gizmoev Yes, I had to restart it again. It showed pressure, wind and UV failing again and stopped sending any observations. I let it go like that for 1 hour to see if it would recover. It didn’t and then started to report it was offline. So I went up and restarted it again.
This is exactly how my first Tempest behaved. First it stopped sending updates after the f/w update from 126 to 134. After restarting, it worked for a couple hours but then showed sensor failures and stopped sending observations. Restart, works for a few hours, then sensor failures and stopped sending observations. Lather, rinse, repeat.
With the first Tempest, this every-couple-hours failure lasted for a couple days. Then the UV sensor went into permanent failure mode. After that, instead of the sensors failing and observations stopping every couple hours, it would go roughly 2 days between failures. Sometimes it would make it 3 days, but I think that’s the longest it went.
@GaryFunk OK, thanks for the clarification. That’s what I thought I had with this report. I’ve now had two Tempests which, after updating from firmware 126 to 134, stopped sending observations until I restarted the Tempest. That felt like a software/firmware issue not a h/w failure and was reproduced 2/2 times. Admittedly I later added issues which look more like h/w failures, but the initial report and title of the post seems like a bug not a h/w failure.
Is this coincidence or is the only common hardware item the hub and power supply? Try switching power supplies to see if things change. If not, maybe the hub is bad.
Well, I can’t answer whether it’s coincidence at this point - just that both Tempests have behaved the same so far. And yes, the only commonality at this point is the hub and power supply. I can certainly try a different power supply for the hub - I have several 3A USB supplies.
Note that the hub looks like it upgraded to firmware 147 last night.
Also note, FWIW, I’ve had to restart the Tempest twice this morning.
@gizmoev question for you, if you happen to know. While I haven’t yet tried it with this tempest, when the first tempest was acting up, before climbing up on the roof I often tried rebooting the hub, as well as doing the hub radio reset procedure. That never recovered proper operation - only rebooting the tempest would. That would seem to me to mostly exonerate the hub - or is there some way that the hub could possibly cause the tempest to lock up?
Also - should I break out these followup issues into another post? The initial reason for my post was to point out that it seemed like there was a problem with the firmware update procedure for the tempest which causes it to stop sending updates until it is manually restarted. All the posts after that are related to problems I’ve experienced after the firmware update (and which have occurred on both my first tempest and this replacement tempest).
@GaryFunk I’m not trying to be argumentative or difficult here, but it feels like you’re making some assumptions that may or may not be true. It may in fact have hit quite a few people, but the correlation hasn’t been made yet. Since the troubleshooting guide includes restarting the device, and many people would likely try that anyway - many people may have recovered themselves without reporting it, and wouldn’t even know there was a firmware update involved.
I only figured out it was a firmware update both times doing some reverse analysis and also because I had taken pictures of my status screen at various points so I knew what firmware it was running at various points. Then looking at all that information I figured out it exactly coincided with the firmware update.
It might also be related to the specific transition of v126 to v134, which might apply to a reasonable number of people, but not everyone. I’ve been in software development, specifically with embedded devices for many years, dealing with remote firmware updates, and these types of bugs are not unusual. I was really just trying to be helpful because for me personally it doesn’t really matter - my device has already made the update and I “recovered” after that. But I thought the developers might want to look into this more and/or put it in their internal knowledgebase to potentially look at when someone reports their device has stopped sending updates.
I’ve only had this happen once and that station had no issues updating the Tempest to v134. Only the apparent lockup of the Tempest after the hub updated to v144. The only other time I’ve had a Tempest loose connection with the hub was with a Field Test unit during field testing. I currently have one field test station with two Tempests and two production Tempest stations. I even had no trouble with a Tempest I had to temporarily move from a first generation hub to get its firmware updated. I know this is only a total of five Tempests from probably different production runs so not sure how representative this sample is.
Personally I think all of what you have posted should stay in this thread, at least until the issue is resolved.
Latest update, FWIW. This replacement tempest continues to follow a fairly similar string of behaviors as the first. Like the first one, after the first day or so of periodically stopping updates every couple hours and intermittent sensor failures, it settled down a bit. The replacement was up, and generally behaving OK for about 24 hours. Then a few hours ago I noticed a weird temperature reading (it was reading -40F when the actual temp was about 55F). Then shortly after that the pressure, wind and UV sensors failed (though oddly not the temp sensor). I restarted the tempest, which cleared the sensor failures, but the temp stayed very low for a while and gradually came back up. Attached is the temp graph
I worry that I’m including too many different things in a single post, especially unrelated to the original topic. But for now I guess I’ll keep things located here.
Another issue I discovered this morning. I was looking at my wind chart and couldn’t figure out why the scale seemed so weird. Then I scrolled back and found that last night it recorded a wind speed over 800MPH. Obviously not a real reading … It would seem like the AI or at least some type of bounds-checking would catch something like that. It’s also frustrating that now my wind charts are virtually unusable due to that anomalous reading.
I have no idea if this would trigger some of the behaviors I’ve been seeing, but I noticed something odd today when I was reviewing my network events. It seems the hub has been psychotically roaming between 4 of my APs. It was roaming roughly every 2 minutes, though sometimes it was as frequent as twice in a minute.
I’m actually surprised it could even reach a couple of the APs because I’ve tried to position them and set their radio levels low enough that things should be fairly “cellularized”. And the hub is literally sitting about 4 feet away from one of the APs. I’m somewhat baffled as to what would cause it to try and roam at all, let alone roam at that frantic rate.
To try and tame the hub I just created a new SSID just for the hub and assigned it only to the AP that’s 4’ away from it. We’ll see if that improves things at all.
FWIW, I know that some devices don’t like when you share an SSID between 2G and 5G, so I have always had separate SSIDs for my 2G and 5G networks. And as mentioned, I’ve worked hard at setting power levels and channel frequencies to cellularize the system as much as possible. Also note that the equipment I use is from Ubiquiti. I can list the exact equipment and configs if it would help.
I have a lot of IoT and other devices with cheap networking, and this is the first time I’ve seen behavior like this.
Well, it was up and seemed-to-be-mostly-working for almost 2 days. Then it reported multiple sensor failures and stopped sending observations. I went back up on the roof for the umpteenth time and restarted the Tempest. After that the sensors reported OK, but the wind is clearly wrong. We have a fairly consistent, moderate wind right now - roughly 10mph from the West. But the Tempest is consistently reporting that it’s coming from the NE.
I tried rotating the tempest 180deg but it continued to report the wind coming from the NE. I tried rotating it a few other directions but it always reports the wind from the NE.
The sensors are generally showing up as “OK” but occasionally it reports a wind failure. I’ve been seeing that on-and-off for a while and have mostly been ignoring it because it seemed to be properly reporting the wind (at least best I could tell). But no more.
Man, I have spent sooooo much time on this device between the first one and this replacement. And I’ve completely lost count of how many dozens of times I’ve had to go up on the roof to restart it. This has become extraordinarily frustrating.
Obviously not a real reading … It would seem like the AI or at least some type of bounds-checking would catch something like that. It’s also frustrating that now my wind charts are virtually unusable due to that anomalous reading.