Obsolete : New Tempest firmware: v134

I can confirm the issue is heavily investigated on several fronts to try to pinpoint what or where the issue is causing the reboot. Not to many stations and not on a regular scheme … nice challenge for the engineers :grin:
You’ll know when it is found and solved and again thanks to all having the issue to be patient.

6 Likes

Thanks for the update! :slightly_smiling_face:

Thanks, @eric. And thanks to all of you how have noticed this for your vigilance and attention to detail!

Yes, we have been actively investigating this issue for a while now and we are close to a resolution. Please hold off reporting this particular issue to support for now. We will begin testing a new firmware build tomorrow. Meanwhile, here are a couple points to note:

  1. A small number of reboots is perfectly normal. There are dozens of software watchdogs in the code and they can be pretty sensitive.

  2. Sometimes a watchdog can be a bit overzealous. But it’s better to err on the side of over-zealousness when it comes to hardware watchdogs (lest you need to physically power-cycle your Tempest).

  3. You won’t notice most reboots in the data because they happen so quickly (as quickly as 4 seconds) unless you are looking at the uptime.

  4. An initial analysis of device status data shows that the number of Tempests rebooting more than once every few days is a very small. We are in the process of anlyzing more data to look at longer periods of time and across firmware versions to fully characterize the issue.

  5. The latest firmware candidate (which attempts to slightly and carefully reduce the sensitivity of some of the watchdogs) has been running since Monday on a small number of units and not a single one has rebooted. But the nubmer of units is too small to say whether it’s “better” than the current build.

Finally, we have received @GaryFunk’s Tempest, which was one of the first identified to be rebooting more than it should. However, we have not “torn it down” yet for one simple and frustrating reason: it’s been running flawlessly… we can’t get it to reboot.

Thanks again for your support and patience.

9 Likes

Thanks for the excellent update! I just returned my unit that was rebooting everyday. Hopefully it will be in your hands in a few days. I appreciate your excellent customer service.

2 Likes

May I suggest you send it back to me and let’s see if it remains stable to starts rebooting? I’m not convinced that the issue is not environmental.

What environmental scenarios do you think could be triggering the reboots?

Heat, EMI or RFI…

Here’s another one - I checked my FT Tempest…note the fun started when it received firmware 134.

vince@nuc2:~$ influx -precision rfc3339
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 1.7.3
InfluxDB shell version: 1.1.1
> use testdb
Using database testdb
> select * from device_status where time > now() - 60d and uptime < 65;
name: device_status
time			debug	device	firmware_revision	hub_rssi	rssi	sensor_status	timestamp	uptime	voltage
----			-----	------	-----------------	--------	----	-------------	---------	------	-------
2020-07-06T20:11:48Z	0	tempest	129			-127		-127	0		1594066308	44	2.57
2020-07-14T12:11:23Z	0	tempest	129			-127		-127	0		1594728683	6	2.6
2020-08-05T12:15:19Z	0	tempest	134			-127		-127	0		1596629719	13	2.6
2020-08-06T15:48:43Z	0	tempest	134			-127		-127	0		1596728923	8	2.56
2020-08-07T07:35:24Z	0	tempest	134			-127		-127	0		1596785724	1	2.57
2020-08-07T18:09:42Z	0	tempest	134			-127		-127	0		1596823782	1	2.58
2020-08-08T05:49:08Z	0	tempest	134			-127		-127	0		1596865748	1	2.58
2020-08-08T23:44:54Z	0	tempest	134			-127		-127	0		1596930294	1	2.59
2020-08-09T09:03:24Z	0	tempest	134			-127		-127	0		1596963804	1	2.56
2020-08-09T17:32:05Z	0	tempest	134			-127		-127	0		1596994325	7	2.56
2020-08-10T02:18:48Z	0	tempest	134			-127		-127	0		1597025928	1	2.55
2020-08-10T11:48:01Z	0	tempest	134			-127		-127	0		1597060081	1	2.55
2020-08-10T20:25:10Z	0	tempest	134			-127		-127	0		1597091110	1	2.55
2020-08-11T04:11:08Z	0	tempest	134			-127		-127	0		1597119068	1	2.55
2020-08-11T21:16:56Z	0	tempest	134			-127		-127	0		1597180616	1	2.55
2020-08-12T07:55:15Z	0	tempest	134			-127		-127	0		1597218915	1	2.55
2020-08-12T16:52:16Z	0	tempest	134			-127		-127	0		1597251136	1	2.54
2020-08-13T12:38:43Z	0	tempest	134			-127		-127	0		1597322323	1	2.55
2020-08-14T00:18:00Z	0	tempest	134			-127		-127	0		1597364280	1	2.61
2020-08-14T12:30:48Z	0	tempest	134			-127		-127	0		1597408248	1	2.59
2020-08-15T13:37:41Z	0	tempest	134			-127		-127	0		1597498661	1	2.59
2020-08-18T03:35:45Z	0	tempest	134			-127		-127	0		1597721745	1	2.61
2020-08-19T05:35:52Z	0	tempest	134			-127		-127	0		1597815352	1	2.59
2020-08-20T04:50:36Z	0	tempest	134			-127		-127	0		1597899036	1	2.58
2020-08-20T23:46:57Z	0	tempest	134			-127		-127	0		1597967217	1	2.6
2 Likes

Thankfully almost 14 d without a reboot

Follow the bouncing Tempest…scale on left is uptime…

2 Likes

I suggested that the Tempest be returned to me to see if it continues rebooting. I’m almost convinced it’s an environmental issue. Probably from SETI talking to the aliens.

2 Likes

My Tempest Sensor (Unit #3) made it almost 4 days before it rebooted. Longest uptime since v134 was pushed down.

1 Like

Mine reboots too. But I’m having zero issues and I wouldn’t even know unless I looked at the uptime.

1 Like

I’m still using my first generation hub so my new Tempest is still on 129. Rock solid with no reboots so I’m in no hurry to upgrade to the new hub. That can just stay in the box for now. Station 1565 - keeping that four digit ID is also one reason I haven’t installed the new hub.

1 Like

So I guess there is no update on the reboot issue ?

1 Like

No real update. Some progress made, but no breakthroughs. Remember though, even a reboot or two a day is not something to be concerned with and won’t affect the data in any tangible way.

Starting this week (I believe I’ve had the update now for a few weeks), my station will stop sending data in the AM (usually around 7-9am). I will get an email stating " “Your WeatherFlow Tempest device named ST-00001980 stopped sending data. Check that it has power and/or try moving your Tempest closer to your WeatherFlow hub.”
After about 30 minutes It will resume sending data.
Submitted a ticket a few days ago, waiting to hear back on how to troubleshoot or if this is known issue. Running latest FW on HUM and Tempest.

It looks like your hub is having the issue. The RSSI value is not too great. Can you move it so it has a stronger signal to your Wi-Fi? While you’re at it, see if you can also improve the RSSI to the Tempest sensor as well. It is only marginally better.

It’s back…

1229 has rebooted 11 times in 20 hours.

Must be EMF you give off, @GaryFunk. :rofl:

1 Like