System hiccup today

Hi folks. We had a problem today that started around 2:40 PM ET with an unusual CPU jump on a primary processing server cluster. We’ve been working to identify and correct the issue since then, and things should be relatively stable now, but you may notice issues with some components today, including:

  • Graphs missing data.
  • Lightning Strikes missing data.
  • All notifications types not sent.
  • Data not sent to WU.
  • IFTTT Real Time System not triggering events.

Other systems may have been affected too. Let us know if you see anything else. We’re continuing to work the issues now. Thanks for your patience!

19 Likes

Growing pains are rough, David. But necessary. Everyone at WeatherFlow is doing a tremendous job! Proud to be part of the WeatherFlow Family.

13 Likes

A post was merged into an existing topic: Missing Data

Thank you for keeping us updated.

2 Likes

To follow up on this, we believe we’ve identified the source of the problem and have taken measures to make sure it doesn’t happen again. You may notice some small gaps in graph data, but otherwise things should be completely back to normal. Please let us know if you notice anything out of the ordinary.

6 Likes

A post was merged into an existing topic: Weather Underground Failure?

Would this cause the “Online” link to disappear off the grid page as well?

I was clicking it pretty regular to check my Tempest firmware, and suddenly it was gone. I just now checked again and it was back, which if my station was offline it would make sense it would disappear.

I’ve since bookmarked the status page it links to, so I’m all set, just curious.

Today, I was watching a storm go through PCB and if my memory serves me right, I saw the wind gust up to 28mph. But on the station’s historical records, it’s showing 25.6mph as the PWG. I think it did the same thing on Monday when I saw a 57mph wind gust (if my memory serves me right) but it only archived 54.9mph PWG. Station ID is 24126

If your station is offline, the station information will still be displayed at the bottom of the page. But instead of a green dot it will be a red dot and it will say Offline.

I suspect what happened when we had a hiccup is that the web app did not get a response from the server when asking for the diagnostic info for your station. With no response, the safest thing to do is to not display the status information.

Also, if you are viewing a share page you do not see the status link at the bottom of the page. This could potentially happen if you get signed out and go to your station URL. However, given that this was happening during our hiccup I doubt that was the case.

My WU station is showing offline for the past few hours. It’s been working very well for the past month or so. Is the Tempest to WU connection down again ?

4 Likes

Yes, we had another hiccup last night that was very similar to the first. The good news is the instrumentation we put in place after the first one helped us see where the core issue lies, and we’re working toward a better solution.

8 Likes

Ive been having troubles too, as soon as i hooked up my Tempest hub, been getting knocked off. Have my Sky/AIR hub and Tempest Hub working

My Tempest to Wunderground feed was down overnight until 5:30am ET this morning.

It seems like y’all might need some better auto scaling or just some honkin server iron.

This has been happening occasionally this past week. My Status page is completely blank, and the “Online” link to the status is missing from the grid. When the online link isn’t there, a map shows up instead (the map does not normally show up).

I thought I was getting the firmware update, but it’s back to normal now I’m still at 1.29

I’m using Microsoft Edge Version 84.0.522.49 (Official build) (64-bit) browser in case it’s a rendering thing.

@atomicinternet The map showing is a good clue that you are looking at your share page instead of being signed in and looking at your station.

What does the header on that page look like?

If it looks like the image below you are not signed in.

If it looks like the image below here and you have a My Station option in the header then you are signed in to a different Tempest account that does not own the station you are viewing.

Yep, that was it. Thanks!

I opened a private tab and saw the map/no status. Once I went to the logged in tab it was normal.