Lower Tempest Bandwidth Capability (either always or at least by API configuration)

Tempest Owners Feature Requests
21

This is a great subject; “monitoring weather station with limited bandwidth” (OP’s report).

The conversation in the thread toggles between “not enough bandwidth” and “not enough throughput”. Each have different solutions.

Either way, the weather station itself has no bearing.

You are describing a networking, or data cleansing issue that is handled beyond the weather station.

Simplified solutions include:

  • LoRa for long distance data transmission.
  • Satellite
  • Caching
  • Storing-and-forwarding data.

This is a common issue and has real solutions, just needs to be stratified more specifically (e.g. maybe to don’t need ALL telemetry, maybe some telemetry is more important that other, etc.).

22

I got back to this after being away for a while.

Along the way I ran into a number of issues but I think I am pretty close to a working solution now. But I do have a question that maybe some developers can help out with.

The general approach is:
a) capture all traffic coming from or going to the hub
b) for dns responses as to where is mqtt.weatherflow.com, I put in my own IP address (and bump up the ttl value fwiw)
c) I capture all tcp hub data. If any of that data is heading outside my subnet, I drop it (with the dns rewrite most data should now be coming to my own IP)
d) I created an mqtt proxy that listens on port 1883 and gets all the traffic that was originally going to mqtt.weatherflow.com. I process the mqtt traffic to decide what to drop and keep. For what I want to drop I still send the necessary responses back to the hub as if they had gone to mqtt.weatherflow.com. For stuff I do want to send on I create my own link to mqtt.weatherflow.com and send the data on its merry way and then forward the responses back to the hub

In general this works great and my bandwidth is down to about 50MB/month which works for me. Furthermore, I believe WF gets all the relevant data they can benefit from including quick changes in wind direction etc. Clearly other filtering approaches could be employed to further reduce the data but I am trying to get WF to benefit from the weather station too.

If anyone wants details as to how to get this done you can email or call.

The issue I am having is that I am still getting more mqtt disconnects (and new dns requests and mqtt subscribes etc) between the hub and my mqtt server. Generally the drop/disconnect happens right after the hub sends an mqtt ping request even though I immediately respond with an mqtt pingres. The log below shows a few seconds of packets that my mqtt proxy is receiving and what it is keeping and dropping. Towards the bottom you can see the pingreq and then dns traffic. Any help as to what makes the hub want to generate a pingreq (and ignore the pingres) is GREATLY appreciated.

2023-09-06 12:11:34,848 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:34,862 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:37,821 - irricloud - INFO - mqtt publish kept: rapid/ST-00107879 qos: 0 kept: [b'{"serial_number":"ST-00107879","type":"rapid_wind","hub_sn":"HB-001189
00","ob":[1694027456,0.36,254]}|0x5a5cf91d9dd7e202']
2023-09-06 12:11:37,848 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:37,860 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:40,830 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:11:40,849 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:40,863 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:43,665 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:11:43,690 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:43,704 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:44,437 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:11:46,668 - irricloud - INFO - mqtt publish drop: status/HB-00118900 no qos or response
2023-09-06 12:11:46,693 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:46,705 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:49,663 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:11:49,689 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:49,703 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:52,820 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:11:52,845 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:52,863 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:54,444 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:11:55,816 - irricloud - INFO - mqtt publish drop: status/HB-00118900 no qos or response
2023-09-06 12:11:55,841 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:55,853 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:58,656 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:11:58,678 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:11:58,695 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:01,820 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:12:01,843 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:01,855 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:04,448 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:12:04,821 - irricloud - INFO - mqtt publish drop: status/HB-00118900 no qos or response
2023-09-06 12:12:04,845 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:04,856 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:07,653 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:12:07,677 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:07,694 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:10,656 - irricloud - INFO - mqtt publish kept: rapid/ST-00107879 qos: 0 kept: [b'{"serial_number":"ST-00107879","type":"rapid_wind","hub_sn":"HB-001189
00","ob":[1694027489,1.06,206]}|0x5d9cd52b9e576760']
2023-09-06 12:12:10,680 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:10,691 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:13,819 - irricloud - INFO - mqtt full rate: 0.026KB/s tcp rate: 0.021KB/s udp_traffic_len: 4034 traffic_len: 17875
2023-09-06 12:12:13,822 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:12:13,845 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:13,858 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:14,456 - irricloud - INFO - mqtt publish kept: rapid/ST-00107879 qos: 0 kept: [b'{"serial_number":"ST-00107879","type":"rapid_wind","hub_sn":"HB-00118900","ob":[1694027495,0.73,153]}|0x2b1ee69e1998ec1b']
2023-09-06 12:12:16,654 - irricloud - INFO - mqtt publish drop: status/HB-00118900 no qos or response
2023-09-06 12:12:16,688 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:16,696 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:19,655 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:12:19,677 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:19,689 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:21,061 - irricloud - INFO - mqtt publish drop: rapid/ST-00107879 no qos or response
2023-09-06 12:12:31,071 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 qos: 1 packet_id: 0x12a2
2023-09-06 12:12:31,072 - irricloud - INFO - mqtt got pingreq
2023-09-06 12:12:31,149 - irricloud - INFO -   small gap: 49 ob: 0 instance: 11 obs_rain: 0.0 now: 1694027551 obs_time: 1694027502
2023-09-06 12:12:31,162 - irricloud - INFO - mqtt_s response: 0xd000
2023-09-06 12:12:31,164 - irricloud - ERROR - mqtt
Traceback (most recent call last):
  File "plugins/weather.py", line 1485, in mqtt
    s.send(self.forward_mqtt(now, ipsrc, mqtt_header+data[:pkt_len]))
BrokenPipeError: [Errno 32] Broken pipe
2023-09-06 12:12:31,167 - irricloud - INFO - exit mqtt
2023-09-06 12:12:34,499 - irricloud - INFO - dns changing response from: 54.81.39.221 to: 10.135.0.1 and ttl from: 60 to: 3600
2023-09-06 12:12:34,501 - irricloud - INFO - dns changing response from: 44.217.196.142 to: 10.135.0.1 and ttl from: 60 to: 3600
2023-09-06 12:12:34,503 - irricloud - INFO - dns changing response from: 34.238.102.102 to: 10.135.0.1 and ttl from: 60 to: 3600
2023-09-06 12:12:34,504 - irricloud - INFO - dns changing response from: 3.92.140.50 to: 10.135.0.1 and ttl from: 60 to: 3600
2023-09-06 12:12:34,506 - irricloud - INFO - dns changing response from: 54.156.255.179 to: 10.135.0.1 and ttl from: 60 to: 3600
2023-09-06 12:12:34,654 - irricloud - INFO - enter mqtt addr: ('10.135.0.13', 49188) port: 1883
2023-09-06 12:12:34,655 - irricloud - INFO - mqtt connect clientid: HB-00118900 keepalive: 300
2023-09-06 12:12:34,656 - substation_proxy - INFO - connect_socket mqtt.weatherflow.com:1883
2023-09-06 12:12:34,839 - irricloud - INFO - mqtt_s response: 0x20020000
2023-09-06 12:12:34,849 - irricloud - INFO - mqtt got subscribe flags: 0x2 topics: [('hub/HB-00118900', 1)] rest:
2023-09-06 12:12:34,940 - irricloud - INFO - mqtt_s response: 0x900312a301
2023-09-06 12:12:34,950 - irricloud - INFO - mqtt got subscribe flags: 0x2 topics: [('fw_data/HB-00118900', 1)] rest:
2023-09-06 12:12:35,040 - irricloud - INFO - mqtt_s response: 0x900312a401
2023-09-06 12:12:35,049 - irricloud - INFO - mqtt got subscribe flags: 0x2 topics: [('system', 1)] rest:
2023-09-06 12:12:35,136 - irricloud - INFO - mqtt_s response: 0x900312a501
2023-09-06 12:12:37,651 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
2023-09-06 12:12:37,674 - irricloud - INFO - mqtt publish drop: debug/ST-00107879 no qos or response
23

I should probably have used the term throughput but they are equivalent (in my context).

I have a certain amount of bandwidth (B/month) that I can devote to WF traffic. That is my throughput.
If the hub exceeds that output I am toast. The hub has the further property of “remembering” stuff it wanted to send and sometimes does not get current data out in a timely fashion because it has such a backlog.

While I may have more bandwidth (B/s) available from a short term bandwidth point of view, that does not change the B/month that is available for this purpose. So, for example, something like store/forward does not work. The only solution is to (effectively) reduce the hub output.

24

Also, can someone describe how the following mqtt subscriptions are used:
a) fw_data/hubid: Are fw updates performed as publishing to that topic from the mqtt server? Or is it something else?
b) system: I see time is passed along through that. Anything else?
c) hub/hubid: what is this for?

thanks

25

sorry - I have no WF gear any more so I can’t try to help much on this one. My only suggestion would be to try to mirror the traffic from your hub to WF somehow (if possible) to try to figure out what you can impersonate vs. what you can let through outbound or maybe see if you can rate limit.

I still have no idea what you mean by that.

26

I basically am pretty confident that what I am doing is comparable to what the “real” mqtt.weatherflow.com is doing but I have reduced the bandwidth by an order of magnitude.

I actually have logs that shows the problems I see on my end also exist without my filtering.

When you say you have “no idea” what I mean by if my bandwidth exceeds my limit I am not sure how to explain it better.

Said another way, under the circumstances I am trying to address, if the hub uses more than 100MB/month, then my internet shuts down for the rest of the month making my application useless. So, if the hub generates 100MB in 6 days (which it does), I have 30 days with no internet. That is what I call toast:)

1 Like
28

fwiw WF developers, I have found 4 instances of invalid mqtt data being sent from the hub in 2.5 hours. Obviously this is not catastrophic but in case you were unaware:) It is NOT the case that data is just getting spread over multiple packets.

This looks to me like there are two bytes (‘07’) prepended to what would likely be a valid message.
(The previous message looked fine.)
10.135.0.13->54.156.255.179: publish flags: 0x0 d: b’070\x89\x01\x00\x11rapid/ST-00107879{“serial_number”:"ST-00107879’

Here are 3 packets in sequence that are messed up as well:
10.135.0.13->44.217.196.142: pubrec flags: 0xf d: b’umber":“ST00107879”,“type”:“wind_debug”,“hub_sn”:“HB-00118900”,“ob”[1694139331,30,62,0,0,1420,1387,1347,1458’
10.135.0.13->44.217.196.142: pubcomp flags: 0x3 d: b’ats":[26,1,0,3,33745],“mqtt_stats”[23,4],“freq”:904000000,“hw_version”:1,“cellular_status”:2,“cell_rssi”:99}|0x5596’
10.135.0.13->44.217.196.142: pubcomp flags: 0x3 d: b’ats":[26,1,0,3,33745],“mqtt_stats”[23,4],“freq”:904000000,“hw_version”:1,“cellular_status”:2,“cell_rssi”:99}|0x5596’

29

I believe this was a mistake I made while while trying to compare “real” traffic with my results.

Sorry.