New MQTT feeds for optimising the web interface

Back when SondeHub first got off the ground there were only a few tracking stations. We published all data to the web clients as it came in. Today there are 1300+ receiving stations feeding data and many older computers and mobile phones struggle with showing the unfiltered map.

A longer term goal as we scaled up has been to improve the web experience by only receiving and drawing the data that's relevant to the user. Most people only look at a small section of the map and don't require the full feed.

We already have topics in our MQTT that allow the web interface to only show selected radiosondes based on serial number. The idea is that we extend this functionality to only display the radiosondes for the map extent visible by only subscribing to the serial numbers in that area. The tricky part here is finding which serial numbers are for an area. Luckily there's already an API that lists all sondes currently in flight.

That leaves us with one missing piece - what happens if a sonde is only just launched - how does the web client find out about this radiosonde.

As of yesterday we have some new low bandwidth feeds - sondes-new/#. This feed provides an "at least once" broadcast of a radiosonde when its first seen. Due to technical constraints it's not guaranteed that it'll only be sent once - however the throughput is slow enough to solve our above problem.

This paves the way for the future web interface changes to improve usability and performance.

AWS Outage

Just a quick note here that this morning AWS suffered an outage in the us-east-1 region. As we are hosted in this region we were also impacted. Once the AWS outage was cleared SondeHub recovered without human intervention (I was asleep). Most of the radiosonde software currently doesn't retry sending data, as a result there were a few hours of lost data.