Mastodon 3.0

Hello Patreon! I'm finally writing this. The release of 3.0 was somewhat stressful, because it was wrapping up a very long period of work. On one hand it's nice to be able to work on something for longer without interruptions, because it gives an opportunity to find bugs and usability issues as they come, change your mind about features, with less bureaucracy inbetween; but on the other hand it just builds up a lot of pressure to deliver, and more changes equals more bureaucracy in the end, as I have to transform a 563-long commit log into something a non-developer reader would find useful, and then select which of the changes deserve to be editorialized into a blog post. This time there's been so many new and changed features that the blog post had to be aggressively trimmed to the most impactful ones, and I'm still not 100% certain we selected the right ones to write about because the impact varies from person to person.

I pushed out v3.0.0 on October 3, but without announcing it on social media as the blog post wasn't ready yet (it needed some last-minute rewrites). The thinking was, make the code available as soon as possible, and the announcement can come later when we have a blog post that people could submit to link aggregator sites, at a more convenient time when link aggregator submissions tend to do better. Well, it didn't go as planned, as somebody simply submitted the long changelog on GitHub to Hacker News almost straight away. Can't complain about that outcome, though, as it got on the frontpage. Who knows if the blog post would've gotten the frontpage? I have a theory that people on Hacker News tend to click on familiar GitHub/Medium URLs more than on custom domains like blog.joinmastodon.org.

Unfortunately, just a day or so after that, Wasabi, the object storage provider used by my Mastodon server, mastodon.social--basically, the service that stores people's file uploads--started experiencing a major outage. So all the hype from new features, and new users from the news about them, were met by broken uploads, missing avatars, and general slowness. As the day progressed, things only got worse. What should've only been affecting file uploads was for some reason affecting the entire database, leading to ever increasing delays on users' home feeds. This led me on a bug hunt. Long story short, for those interested: The paperclip gem uploads files to storage within a database transaction (and okay, fair enough, can't blame that behaviour as you wouldn't want to save a row if the file didn't actually save). So if the storage (in our case, Wasabi) is slow as a snail, the transaction will be kept open for a long time.  The long-running transactions were not so much locking up the database per-se, as simply exhausting the connection pool. But we tell it to time out after 5 seconds, so why were transactions being kept open for multiple minutes? Turns out the aws-s3 gem has a built-in, on-by-default, synchronous retry mechanism with exponential backoff. Turning that off and increasing the connection pool got us through, and it only cost a night and my sleep schedule.

Unfortunately Wasabi remained inoperational for like a whole week, if not more. I've experimented with using an S3 proxy service (flexify.io) to migrate mastodon.social to a different object storage provider, but discovered that with the amount of data we generate per hour (5 GB), the pricing of the proxy service (at 5 GB per hour, $1 USD per hour), and the fact that until Wasabi becomes operational the old data could not be moved to the new provider and the proxy service would continue to be necessary, I just could not afford it, so I unplugged it again. Thankfully, a day or two later, Wasabi finally came back online. Super stressful time though, and I am looking into changing providers long-term.

Anyway, by that point, I already had some fixes for bugs discovered in v3.0.0, and the blog post wasn't finished (due to illness, Eleanor could not work on it, and my hands were full with the Wasabi issues), so I decided to wrap up v3.0.1 while getting outside help for the blog post from Tremaine Fiske (professional copywriter). Finally, on October 11, both the blog post and v3.0.1 went live.

Still though there was so much uncovered ground left that I wrote an additional blog post just to explain how to use all of the new APIs and admin features, and then another one explaining how to add Mastodon sign-up to mobile apps--while not a 3.0 feature, I find that it is still severely lacking from current Mastodon apps. Then, I had a more or less spontaneous idea to record a "patch analysis" video--something people do for video game patches, especially multiplayer ones like Dota 2, where they go over changes one by one and discuss its meaning and impact--and hopped onto Twitch with Kurtis Rainbolt-Greene, where over the next 1h 15m I proceeded to gradually lose my voice. And we only covered one section of the changelog!

So here's where we're at now. I'm trying to get my thoughts together, focusing on non-development aspects of Mastodon. The joinmastodon.org server submissions backlog has been processed, the documentation is being re-worked with outside help, we're also still going to push a blog post out about Mastodon's now-possible use case in schools. We're looking into providing more merch options besides t-shirts, and might get another (video) ad for Mastodon. At the start of November I'm going to Athens for another Eunomia conference, which while also work, will at least let me switch up my environment for a bit after being holed up inside for 4 months.

Meanwhile, I would love to hear what my patrons want to see in Mastodon in the future. What would you like to see me work on in the next version?