Okay, so, that went poorly. I failed to upgrade our instance and I may have lost posts made between 7 AM and the start of maintenance.
This is a pretty obviously bad outcome, and I'm sorry about that. I've pretty much exhausted my options for today. Adam and I are investigating masto.host as an option to get some real professionals to handle this stuff. I'll update as we know more.
Sorry for the downtime folks! I'm on leave and busy with family stuff. @blinks stepped up and solved it. And since he's amazing, he wrote a post-mortem: https://gist.github.com/blinks/dabac9fb850ad00941f9a50b882a0e38
Bad news: I was tagged in to a thread with some of my least favorite people to talk D&D/DW.
Good news: Because people were wrong on the internet I migrated some 8 year old posts on D&D and AW-style Principles and now you get to read them.
Supporting dice.camp: a few people have asked about how to support dice.camp. The short answer is you don't have to. But if you want to help us meet our costs (about $20/month) we now accept contributions at http://ko-fi.com/dicecamp.
Best guess on problem: something went wrong in the database, which led to Mastodon being stuck in its regular cleanup task using the entire CPU.
The best way to get the db out of this state was the migration that happens as part of upgrades.
These are all wild guesses, nothing concrete enough to track as a bug on the Mastodon codebase.
And we're back!
Last night around 1 AM PST something maxed the CPU on our tiny virtual server. It was pegged so hard the machine was unresponsive. I restarted it, and it didn't come back up cleanly. Unable to get it to settle down, I went back to our last snapshot, which happened just before the machine went wild. (Maybe related to why it went wild?)
I was unable to stand that up either, and ended up upgrading that snapshot to the latest Mastodon code, which somehow unstuck it.
More specifically: I am a noob at docker and didn't want to lose anyone's data. But the latest releases of Mastodon have some features I'd like, and the guide I used to set up this instance added a "how to update" section, and I set up automatic daily backups, so it seems like time to bite the bullet.
So why did updates not happen? Because https://dice.camp/media/j9MGMxDJ13DKjmpnC3w
Low-entropy entity. Dungeon World co-designer. Senior Test Engineer at Google. He/Him.
A Mastodon instance for tabletop gamers.