Mementomori.social experienced 57 minutes of downtime earlier today. Here's what happened and what we've done about it.

What happened: A large volume of connections from external crawler/bot IP ranges accumulated in a stuck state (CLOSE_WAIT) inside our web server (nginx). These zombie connections exhausted nginx's connection pool and connections per worker, which caused it to stop accepting new connections entirely resulting in ERR_CONNECTION_CLOSED for everyone trying to reach the site.

Why it took 57 minutes: The sole admin available (me) was in a meeting when the issue began, and other admins were unavailable. There was no automated recovery in place to detect and resolve this type of failure.

How it was resolved: Nginx was restarted, which cleared all stale connections and immediately restored service.

What we've done to prevent this:
- Installed and configured Monit, a process monitoring tool that automatically checks all critical services (web, background workers, streaming, database, cache, and nginx) every 2 minutes and restarts them if they become unresponsive
- This means similar outages should now self-heal within minutes, even when no admin is available

I'm really sorry for the disruption. We take the reliability of this instance seriously, and this change should make a real difference moving forward. I'm also planning to apply other mitigations, as some parts of our instance may have been underconfigured.

#MementoMoriSocial