On May 20, Mintlify experienced a serious outage which caused slowness or total unavailability for large parts of the platform, including search, docs updates, and our dashboard, and in some cases led to customers being unable to view their docs at all. This outage lasted for approximately two and a half hours.
In short: starting from around 12:30AM PDT, our primary database cluster became seriously unstable, starting with extreme latency on all queries, followed by multiple full-node rollovers.
The initial instability was caused by the combination of
an inefficient database query,
bad application logic which called that query more often than it should have, and
an automated web crawler hitting our sites in a way that triggered this bad logic a large number of times in a very short period.
The conditions for this outage had existed for a number of weeks; we just hadn’t hit them all at once until last night.
After the initial instability began to subside, further database load was incurred when other previously-interrupted services and queries began to come back online. This caused a thundering herd effect which brought down the database again, and caused recovery to take longer than it otherwise should have.
We saw full recovery by around 3:00AM PDT.
Our team was immediately notified via our automated alerts when our infrastructure became unstable at 12:22AM. We began investigation at 12:25AM. Within the next few minutes, we had multiple engineers online and coordinating investigation, and we updated our status page to reflect the outage. While we were investigating, our database cluster went down completely. We scaled database capacity to try to put a band-aid on the issue. However, after coming back up, even with the additional capacity, the database went down again. Other services, finally able to reconnect, overwhelmed the scaled-up cluster and caused it to go down again. At this point, we had identified the offending logic, we promptly shipped a codebase update to disable it, and once again increased database capacity. After this, the issue was resolved for all customers.
The team has already taken steps to prevent issues like this one from happening again, and is actively working on others:
We’ve already updated the offending database query and business logic to prevent this specific failure case from causing undue strain on our database.
We’re implementing multiple guardrails to try and prevent a single service or worker from causing such a broad impact—especially a non-critical service, like the one which caused the failure here, which should not have been able to cause downtime for mission-critical services like search and content updates.
We’re deploying a MongoDB connection pooler
We’re setting explicit connection bounds in our ORM’s connection logic
We’re setting a lower query timeout for all queries made from the application (to stop runaway queries in the first place)
