Search functionality experienced a service disruption affecting both unauthenticated and authenticated users. The incident lasted approximately 92 minutes. A fix for unauthenticated sites was deployed first, but authenticated sites required additional remediation due to a separate edge case in the authentication path.
Affected window: 7:12 PM – 8:44 PM PT
Users affected: All users relying on search in authenticated contexts
User experience: Search queries returned errors or failed to return results during the affected period
An upstream provider initiated an index migration, which caused search functionality to begin failing. Unauthenticated and authenticated search requests followed different code paths, so failing over to backup indices resolved the issue for unauthenticated sites first. Authenticated sites hit a separate edge case that required additional remediation, extending the total incident duration.
Note: We are completing our migration to a completely new search system tomorrow, which provide significantly improved stability going forward.
Resolved
Search functionality experienced a service disruption affecting both unauthenticated and authenticated users. The incident lasted approximately 92 minutes. A fix for unauthenticated sites was deployed first, but authenticated sites required additional remediation due to a separate edge case in the authentication path.
Affected window: 7:12 PM – 8:44 PM PT
Users affected: All users relying on search in authenticated contexts
User experience: Search queries returned errors or failed to return results during the affected period
An upstream provider initiated an index migration, which caused search functionality to begin failing. Unauthenticated and authenticated search requests followed different code paths, so failing over to backup indices resolved the issue for unauthenticated sites first. Authenticated sites hit a separate edge case that required additional remediation, extending the total incident duration.
Note: We are completing our migration to a completely new search system tomorrow, which provide significantly improved stability going forward.
Monitoring
Search functionality experienced a service disruption affecting both unauthenticated and authenticated users. The incident lasted approximately 92 minutes. A fix for unauthenticated sites was deployed first, but authenticated sites required additional remediation due to a separate edge case in the authentication path.
Affected window: 7:12 PM – 8:44 PM PT
Users affected: All users relying on search in authenticated contexts
User experience: Search queries returned errors or failed to return results during the affected period
An upstream provider initiated an index migration, which caused search functionality to begin failing. Unauthenticated and authenticated search requests followed different code paths, so failing over to backup indices resolved the issue for unauthenticated sites first. Authenticated sites hit a separate edge case that required additional remediation, extending the total incident duration.
Note: We are completing our migration to a completely new search system tomorrow, which provide significantly improved stability going forward.
Identified
We have identified an issue with search for Mintlify deployments which have authentication enabled and are working on a fix.
