Post-Mortem Update:
Impact: Updates to customers documentation did not display.
Timeline:
- Jan 15th 17:15 PR merged for caching feature flags
- Jan 16th 06:29 Incident Declared
- Jan 16th 06:59 L2 Cache Disabled, updates displayed correctly
- Jan 16th 08:04 L2 Cache PR merged, and enabled
Root Cause:
A recent PR introduced caching for feature flags, which changed the feature flag getter to return a Promise. This required adding `await` at every call site, but one reference was missed.
One flag controlled whether cache invalidation targeted dev or prod. This change incorrectly lead the flag to evaluated as `true`, which lead to triggering revalidations against our dev environment
Steps to resolve:
To fix this immediately, we triggered a kill switch to directly hit our Layer 1 cache. While we patched a fix to our servers revalidation code. When the fix to our internal caching solution went live we
Action Items:
- Adding synthetic end to end tests for making sure the cache gets purged on updates. Previously our update tests only verified that the update was completed and not that the cache was reset.
Resolved
Post-Mortem Update:
Impact: Updates to customers documentation did not display.
Timeline:
- Jan 15th 17:15 PR merged for caching feature flags
- Jan 16th 06:29 Incident Declared
- Jan 16th 06:59 L2 Cache Disabled, updates displayed correctly
- Jan 16th 08:04 L2 Cache PR merged, and enabled
Root Cause:
A recent PR introduced caching for feature flags, which changed the feature flag getter to return a Promise. This required adding `await` at every call site, but one reference was missed.
One flag controlled whether cache invalidation targeted dev or prod. This change incorrectly lead the flag to evaluated as `true`, which lead to triggering revalidations against our dev environment
Steps to resolve:
To fix this immediately, we triggered a kill switch to directly hit our Layer 1 cache. While we patched a fix to our servers revalidation code. When the fix to our internal caching solution went live we
Action Items:
- Adding synthetic end to end tests for making sure the cache gets purged on updates. Previously our update tests only verified that the update was completed and not that the cache was reset.
Resolved
Updates are now being processed to websites. A postmortem report will be posted soon
Monitoring
Impact: Updates to documentation sites have been delayed
Timelines: This has impacted all sites from Jan 16th 2026 12:00 PM PST to 7:04 PST
Status
: We had reports of updates to documentation sites being delayed due to our cached pages not being invalidated
We disabled the L2 cache to allow updates to be displayed, and we are monitoring to confirm the fix is updated for all customer
