Major incident

WorkOS: Elevated API Errors

Current status of WorkOS: Operational — live WorkOS status

Timeline

1. Sep 18, 2026, 22:49 UTC
   investigating
   We are investigating an issue with our API. We apologize for the inconvenience and will share an update once we have more information.
2. Sep 18, 2026, 23:01 UTC
   investigating
   We're continuing to investigate this issue to identify the root cause. We'll share another update soon.
3. Sep 18, 2026, 23:05 UTC
   identified
   We've identified the root cause, and we're seeing services are beginning to recover. We'll continue to monitor closely and share updates.
4. Sep 18, 2026, 23:09 UTC
   monitoring
   Our initial remediations are in place, and services have recovered. We're continuing to monitor closely.
5. Sep 18, 2026, 23:29 UTC
   resolved
   Our remediations continue to be effective, and services continue to be recovered. We will continue further investigation and additional preventative measures.
6. Sep 22, 2026, 00:53 UTC
   postmortem
   # Elevated API Errors **Date:** 09/18/2026 **Duration:** 22:40–23:03 UTC \(23 minutes\) **Status:** Resolved ## Summary On September 18, 2026, from 22:40 to 23:03 UTC, WorkOS experienced an outage affecting authentication and other API requests. Customers encountered failed sign-ins, API errors, timeouts, and long delays. Authentication was most severely affected. An infrastructure failover in the cache supporting our feature-flag system exposed weaknesses in application connection recovery and fallback behavior. Some connections remained stalled after the cache was available again, delaying requests and exhausting database connection pools. Service recovered by 23:03 UTC. We subsequently deployed changes to improve connection recovery. We’re sorry for the disruption this caused you and your users. ## Background WorkOS uses feature flags to control application behavior. The affected backend services read flag data from a shared Redis datastore populated by LaunchDarkly Relay. Each application also keeps a local cache, with a 30-second expiration. While an entry is valid, the application can evaluate it locally; after it expires, the application reads Redis. The Redis cluster already had a primary and replica in separate AWS availability zones, with automatic failover enabled. This redundancy allowed AWS to promote a replacement primary. Applications still needed to detect failed connections and reconnect to that primary. ## What Happened ### Affected products Some requests experienced errors or delays across the following products: * **AuthKit:** Sign-in, token requests, user-management operations, and hosted authentication pages. * **Single Sign-On \(SSO\):** Authorization and SAML callback requests. * **WorkOS API:** Selected operations, including organization requests, API-key validation, and connection requests. * **Dashboard:** Some operations that relied on the affected APIs. * **Admin Portal:** Portal link generation and monitored setup requests. * **Directory Sync:** SCIM OAuth token requests, plus delays and client disconnects in Directory Sync API traffic. * **Audit Logs:** Management operations, particularly action-schema creation. ### Customer impact Customer impact lasted from **22:40 to 23:03 UTC**. Across our public edge, **24.1% of recorded requests returned server errors**, and **17.2% ended with the client disconnecting**. The table breaks down request groups by product. | Product | Server errors \(HTTP 5xx\) | Client disconnects \(HTTP 499\) | | --- | --- | --- | | AuthKit, User Management and organization APIs | 37.22% | 24.59% | | Hosted AuthKit \(default-domain proxy\) | 7.27% | 13.77% | | SSO and connection APIs | 2.22% | 2.31% | | Dashboard | 27.10% | 8.08% | | Admin Portal | 24.61% | 7.47% | | Directory Sync APIs | 0.08% | 2.80% | | Audit Logs management APIs | 22.82% | 3.55% | | Audit Logs event capture | 0.0006% | 0.11% | #### Latency Samples * **Authenticate POST requests:** At the peak, one-minute median latency reached approximately 32 seconds and p95 latency reached approximately 45 seconds, compared with roughly 65 ms and 140 ms before the incident. * **Directory Sync:** Peak one-minute p95 latency for selected SCIM user-creation and PATCH-update operations reached approximately 22 seconds, compared with roughly 65 ms before the incident. ### Root cause and contributing factors Around 22:40 UTC, the Redis primary became unavailable to application clients. AWS ElastiCache promoted its replica, completing the failover at 22:40:17 UTC. New connections could reach the promoted node, but some existing application connections remained stalled. Three application behaviors prolonged and amplified the interruption: 1. **Flag-call timeouts did not recover stalled connections.** An application-level deadline returned a fallback value after two seconds, but it did not cancel the underlying Redis command or replace the connection. Additional reads accumulated while clients waited to det

Updates as published by WorkOS (original).