Critical incident

LiveKit: Elevated Reports of Participant Connection Latency and Errors In US East Region

Affected: US East - Real Time Communication

Current status of LiveKit: Operational — live LiveKit status

Timeline

1. May 28, 2026, 14:53 UTC
   investigating
   We are currently investigating reports of spikes in participant connection latency and errors
2. May 28, 2026, 15:35 UTC
   identified
   We are observing database connection timeouts in the US East region and are currently seeing an impact to all services in the region. We are currently draining the region and routing away all traffic to other regions.
3. May 28, 2026, 15:50 UTC
   monitoring
   US East drain is complete and all new traffic is now routing to other regions. Error rate is trending down and we'll continue monitoring.
4. May 28, 2026, 16:27 UTC
   monitoring
   Connection errors have fully cleared since the US East drain completed at 15:44 UTC. We'll continue monitoring. We will be sharing a detailed RCA.
5. May 28, 2026, 18:27 UTC
   monitoring
   Service is operating normally with traffic routing through other regions. We are working on bringing US East back online and will share a detailed RCA once complete.
6. May 28, 2026, 20:13 UTC
   resolved
   Connection errors have remained cleared since the US East drain at 15:44 UTC, and as of 19:30 UTC, US East is back online. We'll share a detailed RCA in the coming days.
7. May 30, 2026, 18:18 UTC
   postmortem
   ## Summary LiveKit Cloud validates incoming API and connection requests via an internal authentication service, accessed over gRPC. The service has been in place since 2022 and is designed for resilience: it runs as multiple redundant instances, has pod-level health monitoring that automatically reaps unhealthy pods, and is backed by in-memory caches so it can survive transient database failures. On 2026-05-28, between 13:55 UTC and 15:45 UTC, a percentage of requests and new connections in our US East region failed or timed out. The root cause was a rare failure mode on a single instance of the authentication service. The instance remained reachable and its TCP connections stayed alive, but it began responding to gRPC requests extremely slowly, in a way that did not trip our existing pod-level health checks or cause gRPC clients to fail over. We sincerely apologize to customers whose traffic was disrupted. We've let you down, and we are taking this very seriously. In addition to the corrective actions outlined below, we are performing a thorough audit of our systems for other failure modes we may not have anticipated. ## Timeline and Impact \(all times in UTC on 2026-05-28\) * **13:55**: Elevated rate of authentication failures observed. * **14:15**: On-call team notified; investigation begins. * **14:46**: Error rate continues to climb; incident escalated to Sev 1. * **15:35**: US East drained; error rate begins to subside. * **16:32**: Root cause identified; faulty instance shut down. * **16:32**: Webhooks backlogged during the incident begin to deliver. Between 2026-05-28 13:55 UTC and 15:35 UTC, a percentage of API requests and new connection attempts in our US East region failed or timed out. This included new participant connections, SIP connection requests, and agent connections originating in US East. The blast radius of this incident was substantially reduced by our local in-process auth cache. Each service maintains an in-memory cache of recently-seen authentication details, which allowed the majority of requests to continue flowing whenever they landed on a server that had already cached the relevant credentials. The failures concentrated on requests that landed on servers without those credentials cached, which disproportionately impacted customers with lower overall traffic to US East. Existing realtime sessions in US East continued to operate, and all traffic in other regions was unaffected. The impact subsided once US East was drained and traffic was rerouted to neighboring regions. There was also a secondary effect for customers who subscribe to outgoing webhooks. During the incident, webhook deliveries that depended on the affected auth path were queued internally while waiting for a valid signing token. Once the faulty instance was removed and the previously stuck operations unblocked, the queued webhooks were delivered in a burst rather than smoothly over time. Customers whose webhook endpoints have limited throughput may have observed a short-lived spike that exceeded their normal load. ## Root Cause The trigger was a rare hardware failure mode on one instance of our authentication service. Typically, when hardware fails, the underlying machine is shut down either by the hypervisor or by Kubernetes \(via failed health checks\), and gRPC clients reconnect to a healthy instance. In this case, the faulty machine did not terminate. It remained reachable and continued to respond, but extremely slowly. The TCP connection to that instance stayed up, so gRPC clients that had pinned their requests onto that connection continued to use it, and pod-level health checks did not fire. As a result, a percentage of services in US East were unable to fail over to healthy instances and saw their auth requests time out. Two additional fallback mechanisms did not engage as fully as we had designed: 1. **Client-driven cross-region failover.** LiveKit clients can automatically fail over from one region to a neighb

Updates as published by LiveKit (original).