Major incident

LiveKit: Connection failures and SIP transfer errors in US Central

Affected: US Central - SIP, US Central - Real Time Communication

Current status of LiveKit: Operational — live LiveKit status

Timeline

1. Jun 22, 2026, 13:05 UTC
   monitoring
   We have begun routing traffic away from US Central after our monitoring system triggered due to a high API error rate. Customer impact may include WebRTC connection failures and SIP transfer errors.
2. Jun 22, 2026, 13:59 UTC
   monitoring
   Update: As soon as the US Central region became unresponsive around 12:25 UTC, traffic was automatically re-routed to the nearest healthy region. Customers may have noticed failed API requests for approximately 1 minute while the re-route took place. Failed SIP Transfers continued until we finished manually draining the impacted region at 13:09 UTC. We are also investigating impact to Agent Dispatches and will post another update when we know more.
3. Jun 22, 2026, 14:40 UTC
   resolved
   We are resolving this incident as the mitigation is in place and successful. We have received some reports of self-hosted agents which connect to US Central needing to be restarted in order to continue receiving dispatches and will investigate opportunities to improve this behavior. Apologies for the disruptions and we will follow up with a postmortem as soon as possible.
4. Jun 23, 2026, 08:32 UTC
   postmortem
   ## Summary LiveKit Cloud is a globally distributed system. Our data centers continuously probe one another, and our overlay network uses those probes to choose healthy paths between regions. When a brief network disruption occurs in or between data centers, the overlay is designed to mark affected paths as unhealthy, route traffic around them, and return them to "healthy" once the underlying network has cleared. On June 18, 2026 \(US East 1\) and again on June 22, 2026 \(US Central\), a short packet-loss event in the affected data center triggered a compound failure in this mechanism. Two separate defects, one in the overlay network and one in our link-monitoring service, interacted in a way that left the affected region effectively isolated from other regions long after the underlying network had cleared. For most customers and most products, the impact was bounded. Our global routing automatically moved traffic to other US regions within approximately one minute of onset. SIP transfers anchored in the affected region took longer to recover but cleared within approximately thirty minutes. A small number of LiveKit Agents running self-hosted workers experienced an extended outage in agent dispatch, up to about two hours in some cases. The majority of agent workloads continued to receive jobs normally after the routing flip. The workaround at the time was to restart the affected agent worker process so that it re-registered against a controller in a healthy region. This is a separate defect in our agent dispatch routing, and we are fixing it. This post-mortem describes what happened, why the second incident occurred so soon after the first, and the steps we are taking to prevent recurrence. ## Impact * **Realtime connections:** minimal impact. Clients automatically reconnected to alternative data centers. * **API:** approximately 1 minute of degraded availability while traffic rerouted to a nearby region. * **SIP transfers:** approximately 30 minutes for in-progress transfers anchored in the affected region. * **Agent dispatch:** approximately 1 minute for the majority of agents. A small number of agent workers saw extended dispatch outages of up to approximately 2 hours until the worker process was restarted. * **Egress**: A few number of in-progress egresses running in the affected regions were ended prematurely during the approximately 1-minute outage window, between 16:47 and 16:48 UTC on June 18, and between 12:24 and 12:25 UTC on June 22. These egresses were incorrectly marked as successful and no error was surfaced to indicate the recording had failed. ## Root Cause The trigger in both incidents was the same: a short packet-loss event in the affected data center. The expected behavior was that affected paths would be marked unhealthy briefly while the overlay rerouted traffic around them, and would return to "healthy" once the underlying network cleared. Two separate defects interacted in a way that kept routing state stuck long after the network had recovered: 1. **Inbound and outbound freshness collapsed onto a single indicator.** In the overlay network, the two directions of a connection \(inbound and outbound\) shared a single "freshness" indicator. When fresh data arrived in one direction, our code incorrectly assumed data in the other direction was also fresh. As a result, the affected region continued to treat its outbound connections to other regions as unreachable based on stale inbound data, even after outbound traffic was not impacted. 2. **Link-monitor key exchange could de-sync during a disruption.** Our link-monitoring service relies on a key exchange between regions to validate probe traffic. The packet-loss event caused this key exchange to fall temporarily out of sync between the affected region and its neighbors. With probes failing validation, the link monitors in neighboring regions marked the affected region as "to avoid," and held that state past the actual network recovery. Either defec

Updates as published by LiveKit (original).