Minor incident

LiveKit: Investigating reports of elevated API failures in Chicago region

Affected: US Central - Real Time Communication

Current status of LiveKit: Operational — live LiveKit status

Timeline

1. Oct 19, 2026, 17:02 UTC
   investigating
   We are currently investigating reports of elevated API errors across our services in the Chicago region.
2. Oct 19, 2026, 17:11 UTC
   monitoring
   A fix has been implemented as of 17:00 UTC and we are seeing the errors return back to baseline. We are continuing to monitor before marking this resolved.
3. Oct 19, 2026, 17:34 UTC
   resolved
   Between 16:35-17:03 UTC, Chicago region experienced about 11% failure rates in API errors, failed agent dispatches, and issues connecting to rooms. We faced another wave of the same issue between 17:47-17:53 UTC where the failure rates were less than 10%. Room Service APIs called from other regions targeting to a room hosted in Chicago region were also affected during this period. API requests made with recent server SDK versions are automatically retried in other regions, so customers on those versions should not have seen these errors. The last related errors occurred at 17:53 UTC, and all affected services are operating normally since. We will share a detailed RCA with more information soon.
4. Oct 22, 2026, 00:47 UTC
   postmortem
   ## Summary On October 19, 2026, between 16:34 and 17:03 UTC and again between 17:47 and 17:54 UTC, some room management, agent dispatch and SIP API requests failed for rooms hosted in our US Central \(Chicago\) region. Requests from other regions that targeted those rooms also failed. During these windows some rooms could not be created or joined, and some agents were not dispatched. The cause was CPU exhaustion on the node running one of our message bus instances in US Central. A recently deployed overlay network used more CPU than it requested, and because our monitoring tracked CPU per pod rather than per node, we didn't see the node run out. That message bus instance began dropping messages, and a room-coordination service that depended on it stopped completing requests. We have moved the message bus in US Central onto larger nodes and resized other regions at similar risk. Node-level CPU alerting and automatic recovery for the coordination service are in progress. ## Impact Between 16:34 and 17:03 UTC, and again from 17:47 to 17:54 UTC, room management \(RoomService\), agent dispatch \(AgentDispatchService\) and SIP APIs failed for some rooms hosted in US Central. Because every region relies on the same coordination step, requests from other US regions that targeted those rooms failed as well. Affected requests returned a 503. Agent dispatch was the most affected. During the first window, 26.5% of rooms that requested an agent did not receive one. During the second window, 17.5% did not. Rooms hosted in other regions, and requests that did not target US Central rooms, were not affected. ## Root Cause Changes to room state, such as creating a room or dispatching an agent, must be confirmed by several servers before they take effect. A coordination service gathers those confirmations over our internal messaging system. If a quorum doesn't arrive within one second, the change fails. We recently rolled out a new overlay network for traffic between regions. It is more resilient to packet loss, but uses more CPU than its predecessor. In US Central, the message bus had little CPU headroom, and traffic was uneven: one of the three instances carried more than the other two combined, and its load had nearly doubled over the preceding hours. The overlay runs on every node and carries each message the message bus sends or receives. To guard against packet loss, it sends each message over several relay paths, so its CPU use climbs with message traffic. So that time-sensitive packet delivery is never throttled, the overlay can use more CPU than it requests, and it runs at a higher scheduling priority than the message bus. On the busiest instance's node, the two together pushed CPU above 90% and starved the message bus. At 16:34 UTC, routine node maintenance moved another messaging server to a new host, and its clients reconnected to the remaining servers. Shortly after, one coordination instance stopped getting replies through the overloaded server. New requests still reached it, but the responses it needed to finish them did not. Within four minutes it had stopped completing requests entirely. The coordination service is built to tolerate the loss of an instance: when one fails, another takes over its work. But this instance had not failed outright. Some messages still reached it, so it kept reporting itself healthy, and callers kept waiting on it instead of moving on. They retried, and request volume to it rose more than tenfold. Its unfinished updates also blocked changes to the same rooms in other US regions. We ended the first window by replacing the instance manually. The second window began when the instance that took over its busiest work stalled the same way, because the message bus was still short of CPU. Our monitoring did not catch the saturation. We monitored CPU for each pod against its own request, and relied on Kubernetes to keep nodes from being overloaded. That works only while every workload stays with

Updates as published by LiveKit (original).