Minor incident

LiveKit: Disconnections and failed connections in multiple regions

Affected: US East - Ingress, US West - Ingress, US West - Real Time Communication, US Central - Real Time Communication, UAE - Real Time Communication, Israel - Real Time Communication, Saudi Arabia - Real Time Communication, Europe South - Real Time Communication, US Central - Ingress, US East - Real Time Communication, Europe Central - Real Time Communication

Current status of LiveKit: Operational — live LiveKit status

Timeline

1. Oct 31, 2026, 07:43 UTC
   resolved
   This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly. During this window: - Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join. - Most agents reconnected and continued their conversations, but a small number of agent sessions ended or restarted. Some SIP call transfers also failed. - About 1.3% of Room Service API requests failed during this period. ListParticipants and GetParticipant were the most affected. Some WHIP ingress update and delete requests also failed. The bug was triggered when a specific type of data message was relayed between servers hosting the same multi-region room. We stopped the trigger at 07:43 UTC, and there were no further restarts after that. A fix was deployed to all regions by 08:49 UTC. Apologies for the impact. We will follow up with a postmortem sharing more details on impact and RCA as soon as possible.
2. Nov 3, 2026, 13:54 UTC
   postmortem
   **Summary** On October 31, 2026, between 07:10 and 07:43 UTC, an uncommon data pattern surfaced a latent software bug which caused media servers in several regions to restart repeatedly. A small percentage of participants were disconnected and the vast majority automatically reconnected. Impact was concentrated in North America. We stopped the trigger at 07:43 UTC and deployed a fix to all regions by 08:49 UTC. We apologize for the disruption this caused. **Root Cause** Applications can open their own data channel and send raw data over it, outside of LiveKit's standard data message format. Media servers forward this raw data to other participants in the room, wrapping it with the sender's identity so that receivers know who sent it. When a room spans multiple regions, the media servers hosting it also forward this data to each other. That server-to-server step sent the raw data without the sender's identity, and the receiving server assumed the identity would be present, which caused it to crash. This gap in our server-to-server protocol had been present for some time and was not introduced by a recent deployment. Starting at 07:10 UTC, one application sent raw data over its own data channel in a room that spanned multiple regions. Every server that received the forwarded message crashed at the same time, while the server the sender was connected to kept running and kept sending. Participants on the crashed servers automatically reconnected to other servers. Those servers then began hosting the room, received the next message, and crashed as well. This cycle repeated until we stopped the trigger at 07:43 UTC. In total, media servers in 12 of our data centers restarted, some of them up to 10 times. **Timeline \(UTC\)** 2026-10-31 07:10 - An application begins sending raw data over its own data channel in a multi-region room. Media servers in US East and Canada restart within seconds. 07:13 - Our monitoring detects the restarts and an incident is opened automatically. 07:10 to 07:43 - Media servers in 12 data centers restart repeatedly as participants reconnect to new servers. 07:43 - We block the traffic triggering the bug. There are no further restarts after this point, and customer impact ends. 07:46 - A fix for the bug is ready. 08:04 - The fix begins rolling out, starting with a small set of regions. 08:49 - The fix is deployed to all regions. **Scope of Impact** Realtime sessions: * Across all regions, about 1.5% were disconnected more than once within five minutes. * In US East, US Central, US West, Canada and Mexico, about 6% were disconnected more than once within five minutes. * The median time from disconnect to rejoin was about 3 seconds. Agents: * Agents were disconnected and reconnected the same way as other participants, with the same median reconnect time of about 3 seconds. An agent that reconnects keeps its conversation state, so most conversations continued after a short gap. Fewer than 1% of active agent sessions did not recover. Ingress, Egress and SIP: * Most active WHIP ingress streams were interrupted, and reconnects typically took about 40 seconds. Some WHIP ingress update and delete requests also failed. * About 5% of active egress sessions were disconnected. Most reconnected within a few seconds. * About 1% of SIP participants were disconnected, and half of them rejoined in under a second. Some SIP call transfers also failed. APIs: * About 1.3% of Room Service API requests failed during the window. ListParticipants and GetParticipant were the most affected. How to tell if you were affected: participants in affected sessions saw a reconnecting state, or a brief loss of audio and video. In session analytics, an affected participant appears to leave and rejoin the same room with the same identity between 07:10 and 07:43 UTC. **Detection and Response** Our monitoring detected the server restarts about 3 minutes after they began, and an incident was opened automatically. Iden

Updates as published by LiveKit (original).