Minor incident

LiveKit: Degraded SIP Performance in US West

Affected: US West - SIP

Current status of LiveKit: Operational — live LiveKit status

Timeline

1. Oct 16, 2026, 16:04 UTC
   investigating
   We are investigating degraded SIP performance in US West starting at 14:49 UTC affecting new outbound calls.
2. Oct 16, 2026, 16:12 UTC
   identified
   We have identified the source of the issue and are working to route around the impact.
3. Oct 16, 2026, 16:35 UTC
   monitoring
   We applied mitigation as of 15:58 UTC and SIP outbound connection success has returned to normal. We are continuing to monitor before marking this resolved.
4. Oct 16, 2026, 16:57 UTC
   monitoring
   We estimate that approximately 14% of outbound calls and 5% of inbound calls in US West failed between 14:49 UTC and 15:58 UTC. We will follow up as soon as possible with more details.
5. Oct 16, 2026, 17:30 UTC
   resolved
   This incident has been resolved. Between 14:49 UTC and 15:58 UTC, approximately 14% of outbound calls and 5% of inbound calls in US West failed. We will follow up with more details surrounding the root cause and preventative actions in a postmortem.
6. Oct 22, 2026, 14:22 UTC
   postmortem
   **Root Cause** On 2026-10-16, a subset of SIP calls in our US West region failed. Two of our SIP servers advertised IPv6 addresses for signaling and media, which many carrier providers don't support. Carriers rejected outbound calls routed through these servers, and inbound calls were answered but dropped almost immediately without audio. We had recently started running SIP in a US West cluster that uses both IPv4 and IPv6 networking and our SIP servers didn't enforce IPv4 addresses. **Technical Details** When a SIP server starts, it determines its public address by querying an external address-lookup service. On the two affected servers, our primary lookup service didn't respond in time so they fell back to a backup service that returned their IPv6 address instead. Nothing in the SIP server checked that the address it advertised was IPv4. **Timeline \(UTC\)** 2026-10-16 14:49 - First affected SIP server started with an IPv6 address; call failures began 2026-10-16 15:41 - Second affected SIP server started with an IPv6 address 2026-10-16 ~15:48 - Customers began reporting failures to our support team 2026-10-16 15:58 - SIP traffic began moving away from the final affected cluster due to automatic cluster draining 2026-10-16 15:59 - Last affected call; customer impact ended 2026-10-16 17:08 - Began rolling out the long term fix requiring the usage of IPv4 2026-10-16 ~21:28 - Fix rollout completed 2026-10-16 ~22:30 - SIP traffic restored to the affected cluster **Scope of Impact** Between 14:49 and 15:59 UTC on 2026-10-16, SIP calls handled by the two affected servers in US West failed. During that window, about 14% of outbound calls in US West were rejected at setup and about 5% of inbound calls were answered and then dropped without audio. **Detection and Response** Our customers noticed this before our monitoring did. Our alerts watch for error-level logs, but carrier rejections are logged as routine events since they happen during normal operation. We also weren't tracking call failure rates on individual servers so our on-call engineers were paged in manually about an hour after the first failure. **Mitigations and Follow-ups** * We deployed a fix so SIP servers only use IPv4 addresses when determining their public address. * We'll add safeguards and automated tests to ensure a SIP server's call setup messages only contain IPv4 addresses. * We are additionally adding improvements in monitoring so that we can proactively detect and prevent these sorts of failures. We apologize for the disruption this caused. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

Updates as published by LiveKit (original).