Critical incident

LiveKit: Observability upload and Agent Simulations failures

Affected: Cloud Dashboard (cloud.livekit.io)

Current status of LiveKit: Operational — live LiveKit status

Timeline

1. Oct 13, 2026, 10:01 UTC
   investigating
   We are investigating an issue with Agent Insights where agents are unable to upload their observability data at the end of their sessions. Some recent agent sessions may be missing from the dashboard's sessions list or shown as “In progress”. Agent sessions themselves are not affected. We will post an update shortly.
2. Oct 13, 2026, 10:45 UTC
   identified
   We have identified the cause of the Agent Insights issue and are working on a fix.
3. Oct 13, 2026, 11:40 UTC
   identified
   We are continuing to work on a fix for this issue.
4. Oct 13, 2026, 13:15 UTC
   identified
   Agent Insights uploads are now operational and session data has been storing normally since 12:40 UTC. We are still investigating an issue with Simulations. Simulation runs may complete without a transcript, summary, or verdict. We will post an update shortly.
5. Oct 13, 2026, 13:44 UTC
   resolved
   This incident has been resolved. Between 08:15 and 12:35 UTC, agents were not able to upload agent observability data to LiveKit's servers in all regions. Agent sessions completed normally, but, unfortunately, any Agent Insights data which attempted to upload within that window (traces/transcripts/logs/recordings) is unrecoverable. The ability to download existing Agent Insights data was impaired until 13:15 UTC. Agent Simulations were unavailable due to a upstream dependency and came back online at 13:25 UTC. Sincere apologies for the impact. We will follow up with a postmortem as soon as possible.
6. Oct 21, 2026, 23:40 UTC
   postmortem
   **Root Cause** During a security rotation of access credentials, we revoked a storage credential that we believed was no longer in use. It was still being used by our observability platform, which stores the data behind the dashboard and the Analytics API, the Agent Insights data that agents upload at the end of each session, and by the service that serves that data back for downloads and Agent Simulations. Once the credential was revoked, uploads failed in all regions. Restoring service took longer than a credential swap normally would for three reasons: * New credentials had to be set up, and each affected service had to be updated to use them. * The new credentials took longer than expected to become available in all regions \(about two hours in our US East region\). * Once the observability platform was working again, we found additional services that also needed updating. The incident was resolved once they were updated. **Timeline \(UTC\)** * 2026-10-13 08:13 - A storage credential is revoked during a security rotation. Observability uploads begin failing in all regions. * 08:15 - Agent Insights transcript and audio downloads and Agent Simulations begin failing. * 09:39 - Our team detects the failures and opens an incident. * 10:45 - The revoked credential is identified as the cause. * 11:16 - Affected services are reconfigured to use a new credential. * 12:40 - The new credential takes effect in the last region. Observability uploads recover in all regions. * 13:13 - Agent Insights downloads recover. * 13:26 - Agent Simulations recover. * 13:30 - Incident resolved. * 2026-10-17 - Recovery of observability data from the impact window completes \(see below\). **Scope of Impact** * **Observability uploads \(08:13 to 12:40 UTC, all regions\):** Observability data written during this window, including the data behind the dashboard and the Analytics API, failed to store. Nearly all agent sessions that ended in this window also failed to upload their Agent Insights data, so in Agent Insights these sessions were missing from the sessions list, or shown as "In progress". Agent sessions themselves ran normally, and media and calls were not affected. * **Data recovery:** In our resolved update we said all Agent Insights data from this window was unrecoverable. We have since restored most of it. **Session recordings and transcripts from this window are permanently lost.** All other observability data for these sessions has been restored and is visible in the dashboard and through the Analytics API. * **Agent Insights downloads \(08:15 to 13:13 UTC\):** Downloading transcripts and audio for existing sessions failed during the incident window. * **Agent Simulations \(08:15 to 13:26 UTC\):** Simulation runs couldn't load conversation transcripts, so they finished without a summary or verdict, or were cancelled. Runs from this window need to be re-run. * **Billing:** Agent Insights usage is only recorded after a successful upload, so failed uploads were **not** billed. **Detection and Response** We detected this issue about 85 minutes after it began. We did not have alerting on upload failures to observability storage, so it was caught when an engineer noticed elevated errors rather than by an automated alert. We also initially marked the incident resolved once uploads recovered, before confirming that Agent Simulations and downloads had recovered too. **Mitigations and Follow-ups** * We have added alerting that pages our on-call team when observability storage uploads fail, along with a continuous check that stored data can still be read and downloaded and an end-to-end test that verifies our storage credentials still work. * We have built tooling to replay and restore observability data from a past window, which we used to recover data from this incident. * We are moving these services to dedicated service credentials managed with our infrastructure and auditing other services for the same pattern. * We have updated our cr

Updates as published by LiveKit (original).