Major incident

Wurl: SSAI Service Impact 2026-05-15 23:00  UTC to 2026-05-20 2:00  UTC

Affected: SSAI

Current status of Wurl: Operational — live Wurl status

Timeline

1. May 21, 2026, 18:20 UTC
   monitoring
   We are currently investigating an issue affecting our Server-Side Ad Insertion (SSAI) service that impacted a subset of customer accounts. The issue occurred between May 16 and May 19. During this time, some accounts experienced service degradation. At this stage, no outbound communications have been sent as our teams are still assessing the full scope and impact of the incident. We are continuing to work closely with all relevant parties to fully understand the issue and its impact. Further updates will be shared as soon as more information becomes available
2. May 27, 2026, 17:10 UTC
   resolved
   Service Update: Incident Resolved Summary: We have successfully resolved the performance degradation affecting the SSAI (Server-Side Ad Insertion) system. Services were fully restored as of May 20, 2026, at 02:00 UTC. Detailed Findings: The incident, which began on May 15, was caused by a performance degradation in a third-party vendor's tracking and polling workflows. This led to a backlog in processing queues, resulting in delayed or dropped downstream tracking operations and a noticeable decline in the Use Rate (the ratio of impressions to ads returned). Resolution and Prevention: * Full Restoration: Normal impression tracking was confirmed after the backlog was processed and queue consumption rates returned to baseline levels. * Resiliency Measures: Wurl is working with the vendor to implement expanded monitoring for queue depth and consumer lag, along with lower-threshold alerting to ensure earlier detection of similar events in the future. Support: If you have any questions regarding this incident or the attached report, please reach out to our team at [email protected].
3. May 27, 2026, 17:11 UTC
   postmortem
   # **Degraded SSAI Event** ‌ **Date:** 2026-05-26 **Start:** 2026-05-15 23:00 UTC **End:** 2026-05-20 2:00 UTC **Issue Description**On May 15th, 2026, at approximately 23:00 UTC, the SSAI system operated by a third party vendor started experiencing periods of performance degradation. This resulted in intermittent failures across tracking and polling workflows, reducing the system’s ability to consume tracking topics at the expected rate. Consequently, messages accumulated in the processing queue, creating a backlog that delayed and periodically dropped downstream tracking operations and impacted overall system latency and reliability. Service recovery efforts began at approximately May 20th 01:00 AM UTC, at which point queue consumption rates started returning to normal. Full restoration of services, including backlog processing, was completed by approximately 2:00 AM UTC. # **Issue Timeline:** | Date / Time \(UTC\) | Event | Summary | | --- | --- | --- | | 2026-05-15 23:00 | Degradation Begins | Degradation started. Sporadic/intermittent decreases in overall impression tracking continued through May 18th. Initial degradation was minimal and did not trigger alerting. Subsequent significant degradation on May 18th remained below Wurl alerting thresholds. | | 2026-05-19 14:00 | Major Failure | Significant degradation in impression reporting and tracking, resulting in a noticeable drop in overall impressions. | | 2026-05-19 21:13 | Detection | Wurl alerting triggered and was reviewed, identifying a substantial decrease in impression volume. Issue escalated to partner. | | 2026-05-19 21:50 | Initial Response | Engineering team acknowledged the issue and began triage and investigation. | | 2026-05-20 02:00 | Resolution | Services fully restored and normal impression tracking confirmed. | The plot below shows the significant degraded performance periods:  ‌ # **Issue Impact** Impact was observed when measured by Use Rate = Impressions / Wins. The affected periods show a reduction in realized impressions versus wins compared with healthy baseline periods. This indicates that, during the incident windows, demand-side winning opportunities did not convert to impressions at the normal rate, resulting in measurable lost impression delivery. Wurl calculated the impression impact by using the historical use rate for each channel and inventory owner and applying it to the ads returned during the larger time window from May 14 to May 20. # **Corrective Actions** In response to this incident, Wurl is working with the third party vendor on new resiliency and operational safeguards designed to improve early detection, reduce recovery time, and prevent similar queue backlogs from occurring in the future. ‌ These measures include: * Expanded monitoring and alerting around queue depth, consumer lag, processing throughput, and service health metrics to allow for earlier identification of degradation conditions. * Additional redundancy and recovery protections within the tracking services to improve fault tolerance during transient failures or performance degradation events. * Additional use-rate-focused alerting with lower thresholds. * Refining automated alert thresholds and escalation workflows to proactively notify engineering teams before service impact reaches critical levels.

Updates as published by Wurl (original).