Major incident

Sardine AI: Elevated latency on multiple APIs in PROD-EU

Affected: Device APIs, Customer APIs, Business API, Feedback API

Current status of Sardine AI: Operational — live Sardine AI status

Timeline

1. Oct 1, 2026, 14:49 UTC
   investigating
   We are investigating elevated latency and increased response times affecting Customer API, Feedback API and Business API requests in our PROD-EU environment. Our engineering team is actively investigating, and we will provide an update as soon as we have more information.
2. Oct 1, 2026, 15:39 UTC
   investigating
   We are continuing to investigate this issue.
3. Oct 1, 2026, 16:39 UTC
   identified
   A mitigation has been applied and latency is improving. We are continuing to monitor and will post another update later.
4. Oct 1, 2026, 17:03 UTC
   monitoring
   A fix has been implemented and we are monitoring the results.
5. Oct 1, 2026, 19:58 UTC
   resolved
   The elevated latency and increased response times affecting the APIs in PROD-EU have been resolved. Response times have returned to normal levels and have remained stable since the fix was applied. We will continue to monitor the environment closely. A root cause analysis (RCA) will be shared once it is complete. We apologize for the disruption.
6. Oct 9, 2026, 14:29 UTC
   postmortem
   ## Summary On 1 October 2026, from approximately 14:20 to 15:45 UTC, our EU environment experienced increased latency and timeouts on the `/v1/customers`, `/v1/issuing/risks`, and `/v1/feedbacks` endpoints. We added service capacity to end the main customer-impact period and subsequently deployed a code change that resolved the underlying cause. ## Customer impact Customers using the affected EU endpoints experienced slow responses, errors, and timeouts. The verified affected endpoints were `POST /v1/customers`, `POST /v1/issuing/risks`, and `POST /v1/feedbacks`. Other measured endpoints did not show comparable latency increases. The US environment and other environments were not affected. ## What happened and why Under peak EU traffic, a service supporting these endpoints allocated memory faster than it reclaimed it. Part of this work involved unnecessary database reads and writes to calculate a crypto-recipient feature even when the request contained no crypto recipient address. The service's memory management was also not configured to account for its operating memory limit. As instances reached that limit, they became unhealthy and restarted. Their traffic moved to the remaining instances, increasing the load and causing further restarts. The EU environment had insufficient spare capacity, and our automatic scaling settings did not add capacity quickly enough to contain the failure. Our monitoring did not give us a clear, sustained signal of the underlying problem. Some latency alerts cleared on their own and were not thoroughly investigated, and we lacked dedicated alerts for this service's memory pressure and repeated restarts. A customer report escalation brought the disruption to our attention. We also mistook a temporary improvement for recovery while requests were still affected. ## Resolution At 15:33 UTC, we increased service capacity in the EU environment. Tail latency across all three affected endpoints returned to its usual level at approximately 15:42 UTC, and service restarts had stopped by approximately 15:45 UTC, ending the main customer-impact period. This capacity increase was a mitigation rather than the root-cause fix. At 17:59 UTC, we deployed a code change that skips unnecessary crypto-recipient database operations when no recipient address is present. Post-deployment measurements showed a 79% reduction in memory use, a 40% reduction in CPU time, and a 67% reduction in garbage-collection activity. This change eliminated more than 99% of the unnecessary database reads and writes and resolved the underlying memory-allocation cycle. ## Timeline \(UTC\) All events below occurred on 1 October 2026. * 14:20: Customer-impact window began. * 14:39: Our on-call engineer received and acknowledged a latency alert for this incident. * 14:44: A customer reported the disruption to us. * 15:03: We declared an incident and began a coordinated response. * 15:33: We increased service capacity. * 15:42: Tail latency across the affected endpoints returned to its usual level. * 15:45: The customer-impact window ended; service restarts had stopped and the expanded capacity was ready. * 17:59: We deployed the root-cause code fix to skip unnecessary database operations. ## Prevention and action items * We have added capacity in the EU and raised the memory available to the service. * We have changed the service so it manages memory against its real limit. * We are correcting the automatic scaling trigger so it adds capacity before instances fail. * We are adding alerts for memory pressure, repeated restarts and out-of-memory terminations, and per-client latency so we can detect this type of issue ourselves and notify customers first. * We deployed the root-cause code fix that skips unnecessary database operations when no crypto-recipient address is present. * We are continuing work to reduce memory use per request for high-volume business workloads.

Updates as published by Sardine AI (original).