Critical incident

Socure: [SEV 1] Outage Resulting in 5xx Errors

Affected: Sigma Identity Fraud, Global Watchlist Screening with Monitoring, Watchlist Standard, Sigma Synthetic Fraud, Watchlist Plus, KYC, Watchlist Premier, Email RiskScore, Address RiskScore, Phone RiskScore

Current status of Socure: Operational — live Socure status

Timeline

1. Oct 15, 2026, 22:39 UTC
   investigating
   We are currently experiencing an active incident that is impacting service availability. We are currently investigating the issue and will provide updates as soon as we have additional information.
2. Oct 15, 2026, 23:04 UTC
   monitoring
   Socure RiskOS and ID+ services have been restored. An error in a rollout caused a temporary spike in error rates from our services between 18:20 and 18:56 Eastern time. The error has been corrected and services are stable now. We understand this outage is unacceptable and a full RCA will be available as soon as full analysis of the issue is complete.
3. Oct 15, 2026, 23:04 UTC
   monitoring
   Socure RiskOS and ID+ services have been restored. An error in a rollout caused a temporary spike in error rates from our services between 18:12 and 18:56 Eastern time. The error has been corrected and services are stable now. We understand this outage is unacceptable and a full RCA will be available as soon as full analysis of the issue is complete.
4. Oct 15, 2026, 23:08 UTC
   resolved
   Socure RiskOS and ID+ services have been restored. An error in a rollout caused a temporary spike in error rates from our services between 18:12 and 18:56 Eastern time. The error has been corrected and services are stable now. We understand this outage is unacceptable and a full RCA will be available as soon as full analysis of the issue is complete.
5. Oct 17, 2026, 01:30 UTC
   postmortem
   ## **Root Cause Analysis** **Incident:** Socure APIs Outage **Date Range:** 15-Oct-2026 **Impact:** Interruption in service for Socure APIs **Status:** Resolved ### **1. Summary** On 15-Oct-2026, between 18:12 and 19:00 Eastern time Socure experienced an interruption to its RiskOS and ID\+ APIs, causing errors and timeouts to users. The first phase ran from 18:12 to 18:20, and was a major interruption for virtually every service. The second phase of the outage ran from 18:20 to 19:00, and was a partial outage for services that rely on KYC data, where customers saw elevated error rates and latencies for some services, but traffic was still being served. ### **2. Timeline** **Date / Time \(ET\)** | Event **18:12** | A routine cleanup change is merged, moving a source repo from one code repository to another. A bug in our review tool hides the scope of the change. **18:13** | ArgoCD restarts most services in a major Kubernetes cluster all at once. **18:20** | All service pods are running. Partial API service restored, although KYC services are still seeing an elevated error rate. **18:21** | The flood of startup traffic overwhelms our KYC Search service. Error rates spike. **18:42** | SRE diagnoses the cascading failure impacting KYC Search and blocks traffic to it so it can come up. **18:56** | The KYC Search service is fully up, and SRE returns traffic to it. Downstream services stabilize. **19:00** | All traffic running as normal, incident declared stable. ### **3. Root Cause** * **Primary Root Cause:** As a part of an ongoing migration, an on-call SRE made a change to ArgoCD, the system we use to manage the kubernetes clusters Socure runs its services on. The change was scoped to a single cluster, was reviewed carefully before pushing, caused no issues in other environments it was tested in, and should have had no visible impact. Unfortunately, the change updated a legacy setting that did not appear in the change review. That legacy setting governed what services ArgoCD managed, and removed many live services from ArgoCD. In consequence, ArgoCD removed those services from production briefly, before the setting was corrected. * **Contributing Factors:** The KYC Search service is a backend piece of Socure infrastructure that provides services to several of our internal products. When that service restarted from scratch at the same time as all of the clients of the service did, a small number of pods in the service crashed under load. This increased the load on the remaining pods, some of which crashed in turn. This cascading failure prevented the service from coming back on its own the way the rest of Socure’s infrastructure did. Diagnosing this separate failure mode took additional time, and remediating it fixed the remaining errors after the initial incident. ### **4. Resolution** The ArgoCD change was reverted immediately, causing services to come back to their prior state. The KYC Search service required manual effort to block traffic from reaching it until a majority of pods in the service could come up. After the service was healthy, we could restore traffic to it and resolve the overall incident. ### **5. Corrective and Preventive Actions** ### **6. Lessons Learned** * Socure’s tooling for viewing the changes it makes to production did not work correctly. We are reviewing these tools to ensure they operate correctly under all use cases and fix any cases like this where the full scope of a change is not clear. * Augment the existing change review process with agentic support. While in this case, the specific change was not visible in the plan, having an extra reviewer that is supported by an extensive training model will help ensure issues are caught before submission. * Even changes that appear to have no impact, when they are made to the governance system for our production jobs, can be risky, and should be shifted further out of US business hours. * Our KYC Search service is vulnerable to

Updates as published by Socure (original).