Major incident

Socure: RETROACTIVE: [Sev 2] OTP Verifications Degradation

Current status of Socure: Operational — live Socure status

Timeline

1. Jun 15, 2026, 04:00 UTC
   resolved
   On June 15, 2026, between 12:15 AM and 1:00 AM EDT, the OTP Verification endpoint experienced elevated error rates. The system recovered automatically at 1:00 AM EDT. No action is required from customers. Our investigation confirmed the issue was transient. All systems are operating normally. A Root Cause Analysis (RCA) will be published once our investigation is complete.
2. Jun 22, 2026, 21:39 UTC
   postmortem
   ## **Root Cause Analysis \(RCA\)** **Incident:** Elevated 5XX errors on the OTP Verification endpoint **Date Range:** June 15, 2026 — 12:15 AM ET to 1:00 AM ET **Impact:** Customers calling the Socure OTP Verification endpoint \(otp/send\) on the ID\+ production platform or RiskOS workflows experienced HTTP 5XX errors during the incident window, resulting in failed OTP verifications. The endpoint auto-recovered without manual intervention, and no customer action was required. **Status:** Resolved ### **1. Summary** On June 15, 2026, between approximately 12:15 AM and 1:00 AM ET, Socure’s OTP Verification endpoint \(otp/send\) on the ID\+ production platform experienced an elevated rate of HTTP 5XX errors. OTP requests failed or were degraded during this 45-minute window. The endpoint auto-recovered at 1:00 AM ET with no manual intervention required. A concentrated surge in OTP verification volume placed sustained load on Socure’s OTP service, which coordinates OTP generation, delivery, and verification. The service is configured to scale horizontally, but the additional replicas did not successfully come up during the surge window. The existing instances became saturated and restarted repeatedly under load, and resulted in HTTP 5XX errors for the OTP Verification endpoint during this period. Initial investigation considered upstream messaging-provider behavior as a potential contributor; subsequent log and metric analysis confirmed that vendor-side rate limiting was not a material factor during the window, and identified the internal OTP service scaling failure as the primary cause. The service self-recovered as upstream demand returned to baseline. Socure engineering has identified a set of corrective and preventive actions — described below — to address the underlying scaling behavior, strengthen capacity baselines for elevated traffic events, and improve observability around scaling-related saturation conditions. ### **2. Timeline** | Time \(ET\) | Event | | --- | --- | | 12:15 AM | Elevated HTTP 5XX error rates begin on the OTP Verification endpoint \(otp/send\) in the ID\+ production environment. | | 12:17 AM – 12:28 AM | Initial high-volume burst observed on the OTP Verification endpoint; OTP verification request volume rose sharply, placing concentrated load on the MFA Orchestrator service that coordinates OTP delivery and verification. | | 12:41 AM | Incident formally declared and opened within Socure’s incident management system. On-call engineering paged; incident response team assembled. An investigation bridge was convened and telemetry for the OTP service was reviewed in parallel. | | ~12:45 AM | Initial investigation considered upstream messaging-provider behavior as a potential contributor; no broad vendor outage reported on the public status page. Investigation continued in parallel into the health of Socure-owned services in the OTP path. | | 1:00 AM | Upstream OTP verification request volume subsided; service load on the OTP service returned to capacity, and error rates on the OTP Verification endpoint auto-recovered to normal levels. No manual mitigation was required. | | Post Incident | Incident moved to post-incident review; engineering continued investigation and impact assessment to confirm scope of affected accounts and validate the upstream cause. | | June 16 | Detailed log and metric analysis completed. Primary contributor identified as a horizontal scaling failure in Socure’s OTP service: the configured autoscaling did not produce additional replicas during the surge, the existing instances became saturated, and the majority of the observed errors traced to OTP service response failures. Upstream messaging-provider rate limiting was ruled out as a material factor. Investigation into the underlying autoscaling behavior is ongoing with Site Reliability Engineering. | ### **3. Root Cause** **Primary Root Cause:** A concentrated surge in OTP verification request volume placed a sustained loa

Updates as published by Socure (original).