Major incident

Socure: RETROACTIVE: RiskOS API Degradation

Current status of Socure: Operational — live Socure status

Timeline

1. Jun 16, 2026, 07:30 UTC
   resolved
   On June 16, 2026, from 03:45 AM to 04:03 AM EDT, RiskOS APIs experienced elevated error rates. The issue is fully resolved and all systems are now operating normally. The degradation occurred during remediation work tied to an networking error alert. A subset of API requests returned 5XX errors during this window, though the service remained partially available throughout. No customer action is required. A full Root Cause Analysis (RCA) will be shared once the investigation concludes.
2. Jun 19, 2026, 15:40 UTC
   postmortem
   **Root Cause Analysis** **Incident:** RiskOS API Services Degradation During NAT Gateway Remediation **Date Range:** 06-16-2026 **Impact:** Intermittent degradation for a subset of API requests **Status:** Recovered ## **1. Summary** On June 16th, 2026, RiskOS API services experienced intermittent degradation for a subset of API requests between **3:46 AM EDT and 4:02 AM EDT**. During this period, some requests failed while network connectivity was being transitioned as part of a controlled remediation. Earlier that day, Socure observed elevated connection error alerts and identified critically high Cloud NAT port utilization in the RiskOS commercial environment. The condition was not impacting existing customer transactions at the time of detection, but it created a significant capacity risk for outbound connectivity if additional workloads restarted, scaled, or deployed. To reduce the risk of broader impact, Socure performed a controlled network remediation during a lower-workload window. The remediation improved capacity handling and reduced shared NAT resource contention across origin IP blocks. During the transition, some existing connections were briefly interrupted and had to be re-established, resulting in intermittent request failures. The remediation activity completed in approximately 5 minutes, and overall recovery and stabilization completed within approximately 15 minutes. No customer action was required after connectivity was restored. ## **2. Timeline** | **June 16th,** **2026 / Time** | **Event** | | --- | --- | | 12:00 AM EDT | The Socure Engineering team observed elevated connection error alerts and began an investigation. No customer transaction impact was identified at this time. | | 12:30 AM EDT | Investigation identified critically high Cloud NAT port utilization, creating a capacity risk for outbound connectivity. SRE confirmed that existing customer transactions were not being impacted at that time. | | 12:30–3:45 AM EDT | The Socure Engineering team evaluated remediation options, assessed the risk of leaving the configuration unchanged, and planned remediation during a lower-workload window to reduce potential customer impact. | | 3:45 AM EDT | The engineering team proceeded with the planned network capacity remediation. | | 3:46 AM EDT | Intermittent request failures began as some existing network connections were briefly interrupted during the transition to the updated configuration. | | 3:52–4:00 AM EDT | Network configuration updates were applied to improve capacity handling and reduce shared NAT resource contention across origin IP blocks. | | 4:02 AM EDT | Intermittent request failures ended after traffic completed transition to the updated configuration and connections were re-established. | | ~4:02 AM EDT | Platform connectivity stabilized. | | ~4:15 AM EDT | Incident considered resolved after validation and monitoring confirmed recovery. | ## **3. Root Cause** **Primary Root Cause:**The incident was caused by critically high Cloud NAT port utilization in the RiskOS commercial environment. The NAT configuration had insufficient available port capacity to provide safe headroom for normal connection churn, workload restarts, scaling activity, or deployments. **Contributing Factors:** * The residual NAT capacity risk was a consequence of emergency recovery activities undertaken after the June 9 RiskOS service outage, where the primary focus was restoring service availability. * NAT port utilization reached approximately 99–100%, creating a significant risk of outbound connectivity degradation. * NAT capacity was shared across origin IP blocks, which increased the likelihood of port exhaustion under workload pressure. * During remediation, some existing connections were reset while traffic transitioned to the updated network configuration, resulting in temporary intermittent request failures. ## **4. Resolution** Socure updated the NAT configuration to improve capacity handling an

Updates as published by Socure (original).