Major incident

Socure: RiskOS Service Degradation

Affected: RiskOS

Current status of Socure: Operational — live Socure status

Timeline

1. Jun 23, 2026, 02:56 UTC
   investigating
   We are currently experiencing errors affecting our RiskOS APIs. Our team is actively investigating the issue and will provide updates as more information becomes available.
2. Jun 23, 2026, 03:07 UTC
   monitoring
   Our team has mitigated the issue that was causing errors with the RiskOS APIs. We are seeing error rates return to normal levels and are continuing to monitor the system closely. We will provide additional updates if there are any further changes.
3. Jun 23, 2026, 03:24 UTC
   monitoring
   We are continuing to monitor our services and are not observing any errors at this time. We will provide additional updates if there are any further changes.
4. Jun 23, 2026, 03:29 UTC
   resolved
   The issue impacting RiskOS has been resolved, and service has returned to normal operation. Customers should no longer experience errors, degraded performance, or delays when accessing RiskOS services and workflows. Our teams have confirmed system stability and will continue to monitor performance closely. We appreciate your patience throughout this incident and apologize for the impact this outage may have had on your operations.
5. Jun 26, 2026, 21:07 UTC
   postmortem
   **Root Cause Analysis** **Incident:** RiskOS Evaluation API Elevated 502 Errors **Date Range:** June 22, 2026, 10:23 PM EDT – 10:59 PM EDT **Impact:** Customers calling the Socure RiskOS Evaluation APIs in production experienced an elevated rate of HTTP 502 \(Bad Gateway\) responses for approximately 36 minutes. **Status:** Resolved ## **1. Summary** On June 22, 2026, between approximately 10:23 PM and 10:59 PM EDT, Socure’s RiskOS Evaluation APIs in the production environment experienced an elevated rate of HTTP 502 errors. The disruption was caused by a misconfiguration in the aggregations engine — a component that enriches Evaluation API processing by computing velocity based signals across entity data. The misconfiguration was introduced during a configuration update as part of Local Graph GA readiness, and was not caught by existing validation controls. When the service attempted to use this configuration during evaluation processing an unexpected runtime exception occurred that triggered automated service restarts. As the affected services cycled, backend capacity became intermittently unavailable, causing requests during those periods to receive HTTP 502 \(Bad Gateway\) responses. Socure's automated monitoring detected the elevated error rate at **10:24 PM EDT** and the Engineering team began investigation immediately. The team mitigated the issue by disabling aggregations processing to stabilize traffic, remediated the invalid configuration, and validated recovery before re-enabling aggregations. Evaluation APIs returned to normal operation by **10:59 PM EDT**. Aggregations were re-enabled after validation at approximately **11:13 PM EDT**. Requests that received HTTP 502 responses during the incident window were not processed and no corresponding evaluation was created. Customers who experienced HTTP 502 responses between 10:23 PM and 10:59 PM EDT on June 22, 2026 should verify any intended evaluations and resubmit them if necessary. ## **2. Timeline** | **June 22, 2026 / Time** | **Event** | | --- | --- | | **10:22 PM EDT** | A configuration change in Entity Aggregations introduced an invalid value. | | **10:23 PM EDT** | Elevated HTTP 502 error rates begin on RiskOS Evaluation APIs in the production environment. | | **10:24 PM EDT** | Automated monitoring detected the elevated error rate and the on-call Engineering team was notified. Investigation began immediately. | | **10:25 PM to 10:40 PM EDT** | Engineering observed multiple servers failing intermittently. The team reviewed logs and traced the failures across affected servers to isolate the root cause. | | **~10:40 PM EDT** | Engineering identified that the issue was related to an aggregations configuration affecting Evaluation API processing. | | **~10:45 PM EDT** | Engineering began targeted mitigation, initially isolating specific aggregations-dependent components. Instability persisted, and the team proceeded to disable aggregations processing entirely to stabilize traffic. | | **10:55 PM EDT** | Aggregations processing was fully disabled. Evaluation API traffic stabilized. | | **10:59 PM EDT** | Evaluation APIs returned to normal operation. Elevated 502 errors were resolved. | | **10:59 PM to 11:13 PM EDT** | Engineering identified and remediated the specific invalid aggregation configuration. Recovery was validated against the corrected configuration. | | **11:13 PM EDT** | Aggregations re-enabled following successful validation of the corrected configuration. | ## **3. Root Cause** #### **Primary Root Cause:** An invalid configuration was applied to the aggregations engine. When the service attempted to use this configuration during Evaluation processing, it triggered an unhandled failure path inside the processing pipeline. Rather than gracefully rejecting the invalid configuration, the failure path caused service-level crashes and the affected servers entered a restart cycle. While the affected services were cycling, the RiskOS E

Updates as published by Socure (original).