Major incident

unico skill: Instability affecting ID CLOUD capabilities

Current status of unico skill: Operational — live unico skill status

Timeline

1. Jul 21, 2026, 14:30 UTC
   identified
   Dear customer, We are investigating an instability in our IDCloud capabilities, affecting the process creation and process details flows. Our engineering team is mobilized to resolve the issue as soon as possible. We will provide further updates shortly.
2. Jul 21, 2026, 14:46 UTC
   identified
   We are continuing to work on a fix for this issue.
3. Jul 21, 2026, 14:50 UTC
   monitoring
   Dear Customer, Our engineering team has identified and implemented the necessary actions to resolve the instability in the affected IDCloud flow capabilities. At this time, the technical team is conducting a detailed analysis of our data layer and continues to monitor the environment. We reiterate that further updates will be sent shortly.
4. Jul 21, 2026, 16:00 UTC
   resolved
   Executive Summary and Impact On July 21, 2026, at around 11:17 AM BRT, a drop in availability was detected in the IDCloud create-process and process-query flows. During the impact window, a portion of customers experienced elevated latency and failures (timeouts) in these operations, which propagated through the chain of dependent services. The effective impact was concentrated in a window of approximately 23 minutes. At peak, error and latency indicators were above normal operating levels. Root Cause and Resolution The investigation pointed to the degradation of a read database (replica) supporting these capabilities as the origin. This database began to show increased load and query execution time, with internal contention that stalled processing and backed up read requests. As a result, the services above it started accumulating timeouts and triggering protection mechanisms (circuit breakers), producing the cascading effect experienced by customers. Commitment and Next Steps Unico is handling this incident with a detailed postmortem, already being drafted, and will hold a post-incident review (PiR) meeting to consolidate preventive actions. Among the follow-ups already raised during the response are the preparation of diagnostic routines and queries to speed up the identification of contention in this database layer, and improved observability over the databases involved. The detailed preventive actions will be shared in the postmortem. We reaffirm our commitment to the stability and reliability of the platform and apologize for the inconvenience caused during the period of instability. Unico Team.
5. Jul 28, 2026, 12:10 UTC
   postmortem
   **Summary** On July 21, 2026, between 11:41 and 11:43 AM \(Brasília time\), the process creation flow became unavailable, with the main service error rate reaching 42% and p99 latency at 15.75 seconds. The incident was caused by saturation on the biometry database read replica: the database memory pool was no longer able to serve the most frequently accessed dataset, causing normally fast queries to fall back to disk reads, creating a self-reinforcing spiral of accumulated sessions and increasing latency. **Impact** Users of the process creation, authentication, and payments flows across multiple clients received errors and timeouts during the period. The primary database remained intact throughout the incident — the failure was isolated to the read replica. **Root Cause** The root cause was saturation: queries used in the authentication flow were performing extensive scans on critical tables due to missing indexes. Under normal conditions, this behavior was tolerated by the database infrastructure. When memory capacity reached its limit and could no longer keep the most accessed data in cache, each read began fetching data from disk, making the cost of each query exponentially higher. This abrupt collapse effect — where gradual degradation gives way to a widespread failure — became self-reinforcing as sessions accumulated waiting for responses and internal database resource contention increased. Additionally, all read traffic for the critical flow was concentrated on a single replica, with no distribution or alternative route, eliminating any resilience to this type of degradation. **Resolution** The database recovered naturally as the accumulated sessions drained, requiring no manual intervention. The team escalated to the database team and monitored service stabilization. The incident was closed after confirmation that normal operation had been restored. **Lessons Learned** The incident revealed structural gaps in the database layer: missing indexes on critical queries and concentration of all read traffic on a single replica with no alternative route. As follow-ups, the team prioritized creating the missing indexes, distributing read traffic across replicas, and building diagnostic runbooks to accelerate response in future incidents of a similar nature.

Updates as published by unico skill (original).