Major incident

Cornerstone: High Event queue count in US SL3

Affected: Response Time

Current status of Cornerstone: Operational — live Cornerstone status

Timeline

1. Sep 30, 2026, 16:08 UTC
   investigating
   US SL3 swim lane is experiencing delays in processing background or queued tasks. During this time, background tasks, including email processing, may take longer than usual to complete. Resolving this issue is our top priority, and we are working to fix it as quickly as possible. Please check back periodically for updates, which will be posted as they become available.
2. Sep 30, 2026, 16:39 UTC
   resolved
   CSOD Technology Team observed delays in processing background or queued tasks on US SL3 swim lane including emails. The problem began on 09/30/2026 06:27 AM - PDT and was resolved on 09/30/2026 09:20 AM - PDT. During this time, background tasks may have taken longer than normal to complete.
3. Oct 22, 2026, 20:53 UTC
   postmortem
   **Issue Summary:** On September 30, 2026, the US SL3 Production environment experienced delays in processing queued operations, resulting in an unusually high volume of events. Affected operations were unable to process at the expected rate, causing events to accumulate in the queue and delaying completion of activities. **Root Cause:** The issue was caused by database deadlocks occurring during concurrent data merge operations. When these conflicts occurred, the application attempted to retry the affected operations without sufficiently reducing the batch size of learning objects being processed. This resulted in repeated database conflicts, preventing queued events from being processed efficiently and causing an accumulation of events. **Corrective Action:** Cornerstone implemented a fix to improve the application's retry logic and database deadlock handling. The update addressed repeated processing conflicts by adjusting how failed operations are retried, allowing data merge operations to be processed more efficiently. **Preventive Measures:** To reduce the risk of a similar issue, Cornerstone has taken the following actions: * Enhanced retry handling to reduce batch sizes when processing conflicts occur, minimizing the likelihood of repeated database deadlocks. * Improved error handling to ensure database deadlocks are properly identified and managed, preventing repeated attempts to process the same failed operations.

Updates as published by Cornerstone (original).