Critical incident

KnowBe4: KSAT - Unable to Login (US Only)

Affected: Console

Current status of KnowBe4: Operational — live KnowBe4 status

Timeline

1. Sep 17, 2026, 22:44 UTC
   investigating
   We have received reports that users are unable to login to the US instance of KSAT. We are investigating this issue and will update this page when we have more information.
2. Sep 18, 2026, 01:28 UTC
   monitoring
   We’ve implemented a fix and we’re monitoring the results to make sure no further issues occur.
3. Sep 18, 2026, 14:02 UTC
   investigating
   We have received new reports of users being unable to login to the US instance of KSAT. We are investigating this issue and will update this page when we have more information.
4. Sep 18, 2026, 16:47 UTC
   resolved
   This incident has been resolved.
5. Oct 5, 2026, 21:16 UTC
   postmortem
   ## **Summary** On September 17, 2026, a background job archived a large user group and ran as a single unbounded database transaction. This transaction locked a large number of user rows, causing US region logins to time out because they write to the same table. After hitting a platform execution limit, the job was terminated and automatically retried five times in 15-minute intervals, resulting in a 75-minute outage. To resolve the issue, we deployed a fix that decoupled logins from the lock. We then deployed a follow-up fix for a regression introduced by the login change, as well as for the archival job. ## **What Happened** A prior change moved the "archive all users" action from a synchronous, time-out web request into an unbounded background job. When a large user group was archived, this triggered the new background job. The job locked all affected user rows in a single transaction, performing per-user writes and a synchronous external service call before committing. Login attempts, which write to the same table, were queued behind the locked job and timed out for US users. When the job reached a 15-minute platform execution limit, it was terminated mid-transaction, the transaction rolled back, and the job was redelivered to the queue. This cycle occurred five times, resulting in a 75-minute outage. When failures were detected internally at 22:35 \(UTC\), an incident was declared, and an engineer was paged to respond. The mitigation to decouple the write action from the login flow was deployed at approximately 23:44 \(UTC\). The mitigation itself caused a regression: any user in any region who had previously mistyped a password could no longer log in because resets for failed login attempts tried to write to a read-only database. By 09:17 \(UTC\) on September 18, 2026, the fix for this regression was deployed across regions. The original US-only archive job issue recurred twice more between 13:45 and 14:05 \(UTC\) before a permanent fix was deployed at approximately 15:00 \(UTC\). The incident was fully resolved by 19:12 \(UTC\) on September 18, 2026. ## **Root Cause** An unbounded, single-transaction background job for large-group user archival held exclusive locks on many affected rows, blocking the login process's own row write. A prior change to move this operation off a web request timeout and into a background job removed the only remaining limit on its duration. The platform's 15-minute execution limit then triggered automatic kill-and-redeliver cycles, repeating the lock contention five times and resulting in a 75-minute outage window. ## **Findings and Mitigations** We found that the unbounded archive transaction locked a large number of affected user rows, blocking other table writes. This was mitigated in the short term by decoupling the login process's write dependency from the affected table, and then fixed in the long term by correcting the archive job. We also found that uncoordinated worker concurrency amplifies lock contention. This was mitigated in the short term by manually reducing worker concurrency. The long-term fix to stagger and serialize these jobs is planned. Finally, we found that the login-fix regression caused all-region login failures for users with prior failed attempts because writes were rejected by a read-only database. This was fixed via a targeted code change that was deployed and verified across all regions. ## **Preventive Measures** We’re improving our large-group archival operations by bounding them, processing them in small, committed batches, and keeping external calls outside the database transaction. We’re also adding a change-check to a specific background worker to skip writes that don't change anything. Finally, we’re adding direct alerts for database transaction age and purging delays, in addition to periodic indirect checks. ## **Customer Impact and Recovery** Login failures lasted approximately 75 minutes in the US region. Two brief, US-only recurrences occ

Updates as published by KnowBe4 (original).