Minor incident

KnowBe4: Prevent and Protect Degraded Performance (US Instances)

Affected: Mail Flow, Mail Flow

Current status of KnowBe4: Operational — live KnowBe4 status

Timeline

1. Sep 28, 2026, 21:10 UTC
   investigating
   We are currently investigating an issue regarding degraded performance for Prevent and Protect on our US servers.
2. Sep 29, 2026, 13:44 UTC
   monitoring
   We have identified an issue with the US instances of Prevent and Protect that has resulted in degraded performance. A fix has been implemented, and we will continue to monitor this issue.
3. Oct 2, 2026, 10:32 UTC
   resolved
   This incident has been resolved.
4. Oct 6, 2026, 15:51 UTC
   postmortem
   ## **Summary** From Monday, September 28, 2026, at approximately 16:00 \(UTC\), to Wednesday, September 30, 2026, at around 12:35 \(UTC\), some customers experienced delayed mail flow and intermittent problems accessing US instances of Prevent and Protect. On September 28, our monitoring showed heavy network traffic, with spikes at the top of each hour. Some customers began experiencing intermittent problems accessing US instances of the Prevent and Protect consoles. Internal lookups that mail processing depends on started timing out, and messages were queued as a result. Customers reported the issue to our support team, and we posted a status page notice. Performance recovered later that day as US traffic decreased. On September 29, our engineers reviewed the previous day's data across the mail gateways, the internal services they depend on, and the network layer. Mail flow delays and console access issues returned briefly, then eased as network traffic declined, following the same pattern as September 28. On September 30, from approximately 12:19 to 12:35 \(UTC\), our team increased the capacity of the US network services while service continued. Their load dropped immediately after the change, and Prevent and Protect returned to normal performance by around 12:35 \(UTC\) on September 30, 2026. We monitored peak traffic over September 30 and October 1 and saw no recurrences. No data loss occurred as a result of this issue. ## **Root Cause** This issue was caused by a capacity limit in the network services that handle traffic for our US mail-processing services. During peak traffic, and especially during the hourly spikes, the network services couldn't process connections fast enough. This added delay to traffic passing through them and caused some connections to drop. That traffic included internal requests our mail gateways send to look up distribution list members, so the right policies are applied. These requests are subject to a strict time limit so that a slow lookup can't hold up mail processing indefinitely. When the network services delayed requests beyond that limit, they failed. As a result, mail processing slowed, and messages were queued. Two factors made the issue worse and harder to identify. First, some internal requests also used these network services and were affected by the capacity limit. Second, the network services began to degrade before their compute use reached levels that would normally signal a problem. Standard compute monitoring didn't catch the network saturation, so the cause took longer to identify. Finding a safe way to increase capacity without interrupting service also took time, which is why the fix was applied on September 30. ## **Mitigation and Remediation** We found that the US network services didn't have enough capacity to handle peak traffic. We've mitigated this by increasing their capacity in a way that kept service running throughout the change. After the capacity increase, the network services handled peak traffic without further issues. We also found that compute use alone wasn't a reliable sign of network saturation. We've expanded our monitoring to track network health indicators in addition to compute use. Finally, we found it was difficult to find where failing internal requests broke down. We've added logging that shows where internal requests fail and time out. ## **Customer Impact** During peak US traffic on September 28 and 29, some customers experienced delays in processing and delivering inbound and outbound mail through Prevent and Protect. Messages addressed to distribution lists may have been processed more slowly while lookups were retried. Some customers also had intermittent difficulty logging in to or using the US console. The impact eased each day as US traffic decreased, and queued mail was processed once capacity recovered. After the change on September 30, we saw no further degradation during peak traffic. ## **Preventative Measures** To p

Updates as published by KnowBe4 (original).