Critical incident

iPaper: Networking outage

Affected: Viewer, Viewer, Admin, Admin, Backend API, Pop-ups, Auxiliary Sites, PDF Processing Service, SSL Processing Service, Email Processing Service, Feed Processing Service

Current status of iPaper: Operational — live iPaper status

Timeline

1. Sep 13, 2026, 09:37 UTC
   investigating
   We are currently investigating this issue.
2. Sep 13, 2026, 09:41 UTC
   monitoring
   Everything is now operational again - we are still investigating the root cause.
3. Sep 13, 2026, 09:52 UTC
   resolved
   Issue is resolved as this was a side effect of a previous fix. We will be deploying a permanent fix for this soon.
4. Sep 14, 2026, 07:45 UTC
   postmortem
   ## Summary We recently implemented stability fixes for an ongoing issue with our VXLAN, where the IPsec protocol occasionally desynchronizes and causes one-way loss of communication between nodes. These fixes exposed an edge case in key expiration. ## Impact All internal networks were affected for between 41 seconds and 2 minutes. ## Cause A temporary fix was implemented on Friday to prevent keys from rolling every few hours and keep the system stable. However, it caused the system to clean up in-use keys once they reached 24 hours of age. Because of the fix, no new keys were available to take over, so all secure communication was lost while new keys were automatically renegotiated. ## Resolution Connectivity was restored as the software automatically rolled new keys across all servers. The issue has been fixed in the latest version and should not happen again.

Updates as published by iPaper (original).