Major incident

STACKIT: SKE runtime impaired with cascades to other STACKIT services

Affected: Cloud Identity, Network Load Balancer, Kubernetes Engine, PostgreSQL Flex

Current status of STACKIT: Operational — live STACKIT status

Timeline

1. Jun 17, 2026, 14:21 UTC
   identified
   Deutsch Wir hatten eine Störung im SKE Service mit der Verfügbarkeit des Kubernetes API Servers. Unser Team arbeitet daran, die Ursache des Problems zu finden und zu beheben. Über unsere Statusseite werden wir Sie regelmäßig über die Entwicklungen informieren. Ihr STACKIT Team Der STACKIT Support steht Ihnen jederzeit zur Verfügung. English We have identified an issue with the SKE service regarding the availability of the Kubernetes API server. Our team is working to identify and resolve the issue. We will keep you regularly updated on developments via our status page. Your STACKIT team STACKIT Support is available to you at any time.
2. Jun 17, 2026, 14:35 UTC
   identified
   Deutsch Wir haben eine Störung im SKE Service mit der Verfügbarkeit des Kubernetes API Servers. Dies beeinträchtigt auch die Verfügbarkeit weiterer STACKIT Services, die SKE nutzen. Unser Team arbeitet daran, die Ursache des Problems zu finden und zu beheben. Über unsere Statusseite werden wir Sie regelmäßig über die Entwicklungen informieren. Ihr STACKIT Team Der STACKIT Support steht Ihnen jederzeit zur Verfügung. English We have identified an issue with the SKE service regarding the availability of the Kubernetes API server. This also impairs the availability of further STACKIT services using SKE. Our team is working on finding and fixing the cause of the problem. We will keep you regularly informed of developments via our status page. Your STACKIT team STACKIT Support is available to you at any time.
3. Jun 17, 2026, 15:13 UTC
   identified
   Deutsch Wir haben eine Störung im SKE Service mit Beeinträchtigung der Verfügbarkeit weiterer STACKIT Services, wie Load Balancer und Identity Provider. Das Problem wurde identifiziert und wir arbeiten mit Hochdruck an einer schnellen Wiederherstellung der Services. Ihr STACKIT Team Der STACKIT Support steht Ihnen jederzeit zur Verfügung. English We have identified an issue in the SKE service, which is affecting the availability of other STACKIT services such as Load Balancer and Identity Provider. The issue has been identified, and we are working urgently to restore the services quickly. Your STACKIT team STACKIT Support is available to you at any time.
4. Jun 17, 2026, 16:37 UTC
   identified
   We are continuing to work on a fix for this issue.
5. Jun 17, 2026, 16:43 UTC
   identified
   Deutsch Die Ursache der Störung wurde weitestgehend behoben. Einige PostgreSQL Flex Instanzen sind nach wie vor betroffen und wir arbeiten weiterhin an der vollständigen Entstörung. Ihr STACKIT Team Der STACKIT Support steht Ihnen jederzeit zur Verfügung. English The cause of the problem has mostly been resolved. Some PostgreSQL Flex instances remain affected, and we are continuing to work on a full resolution. Your STACKIT team STACKIT Support is available to you at any time.
6. Jun 17, 2026, 16:56 UTC
   monitoring
   Deutsch Die Ursache der Störung wurde behoben und wir beobachten die betroffenen Komponenten weiterhin intensiv. Sollten Sie ab sofort noch Beeinträchtigungen feststellen, setzen Sie sich bitte mit dem STACKIT Support in Verbindung. Ihr STACKIT Team Der STACKIT Support steht Ihnen jederzeit zur Verfügung. English The cause of the problem has been resolved and we are continuing to monitor the affected components closely. If you notice any further problems from now on, please contact STACKIT Support. Your STACKIT team STACKIT Support is available to you at any time.
7. Jun 17, 2026, 17:51 UTC
   resolved
   Deutsch Die Störung wurde behoben und es liegen keine Informationen über weitere Einschränkungen des Services vor. Wir haben die verursachende Komponente identifiziert und stellen in den nächsten Tagen eine Root Cause Analyse bereit. Wir entschuldigen uns für eventuell entstandene Unannehmlichkeiten und werden Maßnahmen umsetzen, die ein erneutes Auftreten der Störung verhindern. Ihr STACKIT Team Der STACKIT Support steht Ihnen jederzeit zur Verfügung. English The incident has been resolved and there is no information about further service disruptions. We have identified the component causing the issue and will provide a root cause analysis in the coming days. We apologize for any inconvenience caused and will implement measures to prevent the error from reoccurring. Your STACKIT team STACKIT Support is available to you at any time.
8. Jun 19, 2026, 17:22 UTC
   postmortem
   Between 15:43 and 18:20 CEST time on 17 June 2026, an outage occurred affecting the connection between the worker plane and the control plane of all STACKIT Kubernetes Engine \(SKE\) clusters in the regions eu01 and eu02. The outage affected all STACKIT Kubernetes Engine clusters in the eu01 region, whereas in eu02 only a fraction of the clusters were impacted for a significantly shorter duration. The incident occurred during an attempted rollback of a STACKIT Kubernetes control plane version update deployed on 15th June 2026. As this update had enabled a load balancing feature that was causing issues for a limited subset of customers, a hotfix was applied to rollback said feature. However, the subsequent hotfix operation failed to correctly revert the associated Layer 7 traffic handling configurations. During the rollout of the hotfix, the system partially deleted the new routing settings before the original configuration was fully applied. Due to this incorrect cleanup, the availability of the kube-apiserver was impaired, interrupting large parts of the communication between the Kubernetes control plane and the worker nodes. As a result, a complex split-brain scenario emerged: while the central control plane components functioned mostly normal, the kubelets on the cluster nodes were completely cut off and could no longer access the API servers. Consequently, the API servers assumed the nodes were unhealthy. This caused the Kubernetes service load balancing to remove the endpoints \(Pods\) from all service routing. However because of the specific connection interruptions the split-brain scenario occurred in, the kube-proxy, the agent on the nodes applying these service load balancing rules, were still able to observe and apply the routing changes. This resulted in interruption of most pod-to-service and external-to-NodePort \(from internet, e.g., STACKIT Network Load Balancer\) traffic in the data plane. The platform's built-in "meltdown protection" mechanism - designed to freeze the routing state and prevent the deletion of healthy nodes after a broad disconnect between data and control plane - failed to trigger in time. Because the cluster nodes were left isolated after the routing information was updated by the kube-proxies, all services utilizing the STACKIT Kubernetes Engine were severely impacted. This led to widespread service disruptions which also affected STACKIT internal services \(e.g., STACKIT Portal, STACKIT Help Center Login, STACKIT IDP, STACKIT Observability, and various STACKIT Database services\). The STACKIT Kubernetes Engine engineering team successfully resolved the disruption via manual intervention by stabilizing the central management plane and restoring all cluster routing states. As the amount of impacted clusters was so large that concurrently repairing them would have caused a control plane overload; this had to be done sequentially, prolonging the recovery time. All clusters returned to 100% health by 18:20 CEST. No data loss or corruption occurred.

Updates as published by STACKIT (original).