Major incident

FTAPI: Partial service unavailability

Affected: FTAPI Cloud Systems

Current status of FTAPI: Operational — live FTAPI status

Timeline

1. Aug 4, 2026, 12:50 UTC
   investigating
   We’re currently experiencing degraded performance issues with for a subset of customers . Our team is currently working to restore normal performance levels. We apologize for any inconvenience.
2. Aug 4, 2026, 13:05 UTC
   identified
   We have identified the likely cause of the current disruption affecting our platform. Our engineering team is applying a fix now. We expect services to be fully available again within approximately the next 30 minutes. We will post another update once the fix is confirmed, or sooner if the situation changes. Thank you for your patience.
3. Aug 4, 2026, 13:28 UTC
   identified
   The fix is taking longer than we anticipated in our previous update. Our team is still actively working on restoring full service, and the issue remains our top priority. We will post our next update within the next 30 minutes regardless of whether the issue is resolved by then. We apologise for the continued disruption and appreciate your patience.
4. Aug 4, 2026, 14:01 UTC
   identified
   Our team is still working on resolving the issue. There is no significant change to report since our last update. We will post the next update within 30 minutes, or sooner if the situation changes. Thank you for your continued patience.
5. Aug 4, 2026, 14:54 UTC
   monitoring
   The root cause has been identified and we have implemented a fix. We are currently seeing services having fully recovered. We are monitoring the situation closely to maintain stability.
6. Aug 4, 2026, 15:34 UTC
   resolved
   This incident has been resolved.
7. Aug 13, 2026, 07:25 UTC
   postmortem
   **Executive Summary**: On August 4, 2026, customers hosted in one of our production environments experienced a full service outage of approximately two hours \(14:40–16:43 CEST\). The outage occurred during planned, downtime-free infrastructure maintenance on our database nodes and was caused by a failure of the external DNS resolvers used during node bootstrap without technical connection to the maintenance work itself. As a result, newly provisioned nodes were unable to join the cluster and the database became unavailable. As of August 4, 17:34 CEST, all services have been fully restored to operational state. ## **Incident Details** As part of a planned infrastructure improvement, a configuration change was applied to the database nodes in the affected production environment. The change was designed to be downtime-free and was carried out under standard change procedures. During the maintenance, and without technical connection to it, name resolution through the external DNS resolvers configured for node bootstrap stopped working. Because newly provisioned nodes depend on this name resolution to register with the cluster, they were unable to join. Existing database components could therefore not be rescheduled, which made the database — and consequently all customer-facing services in that environment — unavailable. The configuration change was rolled back promptly, but recovery was not immediate: as long as name resolution remained unavailable, new nodes still could not join the cluster. The subsequent root cause analysis, carried out together with our infrastructure provider, showed that the failure of the external DNS resolvers had caused the outage and not our maintenance work. Once name resolution was restored, nodes joined the cluster, the database recovered, and all services returned to normal operation within minutes. ## **Timeline \(CEST\)** * 04.08.2026, 13:28 – 13:55: Planned, downtime-free configuration change applied to the database nodes of the affected environment. * 04.08.2026, ~14:40: Name resolution via the external DNS resolvers used for node bootstrap begins to fail. Database components become unavailable; newly provisioned nodes can no longer join the cluster. * 04.08.2026, 14:42: Incident detected internally and incident response process initiated. * 04.08.2026, 14:50: Status page updated – _Investigating_. * 04.08.2026, ~15:00: Rollback of the configuration change initiated; escalation to our infrastructure provider opened. * 04.08.2026, 15:05: Status page updated – _Identified_. * 04.08.2026, 15:30 – 16:20: Joint analysis with the engineering teams of our infrastructure provider; DNS resolution identified as the blocking factor for node bootstrap. * 04.08.2026, 16:38: Name resolution restored. * 04.08.2026, 16:44: Database available again; all services recover. * 04.08.2026, 16:54: Status page updated – _Monitoring_. * 04.08.2026, 17:34: Stability confirmed. Status page updated – _Resolved_. ## **Strategic Prevention & Corrective Actions** To reinforce the resilience of our platform infrastructure and prevent recurrence, we have initiated the following measures: ### **Resilience of DNS configuration for node bootstrap** * We are removing the dependency on the DNS service that failed and are distributing the risk by including several providers in the configuration of our node groups. * This ensures that the failure of a single external resolver can no longer prevent new nodes from joining the cluster. ### **Monitoring & Alerting** * We are adding dedicated monitoring for name resolution and for the successful bootstrap of new nodes, so that failures in this layer are detected and attributed immediately rather than as a downstream symptom. ### **Escalation path with our infrastructure provider** * We are establishing a defined fast-track escalation path with our infrastructure provider for platform-level incidents, in order to reduce the time to substantive engineering engagement.

Updates as published by FTAPI (original).