Major incident

Phrase: Degraded Performance of Branching in Phrase Strings (EU) between August 6, 2026 03:45 PM CEST and August 7, 2026 10:46 AM CEST

Affected: Translation center, API

Current status of Phrase: Operational — live Phrase status

Timeline

1. Aug 7, 2026, 07:27 UTC
   investigating
   We are currently investigating an issue with branching events missing in Phrase Strings (EU), which may cause changes to not be applied when branches are merged.
2. Aug 7, 2026, 07:54 UTC
   investigating
   We are continuing to investigate this issue.
3. Aug 7, 2026, 08:18 UTC
   identified
   The issue has been identified and a fix is being implemented.
4. Aug 7, 2026, 08:30 UTC
   monitoring
   A fix has been implemented. The backlog of missing events is now being processed. Once the queue is fully processed, unmerged branches will automatically receive the missing changes. We are monitoring the queue and results.
5. Aug 7, 2026, 14:12 UTC
   resolved
   Changes made to branches during the incident window are now fully applied. Merged branches that were affected have been identified and impacted customers have been contacted directly. This incident has been resolved.
6. Sep 11, 2026, 06:11 UTC
   postmortem
   ## Introduction We would like to share details about an incident that affected Phrase Strings on the EU platform between August 6 and August 7, 2026. During this period, changes made in branches were not being recorded by our change-processing pipeline. As a result, branches that were merged during the incident could be merged without the changes made to them during that window, and branch comparison screens could show no differences even when differences existed. This post-mortem explains what happened, how it was resolved, and what we are doing to prevent it from happening again. ## Timeline * **August 6, 2026 at 5:44 PM CEST** – A network interruption severed the connection our change-processing pipeline uses to read database changes. The connection failed to recover, and branch changes stopped being processed. The affected component continued to report itself as healthy, so no automated alert was triggered. * **August 7, 2026 at 09:11 AM CEST** – The issue was identified following a customer report. * **August 7, 2026 at 09:19 AM CEST** – The incident was escalated to our highest severity level. * **August 7, 2026 at approximately 10:20 AM CEST** – The affected component was recovered by restarting it, and change processing resumed. * **August 7, 2026 at 10:46 AM CEST** – The backlog of pending changes was fully processed and the pipeline was caught up. * **August 7, 2026 afternoon** – The affected customers were identified and contacted individually with the data needed to restore their changes. ## Root Cause The incident was triggered by a transient network interruption that broke the connection our system uses to stream changes from the database into our branch-processing pipeline. The component responsible for this connection attempted to reconnect automatically, but the reconnection attempt hung indefinitely rather than either succeeding or failing. Because the component never reported a failure, our platform's automatic recovery mechanism — which restarts components that report an error state — was never triggered. The component remained in a state where it appeared to be running normally while processing no changes at all. This is why a standard restart of the connection did not resolve the issue, while fully restarting the underlying process, which established a fresh connection, did. The situation was not detected sooner because our monitoring for this pipeline relied on a signal that continued to report "healthy" throughout the incident, and we did not have a separate measure of whether changes were actually flowing through and being processed. ## Impact * Branches that had changes during the incident but were **not** merged did not lose any data. Their changes were processed once the pipeline recovered. * Branches that **were** merged during the incident were merged without the changes made to them during the incident window. Changes made before the incident were not affected. * Only a small number of customers were ultimately affected. Each was contacted individually and provided with the data and steps needed to restore the affected changes. ## Actions to Prevent Recurrence 1. **Fixed misleading monitoring** – We corrected the monitoring configuration that grouped different production environments together, which had delayed our awareness of the issue. 2. **Adding meaningful health signals** – We are adding monitoring that measures whether changes are actually flowing through the pipeline and how far behind it is, rather than relying on a status signal that can report "healthy" while no work is being done. This ensures a similar stall is detected immediately. 3. **Improving automatic recovery** – We are reviewing the reconnection and retry behaviour of this component so that a stuck connection results in a genuine failure that triggers automatic restart.

Updates as published by Phrase (original).