Major incident

Harness: Prod1: IACM Stages are failing and pipelines are intermittently failing to start.

Affected: Infrastructure as Code Management (IaCM)

Current status of Harness: Operational — live Harness status

Timeline

1. Oct 6, 2026, 21:46 UTC
   investigating
   We are currently investigating this issue.
2. Oct 6, 2026, 21:47 UTC
   investigating
   We are continuing to investigate this issue.
3. Oct 6, 2026, 22:03 UTC
   identified
   The issue has been identified and a fix is being implemented.
4. Oct 6, 2026, 22:30 UTC
   monitoring
   A fix has been implemented and we are monitoring the results.
5. Oct 6, 2026, 22:30 UTC
   monitoring
   A fix has been implemented and we are monitoring the results. we have confirmation of this fixing the issue.
6. Oct 6, 2026, 22:44 UTC
   resolved
   This incident has been resolved.
7. Oct 23, 2026, 02:12 UTC
   postmortem
   # 1. Summary On October 6, 2026, pipelines containing an IaCM stage were unable to start on Prod1 for approximately two hours. Affected users saw an error when the pipeline attempted to create its execution plan. Pipelines that did not use an IaCM stage continued to run normally throughout. The underlying cause was an internal logging defect in the service that handles IaCM stage execution: under a specific, rare timing condition, that service's own logging system could end up waiting on itself indefinitely, which in turn stalled the broader process responsible for preparing pipeline execution plans. The issue was triggered by an unusually large burst of log activity from one customer's pipeline. We resolved the immediate impact by rolling the service back to its previous stable version, and a permanent code fix is currently in testing. # 2. Impact * All pipelines using an IaCM stage on Prod1 were unable to start for approximately two hours. * Multiple customers on the Prod1 cluster were affected. * Pipelines that did not use an IaCM stage were not affected and continued running normally. * No data was lost as a result of this incident. * As a side effect of the rollback used to resolve this incident, an unrelated fix for one customer \(extending how long a certain authentication token remains valid during long-running pipelines\) was temporarily reverted. That fix was reapplied the same evening. # 3. Root Cause The service responsible for IaCM pipeline stages logs its activity through an internal queue: log messages are placed on the queue, and a single background worker continuously reads from it and writes the messages out. Separately, that service has a filter that can rewrite the severity of certain log messages back onto that same queue. Under a specific, narrow timing condition, this created a deadlock, if the queue happened to be completely full at the exact moment a rewritten message was being submitted back onto it, the one worker responsible for freeing up space on the queue would itself get stuck waiting for space to free up, a condition it could never resolve, since it was the only thing that could have freed that space. Once that happened, every other part of the service that needed to log anything also became stuck waiting, which cascaded into the pipeline-plan-creation process failing for any pipeline using an IaCM stage. # 4. Resolution * Rolled the service back to its last stable version, which resolved the immediate incident. # 5. Preventive Actions 1. **Optimize the underlying defect**: Optimize and tune the logging filter so that rewriting a message's severity happens on the thread that generated the log line, rather than being resubmitted through the same background worker's queue, eliminating the deadlock condition entirely. 2. **Enhance monitoring on logging-queue health**: add metrics and alerting on this internal queue so that this class of saturation pages our on-call team automatically rather than relying on customer reports. 3. **Evaluate limits and perform high stress tests with very large pipelines**:

Updates as published by Harness (original).