Minor incident

Harness: Single Customer: IaCM pipeline executions failing intermittently in Prod 1

Affected: Infrastructure as Code Management (IaCM)

Current status of Harness: Operational — live Harness status

Timeline

1. Oct 6, 2026, 11:58 UTC
   investigating
   We are currently investigating this issue.
2. Oct 6, 2026, 11:59 UTC
   investigating
   We are continuing to investigate this issue.
3. Oct 6, 2026, 12:24 UTC
   identified
   The issue has been identified and a fix is being implemented.
4. Oct 6, 2026, 16:21 UTC
   resolved
   This incident has been resolved.
5. Oct 9, 2026, 18:48 UTC
   postmortem
   ## Summary On **Tuesday, 6 October 2026**, IaCM pipelines that use the **OpenTofu plugin** in **Prod1** failed intermittently for a production accountAffected runs injected secrets successfully, then later steps timed out or failed authentication. The same pipeline often succeeded on retry, with no configuration change. Typical failing executions ran for about 90 minutes. Other Harness modules were not broadly impacted. We have no report of customer data loss. Pipelines that finished inside the token lifetime were unaffected by this defect. ## Root Cause The IaCM stages minted authentication tokens that expired after one hour. Policy checks later in a long run then rejected the token, so the step timed out or failed authentication. That one-hour lifetime was fixed. Runs of about 90 minutes crossed the one-hour mark, which is why the failures showed up on longer pipelines and often cleared on a shorter retry. A separate set of plan errors in the same account came from credential setup in the customer workspace and is outside this incident. ## Impact * **Product area:** IaCM pipelines using the IaCM plugins talking to OPA sdk. * **Customer experience:** intermittent timeouts and authentication failures after secret injection, on runs longer than about one hour. * **Environment:** Prod1, one production account * **Data:** No data loss reported. ‌ ## Remediation * **Immediate:** Opened a status page for the affected Prod1 account, and shipped an interim IaCM manager change for one class of the observed failures while the plugin fix was built. * **Permanent:** Shipped an OpenTofu plugin hotfix so token expiry is configurable. In production the lifetime was raised to about 6 hours, so a long pipeline does not outlive the token. The hotfix was validated in QA and deployed to Prod1. We apologise for the disruption caused. Please reach out to your Harness support contact if you have any further questions.

Updates as published by Harness (original).