Minor incident

CircleCI: Increased wait time for Machine and Remote Docker jobs

Affected: Machine Jobs

Current status of CircleCI: Operational — live CircleCI status

Timeline

1. Nov 3, 2026, 14:14 UTC
   investigating
   Some users might see increased wait times for Machine and Remote Docker jobs due to increased usage. Wait times right now are just under 7 mins. Our team is investigating and we'll post another update shortly. We appreciate your patience.
2. Nov 3, 2026, 14:45 UTC
   monitoring
   Our team has applied a fix and we're starting to see wait times recover. We're keeping an eye on this as it continues to rollout and we'll update again shortly. We thank you again for your patience.
3. Nov 3, 2026, 15:08 UTC
   resolved
   Wait times have returned to normal for Machine and Remote Docker. We appreciate your patience as we worked through this.
4. Nov 4, 2026, 22:14 UTC
   postmortem
   ## Summary From 13:55 to 15:07 UTC on November 3, customers running Linux machine jobs and Remote Docker saw increased wait times for their jobs to start, and the delays were longest from 14:16 to 14:41. Some Docker jobs also saw increased wait times. The incident was caused by our usage reaching the vCPU limit on our cloud provider account in our main cloud region. Machine and Docker jobs share this limit, and the additional Docker capacity we have run since the [October 21 incident](https://discuss.circleci.com/t/post-incident-report-oct-21-2026-increased-wait-times-for-gen-2-docker-jobs/54940) brought our usage to the limit as demand increased. Machine jobs did not move to our backup regions because our system did not treat reaching this limit the same way as a shortage of cloud instances. In response, our engineering team reconfigured machine jobs to balance across all of our regions instead of relying on backup regions, and reduced Docker capacity to make room under the limit. By 14:46, wait times had improved, and by 15:07, machine wait times had fully recovered. After we resolved the incident, our cloud provider doubled the vCPU limit in our main region. We have also restored the Docker capacity we reduced during the incident. The original status page can be found here: [Increased wait time for Machine and Remote Docker jobs](https://status.circleci.com/incidents/qqh50c0z6mn3). ## Background Machine jobs need a dedicated virtual machine, so when CircleCI runs a machine job, we create a fresh virtual machine for that job and delete it when the job ends. Remote Docker runs on these same virtual machines. Docker jobs run on shared cloud instances, and each instance can run multiple isolated jobs. Our cloud provider sets a vCPU limit for each region, which caps the total computing capacity our account can run there. Machine and Docker jobs run in the same account, so they count against the same vCPU limit. When our usage reaches that limit, the cloud provider rejects requests for new cloud instances until existing ones shut down. Machine jobs normally run in our main region, with other cloud regions available as backups. If the cloud provider cannot supply the instance types we need in our main region, our system sends new work to a backup region. Following the October 21 incident, we increased the minimum fleet size for our Docker clusters during weekday peak hours, so that capacity is in place before demand arrives. That capacity counts against the same vCPU limit as machine jobs. ## What Happened _\(All times UTC\)_ Starting at 13:55 on November 3, Linux machine jobs saw increased wait times because our usage in our main region had reached our vCPU limit, and our cloud provider was rejecting requests for new virtual machines. We had not noticed that our alert for this limit was no longer receiving data, so it did not warn us as our usage approached the limit. At 13:55, our engineering team was alerted to errors launching new virtual machines and began investigating. By 14:06, 40% of Linux machine jobs waited more than 3 minutes to start. By 14:08, we had confirmed the cause and requested a higher limit from our cloud provider. Our backup regions had capacity available under their own limits, but machine jobs did not move there because our system did not treat reaching our vCPU limit the same way as a shortage of cloud instances. From 14:16 to 14:41, most Linux machine jobs waited 5-9 minutes to start, and some waited more than 20 minutes. At 14:34, our engineering team reconfigured machine jobs to balance new work across all of our regions, rather than only using backup regions when our main region runs short. The change took effect at 14:40, and at the same time we reduced the minimum fleet size for our Docker clusters to make room under the limit. By 14:46, wait times had improved, and nearly all Linux machine jobs started within 3 minutes. Delays of 1-3 minutes for some jobs continued until 15:07. We

Updates as published by CircleCI (original).