Minor incident

CircleCI: Delays for gen2 dedicated IP range jobs

Affected: Docker Jobs

Current status of CircleCI: Operational — live CircleCI status

Timeline

1. Oct 28, 2026, 18:15 UTC
   investigating
   We are investigating delays in starting docker gen2 jobs that use dedicated IP ranges. Jobs are currently delayed by about 10 minutes. No other resource classes are impacted at this time. We will continue to provide updates as more information is available.
2. Oct 28, 2026, 18:51 UTC
   identified
   We’ve identified the cause as limited compute availability from our cloud provider, and we are working to free up additional capacity for these jobs to restore normal start times. We'll provide another update again shortly.
3. Oct 28, 2026, 19:06 UTC
   monitoring
   We are starting to see signs of recovery now, wait times are starting to return to normal. We're keeping an eye on this and we'll update again soon.
4. Oct 28, 2026, 19:19 UTC
   resolved
   Wait times for gen2 dedicated IP range jobs have fully returned to normal. We appreciate your patience while worked through this issue.
5. Oct 31, 2026, 00:22 UTC
   postmortem
   ## **Summary** From 14:20 to 19:08 UTC on October 28, customers running Gen 2 Docker jobs using IP Ranges saw increased wait times for their jobs to start. From 14:20 until 17:02, the delays were intermittent, and from 17:02 to 18:50, the delays were continuous. No other job types were affected. The incident was caused by limited availability of the cloud instances we had configured for the IP Ranges cluster. In response, our engineering team added additional fallback instance types to the IP Ranges cluster and transferred capacity from one of our other Gen 2 clusters to the IP Ranges cluster. By 18:50, wait times had improved, and by 19:08 they had fully recovered. After we resolved the incident, we increased the number of baseline cloud instances we maintain for IP Ranges jobs. We have also configured an IP Ranges cluster in an additional cloud region. The original status page can be found here: [Delays for gen2 dedicated IP range jobs](https://status.circleci.com/incidents/8q9j33r2rssy). ## **Background** IP Ranges is a CircleCI feature that runs jobs from a published static list of IP addresses, so customers can configure their firewalls to allow those addresses. Gen 2 Docker jobs using IP Ranges need to run on their own dedicated cluster of cloud instances, separate from our other Gen 2 Docker jobs. Customers rely on the published IP address list, so we give 30 days’ notice before adding new IP addresses. Because of that notice period, we cannot move IP Ranges jobs to a new cloud region on short notice. When CircleCI runs a Docker job, we schedule it on one of several clusters of cloud instances, and each instance can run multiple isolated jobs. We provision cloud instances using a ranked list of instance types. If the cloud provider cannot supply our first choice, we fall back to the next type on the list. Each request tries our preferred types first, so when several of them are unavailable, each request takes longer and the cluster grows more slowly. We also maintain a minimum fleet size: a minimum number of instances that stay running, so that capacity is ready before demand arrives. Above that minimum, we add instances as demand rises. When there is insufficient capacity to run a job, the job waits in a queue until capacity becomes available. ## **What Happened** _\(All times UTC\)_ Starting at 14:20 on October 28, Gen 2 Docker jobs using IP Ranges saw intermittent delays because the instance types we had configured for the IP Ranges cluster were not available in the quantities we needed. Wait times rose to 5 minutes and then returned to normal several times until 17:02. From 17:02, the delays were continuous. Our wait time alerts did not specifically isolate IP Ranges jobs, so these delays did not immediately alert our engineering team. At 17:22, our engineering team began investigating and confirmed that the IP Ranges cluster could not provision sufficient instances. An unrelated incident at our observability provider from 16:10 to 18:39 also slowed our investigation by affecting dashboards and metric queries. Following the [October 21 incident](https://discuss.circleci.com/t/post-incident-report-oct-21-2026-increased-wait-times-for-gen-2-docker-jobs/54940), we added fallback instance types and additional cloud regions for our main Gen 2 clusters, but the IP Ranges cluster had not yet received those changes. Our engineering team had scheduled work for October 30 to configure the IP Ranges cluster with additional fallback instance types, but that work had not begun at the time of this incident. Because of the 30-day notice, we could not immediately move IP Ranges jobs to another region, so our engineering team reallocated capacity from other CircleCI clusters in the same region. By 18:50, the IP Ranges cluster had provisioned sufficient instances, and wait times began returning to normal. Delays of 1-2 minutes continued until 19:08. We resolved the incident at 19:19. ## **Future Prevention and Process

Updates as published by CircleCI (original).