Major incident

TimeEdit: Partial service outage affecting all customers

Affected: Curriculum, Curriculum, Workload, Workload, Scheduling, Scheduling, REST API, SOAP API, System Services, Reserve, Reporting, Data Manager, Rest API, SOAP API, System Services, Allocate

Current status of TimeEdit: Operational — live TimeEdit status

Timeline

1. Mar 30, 2026, 10:10 UTC
   investigating
   We are currently investigating this issue.
2. Mar 30, 2026, 10:19 UTC
   resolved
   This incident has been resolved.
3. Apr 30, 2026, 08:19 UTC
   postmortem
   ## **Description** Our Viewer product was targeted by a worldwide botnet scraping schedule data, which overloaded one of our production servers and caused intermittent service disruptions. The mitigations required to stop the scraping have temporarily disabled `.txt` and `.csv` exports in the EU and AUS regions. ## **What happened** On Monday, 30th March 2026, traffic to one of our production servers began spiking at 11:39 CET, and by 12:04 customers were reporting 502 Bad Gateway errors when loading Viewer and Core. Investigation showed that a distributed botnet, using thousands of forged user-agent strings and routing traffic primarily through Brazil, India, and the US, was scraping schedule data from Viewer at very high volumes. The acute outage was resolved by 13:11 once the server recovered, but the underlying scraping continued. Geo-based rate limits were insufficient once additional traffic origins were identified, so a broader block on `.txt` and `.csv` requests at the load balancer was applied at 15:01. This mitigation remains in place and currently affects legitimate export functionality for customers in the EU and AUS regions. ## **What we're doing to prevent this in the future** To reduce the risk of similar disruptions, we are implementing several measures, including: * Replacing the temporary block on `.txt` and `.csv` exports with more targeted protections, so export functionality can be restored as soon as possible. * Strengthening bot protection on Viewer with behavioural detection that does not rely on geography or user-agent strings, which forged traffic can easily bypass. * Reducing the number of requests issued on the initial Core page load to limit cascading effects when reload activity increases during partial outages. * Adding alerting on anomalous export volumes and sustained request rates, so prolonged scraping attempts are detected earlier.

Updates as published by TimeEdit (original).