Major incident

Flexera System Status Dashboard: Spot Ocean - Intermittent Connectivity Issues

Affected: Spot API

Current status of Flexera System Status Dashboard: Operational — live Flexera System Status Dashboard status

Timeline

1. Oct 7, 2026, 20:27 UTC
   investigating
   Incident Description: We are investigating reports of intermittent issues affecting Ocean Kubernetes customers. Some customers have reported receiving alerts related to connectivity between Ocean Kubernetes components and their clusters. Technical teams are actively investigating the scope of impact and any associated service degradation. Priority: P2 Restoration Activity: Our technical teams are actively investigating the reported behavior, reviewing customer reports, and validating the scope and impact across affected environments. We will provide additional updates as more information becomes available.
2. Oct 7, 2026, 22:23 UTC
   investigating
   We continue to investigate the intermittent connectivity issues previously reported. At this time, no additional occurrences have been observed since the original impact window. Technical teams are actively reviewing available telemetry and investigation findings to determine the underlying cause and validate the overall scope of impact. Further updates will be provided as more information becomes available.
3. Oct 8, 2026, 00:03 UTC
   investigating
   We have not observed any additional occurrences since the originally reported impact window, and service stability has been maintained. Technical teams continue to investigate the underlying cause of the intermittent connectivity issues reported earlier today.
4. Oct 8, 2026, 00:04 UTC
   monitoring
   Based on the sustained stability observed since the originally reported impact window and the absence of any additional reported occurrences, we are moving this incident to Monitoring. Technical teams continue to investigate the underlying cause while service stability is being monitored. Further updates will be provided as more information becomes available.
5. Oct 8, 2026, 02:24 UTC
   resolved
   We have not observed any additional occurrences since the original reported impact, and service stability has been maintained. Technical teams completed additional investigation into the intermittent connectivity issues reported earlier today. Multiple potential correlations were reviewed during the investigation, and final analysis is still pending. Based on the sustained stability observed and the absence of any recurrence, this incident is now considered resolved. Technical teams will continue monitoring and investigation activities to further understand the behavior observed during the reported impact window, and a formal RCA report will be provided once the analysis has been completed.
6. Oct 20, 2026, 17:50 UTC
   postmortem
   **Description:** Spot Ocean - Kubernetes - Intermittent Controller Connectivity Issues **Timeframe:** October 7, 2026, 11:20 AM PDT - 12:30 PM PDT **Incident Summary** On Wednesday, October 7, 2026, at 11:20 AM PDT, multiple Spot Ocean for Kubernetes environments experienced intermittent controller connectivity alerts. During the incident, the platform temporarily interpreted controller heartbeat information as outdated and suspended scaling operations for affected clusters as a safety measure. The affected controllers continued to report heartbeat information during the incident. However, the timestamp used to validate controller connectivity became outdated within a database component, causing active controllers to appear temporarily unresponsive. The behavior was intermittent, and current heartbeat information subsequently became available without manual intervention, allowing normal scaling operations to resume. Technical teams reviewed the controller heartbeat processing path, database behavior, service telemetry, and related infrastructure signals to determine why valid heartbeat activity was being interpreted as stale. No additional occurrences were observed after the incident window. Following continued monitoring and validation of service stability, the incident was considered resolved. **Root Cause** The incident was caused by an older controller heartbeat timestamp overwriting a more recent valid timestamp within a database component. Although the affected controllers continued reporting heartbeats normally, the outdated timestamp made the most recently stored heartbeat appear older than it was. When the platform evaluated this timestamp, it exceeded the configured heartbeat threshold and triggered a safety mechanism that temporarily suspended scaling operations for the affected clusters. The specific condition that delayed the older database update has not been conclusively established. However, the database update process did not contain a safeguard to prevent an older heartbeat timestamp from replacing a newer valid timestamp. This allowed delayed heartbeat information to create the appearance that actively reporting controllers were unresponsive. **Remediation Actions** The following actions were taken during the incident response: 1. Customer Reports and Service Behavior Reviewed: Technical teams reviewed reported examples to confirm the behavior, establish the impact window, and determine whether the issue was isolated or affecting multiple environments. 2. Controller Heartbeat Activity Validated: Controller activity was reviewed to determine whether the affected controllers had stopped reporting. The investigation confirmed that heartbeat information continued to be sent during the incident. 3. Heartbeat Processing Path Investigated: Technical teams traced how heartbeat information was generated, stored, and evaluated to identify where current heartbeat activity was being interpreted as outdated. 4. Database and Service Health Assessed: Relevant database performance, service behavior, and infrastructure telemetry were reviewed to evaluate potential processing delays and rule out broader platform instability. 5. Timestamp Regression Identified: The investigation determined that an older heartbeat timestamp had overwritten a newer valid timestamp, causing the platform to temporarily interpret the heartbeat as stale. 6. Service Stability Validated: Technical teams continued monitoring following recovery. No additional occurrences were observed, and scaling operations remained stable before the incident was closed. **Future Preventative Measures** The following improvements are planned to strengthen heartbeat processing and improve detection of similar conditions: 1. Heartbeat Timestamp Protection: The database update process will be modified to prevent an older heartbeat timestamp from overwriting a more recent valid timestamp. 2. Improved Diagnostic Logging: Additional lo

Updates as published by Flexera System Status Dashboard (original).