Critical incident

Hosted Mender: Mender Server EU issues with micro tier devices

Affected: Hosted Mender EU

Current status of Hosted Mender: Operational — live Hosted Mender status

Timeline

1. Jul 9, 2026, 18:08 UTC
   investigating
   We are currently investigating the root cause of the issue.
2. Jul 10, 2026, 06:33 UTC
   monitoring
   We found a possible root cause with the new intermediate certificate of https://eu.hosted.mender.io with a different size. A tested workaround is to increase the mbedTLS heap size in your application's prj.conf: CONFIG_MBEDTLS_HEAP_SIZE=98304.
3. Jul 10, 2026, 11:21 UTC
   resolved
   This incident has been resolved by tuning the mbedTLS heap size.
4. Aug 28, 2026, 08:24 UTC
   postmortem
   ## **Executive Summary** An issue was identified where microcontroller \(MCU\) devices, specifically those running Zephyr OS and `mender-mcu` on constrained hardware like the ESP32-S3, lost connection to the hosted Mender EU server \(`eu.hosted.mender.io`\). Specifically, devices failed during the initial TLS handshake with error `-0x2700`, preventing Over-The-Air \(OTA\) updates and inventory reporting. The investigation confirmed that the root cause was **insufficient client-side memory \(mbedTLS heap\)** when processing the server's RSA-4096 certificate chain, rather than a certificate validity or time-synchronization issue. To recover devices already deployed in the field without requiring a physical intervention, the engineering team deployed a **server-side mitigation by switching the production TLS certificate authority to Google Trust Services \(GTS Root R4\)**. This restored connectivity for affected field devices. ## **Impact** Customers using MCU devices targeting the EU instance were not able to perform updates from the 9th to the 15th of July. ## **Technical Root Cause** The hosted Mender EU endpoint utilized a cross-signed Let’s Encrypt certificate hierarchy ending in an RSA-4096 leaf and root certificate. From the 4th of June, letsencrypt \(EU\) started using a different chain from the same root CA, with one more intermediate. _Jan – early Jun 2026 │ leaf → R12/R13 → ISRG Root X1 --> 1 intermediate_ _Jun 4 2026 → now │ leaf → YR1 → ISRG Root YR → ISRG Root X1 --> 2 intermediates_ This change landed at the beginning of July to hosted Mender EU, after automated cert-bot renewal, and then the mender-mcu clients lost connection. During the TLS handshake, mbedTLS requires continuous heap memory to simultaneously hold: * The TLS input and output content buffers \(`SSL_MAX_CONTENT_LEN`\). * The parsed peer certificate chain \(RSA-4096\). * The parsed trust anchors. * The temporary mathematical buffers required to execute an RSA-4096 signature verification. On constrained MCU devices, the default mbedTLS heap configuration \(`CONFIG_MBEDTLS_HEAP_SIZE=40960`, or ~40 KB\) was exceeded. Our tTesting revealed that processing this specific RSA chain required a minimum heap threshold of ~41.5 KB \(42496 bytes\). When the memory allocation failed internally during signature verification, mbedTLS surfaced the out-of-memory exception as a general certificate trust failure \(flag 0x08: `BADCERT_NOT_TRUSTED`\). A mitigation was published, to increase the `CONFIG_MBEDTLS_HEAP_SIZE`, but of course this can’t be applied to devices which lost the connection to the server. ## **Resolution & Recovery** ### **Immediate Field Recovery \(Server-Side\)** Because deployed devices with broken OTA connectivity cannot receive a firmware update to increase their local heap size, engineering deployed a server-side resolution: * Switched the production certificate chain for [eu.hosted.mender.io](http://eu.hosted.mender.io) from Let's Encrypt to **GTS Root R4 \(Google Trust Services\)**. * The GTS Root R4 chain is already bundled within the existing MCU client trust stores and requires significantly lower memory overhead during verification, allowing already-deployed devices to successfully complete the handshake and re-establish OTA connectivity without a client-side patch. * A previous attempt to force a lighter CA Chain with ECDSA root CA failed; although the required heap was less than the new RSA root, it was still heavier than the old RSA root, so our internal validation failed. ### **Long term solutions** * We are implementing an end-to-end failover strategy covering both TLS infrastructure and domain name \(DNS\) routing. * We implemented periodic tests to emulate a MCU device connecting to both US and EU instances to early catch this kind of issues. ‌ We sincerely apologize for the disruption this caused to your fleets, this was a serious failure, and we understand the operational burden it placed on you. We are

Updates as published by Hosted Mender (original).