DARCH Get started

Notes ·

When the internet's root key changed, our outgoing mail waited nine hours

From about 19:00 UTC on 12 October 2026 until 04:20 UTC the next morning, the mail server DARCH runs for itself and its clients could not send mail. Nothing bounced and nothing was lost. The cause was the planned change of the DNS root zone's signing key the day before. This note says what happened, why, and what we changed.

What happened

  • Every message leaving our mail server was held with 4.4.3 Host or domain name not found. The server could not look up any name, Gmail's included.
  • At 04:10 UTC on 13 October the server's DNS resolver was restarted. From 04:20 UTC the held mail went out.
  • Mail servers keep retrying held messages for days, so every message arrived, late. One invitation to 56 people, sent that evening, reached all 56 between 04:20 and 05:30 UTC the next morning. Nothing bounced or expired.
  • Nothing alerted us at the time: no check watched the queue of outgoing mail. We found the delay on 14 October, reading a mail export.

Why

A mail server keeps its own DNS resolver, because spam blocklists refuse lookups that arrive through large public resolvers. Ours, part of the open-source Mailu mail server, checks DNSSEC signatures. It trusts the internet's DNS only as far as it trusts the root zone's key.

  • On 11 October 2026 the root zone began signing with its new key, KSK-2024 (key tag 38696), in place of KSK-2017 (key tag 20326). The new key had been published in the root zone since January 2025.
  • Our resolver ran from an image built on 30 September 2026. Its list of trusted keys held both, but the new one only as pending. Under RFC 5011 a resolver accepts a new key after it has watched it for 30 days: for this image, around 30 October.
  • Once the root zone was signed with the new key alone, our resolver trusted nothing. Its log repeated failed to prime trust anchor -- DNSKEY rrset is not secure, and every lookup failed.
  • Two things are not proven. Why the failure began a day after the change: a cached copy of the old key set running out would explain it. And why a restart cured it: after the restart, the resolver trusted both keys.

What we changed

  • A check now looks up a name from inside the mail server every five minutes and counts the outgoing queue. Failing lookups should raise an alert after about 15 minutes, and so should any message that has waited for over an hour. Both were tested against stand-ins; neither has been seen on the real server yet.
  • On 17 October we updated the mail server. The resolver in Mailu's current images holds both root keys as fixed anchors and refreshes them each time it starts.
  • The failure, its signs and the commands to check a resolver are in our runbooks.

If you run Mailu, Unbound or another validating resolver in a container

  • Check the keys the running resolver trusts, not the file in the image. In Unbound's own queries, _ta-4f66 means it trusts only the old key; _ta-4f66-9728 means both (4f66 and 9728 are 20326 and 38696 in hexadecimal).
  • Search its log for failed to prime trust anchor. Any line after 11 October 2026 means it has lost its way to the root.
  • An image built shortly before the change can carry the new key as pending, and a container recreated from it starts the 30-day wait again.
  • Watch the queue of outgoing mail, not only whether the mail ports answer. Ours answered all night.
  • The old key, KSK-2017, is due to be revoked in the first quarter of 2027. Check again then.

Sources

All notes