Central Europe Region: Occasional incorrect resolution of some domains

The issue has been resolved.

Please note that full resolution may take effect only after the DNS TTL expires. To speed up recovery, we recommend clearing the cache on affected resolvers in the Admin Portal:

1. Navigate to the Resolvers tab.
2. Click the three dots next to the resolver.
3. Select Clear resolver cache.

Root cause and resolution

With the help of Microsoft engineers, they identified the root cause and removed the affected servers from the server group that was returning incorrect DNS responses. The issue was global, but it primarily impacted Central Europe due to the existing routing setup.

Preventative measures

As part of our preventative measures, we will implement a more robust mechanism for detecting and handling invalid DNS replies. This improvement will be included in the next resolver release. We recommend keeping your DNS resolver up to date at all times.

Incident Summary

Since Friday, December 19th, we have observed occasional incorrect resolution of a limited set of domains coming from Microsoft Azure authoritative servers. This incident occurs under the following conditions:

• On-prem resolvers located in Central Europe (CZ, SK, PL)
• Domains served by Microsoft Azure authoritative servers located in Central Europe
• Domains returning an incorrect CNAME response

Affected domains

Selected domains served by Microsoft Azure authoritative servers; we have observed the following set so far:

packeta.com, cz.linked.com, www.identita.gov.cz, nis.identita.gov.cz, ares.gov.cz, planeo.cz, login.eset.com, skoda-auto.sk, skoda-auto.cz

Mitigation steps

Immediate Mitigation

Clear the resolver cache:

• Navigate to Admin Portal → Resolvers → Three dots → Clear resolver cache

Mid-Term Mitigation

Set the default maximum TTL to 3600 seconds (1 hour):

• Navigate to Admin Portal → Configuration → DNS Resolution
• Select the configuration associated with your resolvers
• Open Advanced DNS configuration and apply:
  cache.max_ttl(3600)
  – For more details, see the Knot Resolver documentation
  – You may set the maximum TTL to lower values; however:
  — This may increase response latency for clients (fewer cached domains)
  — This may increase CPU load (higher frequency of contacting upstream servers)
  — Please monitor resolver performance closely if setting a lower TTL

Upcoming Fix

We are also preparing a hotfix for the resolvers to prevent this situation in the future. We will inform you once it is released.

Analysis & Root cause

During the investigation, we identified the following root cause.

Microsoft Azure authoritative servers located in Central Europe (to which your on-prem resolver may or may not be routed) are sending a non-standard (broken) response to CNAME queries:

• NOERROR — the query is understood and handled correctly
• NODATA — no DNS record is associated with the domain (likely a bug on Microsoft authoritative servers)
• Missing SOA (Start of Authority) record
• EDNS (Extended DNS) is set

Because the SOA record is missing from the response, a default TTL value of 32768 seconds (approximately 9 hours) is applied. As a result, incorrect records are cached for this duration until correct records are resolved.

As part of the mitigation process, your assistance in contacting Microsoft Azure Support with the details of the issue would be greatly appreciated.

Please accept our apologies for any inconvenience caused.