DNS
Most DNS change alerts are not incidents
A full run of DNS change alerts with nothing actionable in it. What the noise really is, why the obvious fix is dangerous, and how to spot the real change.
We went looking through a run of DNS change alerts across every domain we were monitoring, expecting to triage a handful of real changes out of the pile. There were none. Not one alert in the run described something a person needed to do anything about, and several of the same domains had been alerting every morning for days with near identical change counts.
The monitoring was doing exactly what it was built to do. That turned out to be the problem.
What the alerts were actually reporting
Every change in that run was an A or AAAA record whose address had moved to a different
address inside the same provider’s published range. Nothing else had moved. No nameserver
changes, no MX changes, no TXT changes, no CNAME retargeting. Just addresses shuffling
within Cloudflare, within Amazon CloudFront, and within Google’s hosted front ends.
If a site sits behind a CDN, its A and AAAA records do not point at the site. They point
at the CDN’s edge, which is anycast, and the provider moves those addresses around whenever
it suits them. IPv6 rotates especially freely. A monitor that treats every address change as
a change to your DNS will therefore report the CDN doing its job, forever, most days
of the week.
That is not a false positive in the strict sense. The record genuinely changed, and the alert was telling the truth. It was just telling the truth about something nobody can act on.
Why this is worse than it sounds
An alert that fires daily and never means anything does not stay in your inbox. It gets a rule, and it goes to a folder, and it stops being read. Then the one alert that describes a nameserver being repointed at three in the morning lands in that same folder, unread, behind forty of its harmless siblings.
This is the ordinary way monitoring fails. It rarely misses the event. It has usually trained you to ignore the channel the event arrives on. If you are paying for DNS monitoring and you cannot remember the last time an alert changed what you did that day, the monitoring is not protecting you, whatever its uptime graph says.
The obvious fix, and the trap in it
The fix is to recognise provider edge ranges and stop paging a human about rotation inside them. Both major providers publish the data you need. It is the exact ranges you choose that decide whether this makes you safer or blinds you.
With CloudFront, use the CLOUDFRONT service only. AWS publishes ip-ranges.json with a
service label on every prefix. The tempting one is AMAZON, because it is the superset and it
matches more. It is also the wrong one, because it contains EC2, and EC2 is where customer
servers live. Classify AMAZON as trusted CDN edge and you have just built a monitor that
stays quiet when somebody repoints your domain at a virtual machine they rented this morning.
With Google, take goog.json and subtract cloud.json. Google documents this derivation
for working out its own default domain ranges. What remains is Google’s own edge and hosted front ends. What you
removed is Google Cloud, which again is customer virtual machines. Same trap, same
consequence.
The general shape of the rule: an address range only counts as edge if the provider, and not one of the provider’s customers, controls what is served from it.
Even then, edge classification only tells you the provider has not changed. It says nothing about
the account. Anyone can point your apex at their own CloudFront distribution and the addresses will
still sit inside the CLOUDFRONT ranges. For CDN-fronted names, the CNAME target or the
distribution behind the alias is the thing to watch, and a change there should always alert.
The change you must never fold away
Even with the ranges right, one case looks like rotation and is not. If the addresses under a single record name move from one provider’s range into a different provider’s range, that is not a CDN rotating its own edge. That is your record being repointed from one service to another, which is either a migration your team did on purpose or somebody moving your traffic without asking. Either way a person needs to see it, so a cross provider move should never be quietly absorbed no matter how confidently both ends are recognised as edge.
The same goes for the CDN being switched off entirely, so that a record drops from edge back to a bare origin address. That is a change in who terminates your traffic.
So the split ends up looking like this.
| Always alert | Safe to fold away |
|---|---|
| Nameserver changes | A or AAAA moving within one provider’s edge range |
MX changes | |
SPF, DKIM and DMARC record changes | |
CNAME retargeting | |
| Edge addresses moving between two providers | |
| CDN switched on or off for a record |
One principle holds the whole thing together: suppression should only ever affect the email, never the record. Keep writing every observed change to history regardless. When something does go wrong six weeks later, the question you will need to answer is when the record changed, and an alerting rule that quietly discarded the evidence will not answer it.
What to do this week
You do not need a platform to act on this. You need an hour.
- Pull your last fortnight of DNS alerts and count how many produced an action. If the answer is zero, treat that as a finding about the monitoring rather than good news about the estate.
- For each recurring alert, check which record type changed. If it is only
AorAAAAand the domain is behind a CDN, you have found your noise. - Confirm that nameserver,
MXand the email authenticationTXTrecords are watched on their own terms and cannot be swept into the same quiet bucket as address rotation. - If you do suppress anything, check that the underlying history is still being recorded.
In practice, most small estates do not have someone with the time to do this triage every morning, which is why the alerts end up filtered instead. If that is you, it is worth having someone hold the pager who will actually read it.