for the longest time, I have always had the belief that google DNS was a great generic DNS server to use.
all of that changed after a recent encounter with presumably the most bizarre issue I have ever seen in the IT industry.
for those who don’t know, google offers a public DNS server that you can set on your internet-connected devices or home router. google DNS is a public favorite due to its speed, availability, and simplicity.
web pages seem to load faster than the DNS servers given by your ISP. as one of the largest entities on the internet, google DNS is hosted in datacenters across the world. as an added bonus, 8.8.8.8 is easy to type.
although there have been concerns about privacy using google DNS, the majority of the world considers google DNS the best.
I used to think the same until recently.
it all started with an issue reported by a customer that their emails stopped working.
to give a bit of backstory, this customer has their domain’s nameservers pointing to a different server than their registrar.
after investigating the issue, it turns out that their website developer canceled their DNS registrar by accident causing their entire domain to stop working. in response, my team moved their DNS back to the original domain registrar
on the same day, we got the website working, and for the most part, emails were delivering.
it was only a couple days later that we discovered emails being sent from gmail and icloud accounts were not delivering.
I began sending test emails using a handful of Gmail accounts I had access to and could not replicate the issue, but we had visual proof it was happening. some of our customer’s clients sent over the bounce backs.
it made NO sense.
when email isn’t delivering, one of the first things you look at are MX records from the problematic domain.
our customer’s MX records were showing correctly from a handful of DNS servers. Cloudflare, Google, Comcast, and our own.
all of them showed a response from Microsoft 365, as they should. since the issue was primarily with gmail senders, I vouched to keep looking at google DNS.
there’s a quote that comes to mind by Lawrence Douglas Wilder that says, “Anger doesn’t solve anything. It builds nothing, but it can destroy everything.”
ironically, anger solved this problem. out of sheer frustration, I spammed this customer’s domain over and over using nslookup.
at random, google DNS would reply with a completely different MX record than what it should be.
a theory passed around is that one of google’s DNS servers is caching the old record and on occasion, when their load balancers send you to one of their many servers, it responds with that erratic DNS entry.
both gmail and icloud emails use google DNS to look up mx records when sending emails. this means that at any point in time, if you have a gmail or icloud email and were to email this customer, you might get a bounce back, but not every time.
I have opened support cases with both google and the domain registrar. both of them are at a loss and say there is nothing that can be done.
to say this is disheartening would be an understatement. for this reason, I have made sure none of my home lab equipment uses google DNS servers until this can be fixed.
I hope you found this story interesting. if anyone reading this has ideas or suggestions, please feel free to share.
catch you on the flipside.
-izebra
Leave a Reply