Information on the DNS outage that occurred on Monday, June 1st can be found on the Comcast Voices blog.
Due to an issue with Akamai not downloading a current copy of the xfinity.com zone, they were serving a zone file with expired signatures, causing DNSSEC validation failure. Comcast’s technicians worked with the appropriate teams to resolve the issue. In the meantime, Comcast temporarily deployed a negative trust anchor to make the site accessible. The DNSViz report of this failure can be found at http://dnsviz.net/d/xfinity.com/VWYAmw/dnssec/.
No doubt many of you heard HBO’s recent announcement that they will be launching a streaming service in the near future. And as Comcast customers tried to visit HBO’s site at order.hbonow.com, many were understandably confused and frustrated when the site wouldn’t load for them properly.
Rest assured that this was not a conspiracy by Comcast to block access to the site. In fact, the HBO team had simply misconfigured DNSSEC on the order.hbonow.com site, making it appear that the site was invalid.
Comcast, along with many other DNS providers including Google’s Public DNS, use DNSSEC to ensure that the site you are being directed to is actually the site where you intended to go. In a nutshell, DNSSEC is a set of standards that allow site owners to sign their domains, which enables DNS servers to confirm that the sites are valid. If a site fails DNSSEC validation, DNS won't resolve and the site simply won’t load. In a time when malware abounds and phishing sites pop up daily, DNSSEC is more important than ever.
Fortunately, HBO was able to fix the issue, and Comcast DNS team quickly cleared our cache servers to make the site once again available to everyone.
For a deeper technical explanation of the DNSSEC issues on order.hbonow.com, visit the Dan York’s excellent summary on the Internet Society blog. For more background on what DNSSEC is and how it works, visit DNSSEC.net.
Comcast saw reports on Slashdot of errors resolving the 021yy.org domain name. Upon investigation it appears the domain is improperly delegated to the authoritative servers for this domain (one of which is ns1.booen.com). Specifically if we query ns1.booen.com and ask for the SOA record, the answer does not say “021yy.org” in the authority section, but rather booen.com. This is a non-authoritative answer, which is not how an authoritative server should work. Furthermore, if we query ns1.booen.com and ask for the NS records for 021yy.org, the server gives an NXDOMAIN response, rather than the authoritative nameservers for the domain, which results in an incorrectly delegated and incorrectly configured zone.
So, in short, and contrary to claims made in Slashdot, Comcast is not blocking access to 021yy.org, and nothing is wrong with our DNS servers. Rather, we recommend that the authoritative DNS administrator work to fix their DNS records and work with their DNS server software vendor to make their server DNS protocol compliant. As a side note we recommend the domain owner increases the TTL on their A and AAAA RRs from just 60 seconds to enhance cache-ability. We’re happy to assist in that process if needed; just use our contact form on this website.
The domain .gov is currently failing DNSSEC validation This is because the chain of trust within the gov domain is broken. The domain owners have been contacted and made aware of the issue. The DNSViz report of this failure can be found at http://dnsviz.net/d/gsa.gov/UguNUw/dnssec/
Customers in our northern California and Utah markets reported sporadic DNS failures when looking up domain names hosted by GoDaddy. Working with GoDaddy we learned that queries from these parts of our network routed to a GoDaddy Anycast node that was experiencing technical issues that caused our queries to timeout, while queries from other parts of our network were answered normally. GoDaddy made some Anycast changes and DNS resolution for our customers returned to normal.
The domain flyinggiants.com is currently failing DNSSEC validation. This is because RRSIG records in the domain are expired. The domain owners have been contacted and made aware of the issue. The DNSViz report of this failure can be found at http://dnsviz.net/d/flyinggiants.com/UaYVFQ/dnssec/.
We have noticed that certain Netgear routers with older firmware are performing DNS queries for the names of Netgear NTP servers at a rapid rate, which we initially thought was a DDOS attack on our DNS recursive resolvers. This seems to occur when a cable modem is reset. The router firmware bug can cause a single router to query at rates of thousands per second (millions per day) and impact that customer’s experience by flooding their connection until the router is reset. The DNS records being queried are: time-a.netgear.com, time-b.netgear.com, time-c.netgear.com
Netgear has confirmed the firmware bug and recommends the end users update their device’s firmware to the latest build for the impacted devices. The following links provide specific instructions on upgrading:
If anyone needs help upgrading or has additional questions, they can contact Netgear directly at (888)NETGEAR. They are aware of the issue and ready to assist customers.
The domain bncr.fi.cr is currently failing DNSSEC validation. This is because several RRSIG records in the domain are invalid, including www.bncr.fi.cr. The domain owners have been contacted and made aware of the issue. The DNSViz report of this failure can be found at http://dnsviz.net/d/www.bncr.fi.cr/UX_OqQ/dnssec/.