How and why the leap second affected Cloudflare DNS

On January 1, 2017, a negative value in Cloudflare's custom RRDNS software caused some DNS resolutions to fail for customers using CNAME records. The issue affected approximately 0.2% of DNS queries and less than 1% of HTTP requests during the outage. The problem was quickly identified and fixed within 90 minutes, with a global rollout by 6:45 UTC. The root cause of the bug was the belief that time cannot go backwards, which led to negative values being recorded in server selection code. Cloudflare has since inspected all its code for other leap second-sensitive uses of time intervals and apologized for any inconvenience caused to customers.