Creative foot-shooting with Go RWMutex

Filippo Valsorda, a developer at CloudFlare, recently experienced an issue with their Go DNS server, RRDNS. The problem occurred when deploying a new module for the authoritative server, which adds IP addresses to DNS answers. After deployment, RRDNS stopped answering queries for beta domains served by the new code. The issue was caused by forgetting to call f.IPTable.RUnlock() in the getCFAddr function, leading to a deferred deadlock when the table reload function ran. Although no customers were affected due to testing on off-production machines and limited use of the new code, this incident highlights the need for better tooling to detect such issues. CloudFlare is currently hiring developers for their DNS server team in London, San Francisco, and Singapore.