Migrating billions of records: moving our active DNS database while it’s in use

Cloudflare, an authoritative DNS provider used by 14.5% of all websites as of October 2024, has successfully migrated its DNS records data to a new database cluster. The migration was driven by the need to optimize the company's main Postgres database cluster (cfdb), which was becoming increasingly strained under the pressures of several services, many unrelated to DNS. The move to a new database allowed Cloudflare to create a more tightly controlled interface to the DNS data and improve the systems that interact with it. The migration process involved building a Change Data Capture and Transfer Service (CDCTS) to ensure no data loss and minimal downtime during the transfer of millions of records daily.