Optimizing Magic Firewall’s IP lists

Cloudflare's Magic Firewall, a replacement for network-level firewall hardware, evaluates gigabits of traffic every second against user-defined rules. The system initially used more memory than desired due to the storage of millions of IP addresses in each namespace. To address this issue, Cloudflare leveraged eBPF maps, which exist regardless of a network namespace, allowing them to share data across all namespaces created for customers. This change significantly improved the efficiency of the product and better positioned it for future growth.