How we built rate limiting capable of scaling to millions of domains

Cloudflare has implemented rate limiting at the edge of its network, which provides several advantages such as easier setup and operation, protection against excessive traffic or layer 7 attacks, offloading performance and memory cost to the edge, among others. The rate limiter works by counting requests per client IP address that match a defined rule, and once the counter exceeds a threshold, further requests are not allowed to reach the origin server. This is an effective protection against brute force attacks on login pages and other abusive traffic like L7 DoS attacks. To implement this rate limiter at the edge of its network, Cloudflare leveraged its existing infrastructure and utilized NGINX's Lua scripting module to create a Twemproxy cluster inside each PoP (Point of Presence). The leaky bucket algorithm was initially considered but later replaced with a sliding windows approach due to its simplicity, accuracy, and efficiency. The rate limiter is already being used by many customers to control the rate of requests that their origin servers receive. It has been handling several billion requests per day and recently mitigated attacks with as many as 400,000 requests per second to a single domain without degrading service for legitimate users.