How we built Pingora, the proxy that connects Cloudflare to the Internet

Cloudflare has developed Pingora, an HTTP proxy built using Rust that serves over 1 trillion requests per day. The new system replaces NGINX and offers improved performance, efficiency, and the ability to support more features for Cloudflare customers. Pingora's design decisions include choosing Rust as the programming language, building a customizable HTTP library, and implementing a multithreaded workload scheduling system with work stealing capabilities. The new architecture allows for better connection reuse ratios, resulting in faster TTFB (time-to-first-byte) and reduced handshake time. Pingora also consumes less CPU and memory compared to the previous service, making it more efficient. Additionally, Rust's memory-safe semantics provide safer feature shipping and help prevent crashes due to undefined behavior.