/plushcap/analysis/cloudflare/how-to-receive-a-million-packets

How to receive a million packets per second

What's this blog post about?

The text discusses the process of achieving 1 million UDP packets per second on a Linux network stack. It begins by explaining the importance of measuring packets per second (pps) rather than bytes per second (Bps), as improving pps is more challenging. The author then describes the experimental setup, including two physical servers with multiple cores and a multi-queue 10G network card. The text goes on to detail various approaches for increasing the number of packets received per second, such as using sendmmsg and recvmmsg syscalls, pinning processes to specific CPUs, and configuring NICs to distribute packets across multiple RX queues. It also discusses the impact of NUMA performance and the use of SO_REUSEPORT flag for load distribution among receiver threads. The author concludes that achieving 1 million UDP packets per second is technically possible on a Linux machine, but it requires careful configuration and optimization to ensure even distribution of traffic across RX queues and receiver processes, as well as sufficient spare CPU capacity. The text also notes that the performance may vary depending on whether the application is processing received packets or not.

Company
Cloudflare

Date published
June 16, 2015

Author(s)
Marek Majkowski

Word count
2129

Language
English

Hacker News points
542


By Matt Makai. 2021-2024.