Unbounded memory usage by TCP for receive buffers, and how we fixed it

In this blog post, we discussed how TCP memory limits can be ignored in the Linux kernel and introduced a patch that fixes it. The issue arises when a sender's window is larger than what a receiver can handle due to limited memory resources. We found that this can result in excessive memory usage or even disconnections after reaching the memory limit set by autotuning. We explained how TCP works, including window scaling and its limitations. The Linux kernel patch we wrote enables TCP window shrinking when necessary to prevent memory overuse and ensures correct behavior during a "window full" situation. We also provided test results demonstrating the effectiveness of the patch in addressing these issues. Finally, we shared the results from deploying this patch on the Cloudflare production network, showing improvements in packet drop rates, memory usage, and other metrics. Overall, our findings highlight the importance of enforcing TCP memory limits to ensure optimal performance and reliability for users.