/plushcap/analysis/sentry/sentry-avoid-rate-limiting-with-query-batching

Avoid Rate Limiting with Query Batching

What's this blog post about?

The text discusses an incident (INC-666) where Sentry's alert rule post-processing step was overwhelmed with too much load, causing alerts that should have fired not to do so. This issue arose due to the high number of Snuba queries made by the system, which led to rate limiting and failure to evaluate many rules. To address this problem, Sentry implemented query batching, reducing the total number of Snuba queries from an average of 15M to under 2M. Additionally, they adjusted the frequency of metric alert evaluations to balance user experience with resource savings. Overall, these changes led to a significant reduction in load on Snuba and improved system performance.

Company
Sentry

Date published
Nov. 13, 2024

Author(s)
Colleen O'Rourke

Word count
1516

Hacker News points
None found.

Language
English


By Matt Makai. 2021-2024.