Avoid Rate Limiting with Query Batching
The text discusses an incident (INC-666) where Sentry's alert rule post-processing step was overwhelmed with too much load, causing alerts that should have fired not to do so. This issue arose due to the high number of Snuba queries made by the system, which led to rate limiting and failure to evaluate many rules. To address this problem, Sentry implemented query batching, reducing the total number of Snuba queries from an average of 15M to under 2M. Additionally, they adjusted the frequency of metric alert evaluations to balance user experience with resource savings. Overall, these changes led to a significant reduction in load on Snuba and improved system performance.
Company
Sentry
Date published
Nov. 13, 2024
Author(s)
Colleen O'Rourke
Word count
1516
Language
English
Hacker News points
None found.