/plushcap/analysis/inngest/inngest-postmortem-2023-08-31

Post-mortem: step processing backlogs on Aug 31, 2023

What's this blog post about?

On August 31, 2023, a backlog of steps processing occurred due to event stream acknowledgement issues within Inngest's HTTP execution drivers. The issue was caused by invalid waitForEvent matching expressions, which led to an error and subsequent nack (not acknowledge) of certain messages. This caused extra load on several services, including the step processing handlers, pause/resume service, cancellation matching, and executor. Inngest introduced mitigations to fail functions when receiving invalid waitForEvent expressions, and over the next 36 hours, they deployed code to improve efficiency in various areas. They apologize for any delays or issues caused during this time period and have taken remedial actions to prevent such occurrences in the future.

Company
Inngest

Date published
Sept. 8, 2023

Author(s)
Tony Holdstock-Brown

Word count
1131

Hacker News points
None found.

Language
English


By Matt Makai. 2021-2024.