Post-mortem: step processing backlogs on Aug 31, 2023

On August 31, 2023, a backlog of steps processing occurred due to event stream acknowledgement issues within Inngest's HTTP execution drivers. The issue was caused by invalid waitForEvent matching expressions, which led to an error and subsequent nack (not acknowledge) of certain messages. This caused extra load on several services, including the step processing handlers, pause/resume service, cancellation matching, and executor. Inngest introduced mitigations to fail functions when receiving invalid waitForEvent expressions, and over the next 36 hours, they deployed code to improve efficiency in various areas. They apologize for any delays or issues caused during this time period and have taken remedial actions to prevent such occurrences in the future.