Best Practices for ML Monitoring and Observability of Demand Forecasting Models

Demand forecasting is a crucial aspect for businesses across various industries. The advent of AI and machine learning has automated this process and made predictions more sophisticated and precise. However, recent events have raised questions about the reliability of these models' performance. Model monitoring and observability are essential to alert teams when these events happen, quantify their impact on models, and provide insights into root causes for quick remediation. Common challenges faced by demand forecasting models include regression model susceptibility to drift, limited feature diversity, and the impact of outlier events like COVID-19. To ensure satisfactory performance, it is crucial to monitor various metrics such as mean error, mean absolute error, mean absolute percentage error, and mean squared error. Observability platforms can help teams visualize and root cause issues quickly, especially during outlier events.