Tonight We Monitor, For Tomorrow, We Test in Production!

In a Test in Production Meetup, Steven Czerwinski from Scalyr discussed the importance of monitoring and observability while testing in production. He presented a case study on latency issues faced by one of their customers and shared key learnings realized after investigating the issue. Monitoring is crucial for testing in production as it helps shed light on hidden performance dark matter within complex distributed systems. Czerwinski emphasized following best practices in monitoring to avoid hindrances during investigations. He also highlighted the importance of consistency, analyzing based on what matters, and understanding averages versus distributions when designing monitoring systems.