How Podium optimized agent behavior and reduced engineering intervention by 90% with LangSmith

Podium is a communication platform that helps small businesses connect with customers via various channels such as phone, text, email, and social media. The company's mission is to improve customer interactions by responding promptly to inquiries, which can lead to higher lead conversion rates. To achieve this, Podium launched AI Employee, an agentic application that engages local business customers, schedules appointments, and closes sales. The use of LangSmith for LLM testing and observability has been crucial for Podium's development lifecycle. The platform allows the company to test and monitor its AI employee's performance continuously, refining the model with new edge cases over time. By using LangSmith, Podium was able to improve their agent's ability to detect when a natural conversation should end, resulting in a 7.5% improvement in F1 scores. LangSmith has also enabled Podium's Technical Product Specialists (TPS) team to troubleshoot issues more efficiently and independently, reducing the need for engineering intervention by 90%. This has led to increased efficiency of the support team and improved customer satisfaction scores. By integrating LangSmith and LangChain, Podium has gained a competitive edge in the space of customer experience tools, allowing them to focus on their core value proposition - helping small businesses capture leads more effectively. The company is currently hiring across roles to help local businesses win and encourages others to try out LangSmith for free or talk to a LangSmith expert to learn more.