Wordsmith is an AI assistant that uses large language models (LLMs) to review legal documents, draft emails, and generate contracts, providing in-house legal teams with a tool that integrates seamlessly into email and messaging systems. The Wordsmith team used LangSmith, a tracing service, to better understand what's happening in production and measure the impact of experiments on key parameters during product development. LangSmith has provided value at each stage of the product development life cycle by enabling quick iteration, establishing baselines for performance measurement, facilitating rapid debugging, and enabling online experimentation. The team plans to integrate LangSmith even more deeply into the product life cycle and tackle complex optimization challenges in the future. With LangSmith, Wordsmith aims to optimize hyperparameters for each customer and use case, creating a personalized experience for their users.