A customer support agent was designed to match the understanding of a human customer, with a focus on speed and iteration. The system used explicit intents for well-defined flows and a RAG architecture for product questions that change often. However, performance diverged from initial results after deployment, with out-of-domain questions performing poorly. To address this, techniques such as augmenting training data, using an LLM hybrid system, or creating more specific intents were explored. After iterating on the training dataset, validation accuracy improved by 23%, and further optimizations on the evaluation set resulted in a 14% accuracy gain. Adding more specific intents increased performance by capturing key concepts from the datasets and reducing out-of-domain classification errors.