Proving ROI for LLM products is crucial to securing resources and identifying areas for improvement.
To measure success, product teams use both qualitative and quantitative feedback, with the former relying on user reviews and session replays, while the latter focuses on metrics such as hard resolution rates, soft resolution rates, customer satisfaction, average session length, time on site, daily active users, error rate, consistency between human and co/autopilot decisions, and cost savings.
These metrics vary greatly depending on the type of LLM product, with consumer entertainment products focusing on engagement and retention, B2B copilots or autopilots measuring quality of decisions and cost savings, and other categories requiring segmentation to account for a long tail of use cases.