Analyzing User Interactions with LLMs to Improve our Documentation

The text discusses the use of AI-enabled chat application Mendable to collect and analyze user questions related to documentation improvement. It presents a proposal to utilize LLMs (Large Language Models) in summarizing and identifying gaps in the documentation based on these collected questions. Two methods are experimented with: clustering similar questions before summarization, and applying map-reduce approach that splits questions into small segments for summarization. The results show trade-offs between the two approaches, with map-reduce offering high customizability but higher cost, while clustering risks information loss but offers lower cost. The combination of these methods is suggested as a promising solution to address this challenge.