Auto-Evaluation of Anthropic 100k Context Window

Retrieval architectures play a crucial role in LLM question answering (Q+A) by retrieving relevant documents before synthesizing them into an answer. The document retrieval step is necessary due to the limited context window size of most language models, but with larger context windows like Anthropic's 100k token model, it becomes reasonable to consider retriever-less options. A taxonomy of retriever architectures includes lexical/statistical methods (TF-IDF), semantic approaches (Pinecone), and retriever-less models like Anthropic's 100k context window. Evaluation strategies for these methods involve auto-evaluators, which can be used to compare performance on tasks such as Q+A over a specific paper or building codes. Results show that the retriever-less model performs well in some cases but may fall short in others due to latency and limited context window size. Overall, retriever-less approaches have appeal for applications with small corpus sizes and non-critical latency requirements.