Evaluating the quality of outputs produced by large language models (LLMs) is increasingly challenging due to the complexity of generative tasks and the length of responses. The "vibe check" approach, which involves subjective human judgments through A/B testing with crowd workers, has limitations such as being expensive and time-consuming. Recent research highlights the need for more nuanced and objective approaches to assess model performance. Factors affecting LLM performance include confounders like assertiveness and complexity, subjectivity and bias in preference scores, coverage of crucial error criteria, and biases like authority, beauty, verbosity, positional, attention, sycophancy, nepotism, fallacy oversight, and others. Researchers have tried various approaches to develop reliable methods for evaluating LLM performance, including using LLM-derived metrics, prompting LLMs with designed prompts, fine-tuning LLMs with labeled evaluation data, and developing guidelines and scaling annotation processes. Galileo's ChainPoll technique combines Chain-of-Thought prompting with polling to ensure robust and nuanced assessment, while the Luna suite provides a comprehensive framework for evaluating LLM outputs in terms of accuracy, cost, speed, scalability, and overcoming issues of human vibe checks and LLM-as-a-Judge biases.