Large Language Models (LLMs) like GPT-4, Claude, LLama and Gemini have contributed significantly to the AI community by enabling organizations to build robust LLM-powered applications. However, even with these advancements, LLMs often hallucinate and generate false information that sounds true. Therefore, it is crucial for organizations to evaluate these models not only for their speed but also for their accuracy and performance. LLM evaluation helps developers understand the model's strengths and weaknesses, ensuring effective real-world application functionality while mitigating risks such as biased or misleading content. There are two main types of LLM evaluation: model evaluation, which assesses the core abilities of the model itself, and system evaluation, which examines how it performs within a specific program or with user input. Common metrics used to evaluate LLMs include response completeness and conciseness, text similarity metrics, question answering accuracy, relevance, hallucination index, toxicity, and task-specific metrics such as BLEU score for machine translation. Various LLM evaluation frameworks and tools are available, including DeepEval, promptfoo, EleutherAI LM Eval, MMLU, BLEU, SQuAD, OpenAI Evals, UpTrain, H2O LLM EvalGPT, which provide standardized benchmarks to measure and improve the performance, reliability, and fairness of language models. By using these tools and frameworks, developers can gain a deeper understanding of their model's strengths and weaknesses, ensuring responsible use of LLMs and mitigating potential risks associated with factual inaccuracies and biases.