BIG-Bench: The Behemoth Benchmark for LLMs, Explained

BIG-Bench is a comprehensive benchmark for large language models (LLMs) developed by over 400 researchers from various institutions. It consists of more than 200 language-related tasks, aiming to go beyond the imitation game and extract more information about model behavior. The benchmark's API supports JSON and programmatic tasks, facilitating easy few-shot evaluations. BIG-bench Lite is a lightweight alternative for addressing computational constraints, offering a diverse set of tasks that measure various cognitive capabilities and knowledge areas. Evaluation results show that the best LLMs can barely score 15 out of 100 on BigBench tasks, indicating room for improvement in model performance and calibration. The benchmark also measures social bias present in models and provides insights into their behavior and approximation to human responses.