A Guide to LLM Hyperparameters

Large language models (LLMs) are crucial for various applications, but selecting the best one requires considering several factors such as parameter count and performance on benchmark tests. Hyperparameters play a significant role in customizing LLMs to specific needs. They govern the training process of an LLM without becoming part of the resulting base model. Commonly used LLM hyperparameters include model size, number of epochs, learning rate, batch size, max output tokens, decoding type, top-k and top-p sampling values, temperature, stop sequences, frequency and presence penalties. Hyperparameter tuning is a process to find the optimal combination of these parameters for better LLM performance. Automated hyperparameter tuning methods like random search, grid search, and Bayesian Optimisation can streamline this process.