Decoding Strategies: How LLMs Choose The Next Word

The text discusses various decoding strategies used in Language Models (LLMs) to generate coherent and contextually appropriate text. It highlights the distinction between next-word predictors and text generators, emphasizing that LLMs don't always output the most probable next word iteratively but employ different decoding strategies for text generation. The article delves into deterministic methods like Greedy Search and Beam Search, stochastic methods such as Top-k, Top-p (Nucleus Sampling), and Temperature Sampling, and novel methods based on information theory like Typical Sampling. It also discusses Speculative Sampling, a technique to enhance LLM inference speed by generating multiple tokens per model pass without changes to the final output. The text concludes with an outlook for future research in this area.