A quick introduction to speculative decoding

Speculative decoding is an optimization technique designed to improve the latency of large language models (LLMs) by leveraging two models: a larger target model and a smaller draft model, both running on the same GPU. This approach reduces latency by generating potential output tokens with the smaller draft model, which can be accepted or rejected by the larger target model, thereby speeding up inference. The technique offers significant improvements in terms of time to first token (TTFT) and time per output token (TPOT), but comes with limitations, such as reduced throughput and quality when used with high batch sizes. To maximize benefits, it's essential to select an appropriate draft model, fine-tune it for specific use cases, and reduce orchestration overhead. Speculative decoding is particularly useful in scenarios where latency is a critical factor, such as code generation or meeting low-latency service level agreements (SLAs) for large models.