ORPO is an innovative algorithm that simplifies the LLM fine-tuning process by directly integrating preference alignment into a single-step supervised fine-tuning. ORPO incorporates an odds ratio-based penalty into the conventional negative log-likelihood (NLL) loss function during supervised fine-tuning, which helps distinguish between favored and disfavored responses. This approach is resource-efficient, eliminating the need for a separate reference model and additional training phases. ORPO has demonstrated superior performance in various benchmark tasks, outperforming state-of-the-art models that use traditional fine-tuning methods. Its integrated preference alignment ensures that the model not only learns the desired domain but also aligns with user preferences simultaneously, leading to more efficient training.