In this case study, Gemma-2B LLM fine-tuned on MonsterAPI outperforms LLaMA 13B in mathematical reasoning tasks. The smaller, fine-tuned model achieved a 68% performance boost over the base model after undergoing optimization for mathematical problem-solving tasks using Microsoft/Orca-Math-Word-Problems-200K dataset. Gemma-2B demonstrated higher accuracy in various aspects of mathematical reasoning, such as numerical variation, arithmetic variation, problem understanding, distractor insertion, and critical thinking. This study highlights the importance of fine-tuning for enhancing model performance and proves that smaller models can outperform larger ones when optimized for specific tasks.