AI models on CPUs: accurate audio transcriptions without breaking the bank

The text discusses how Mux was able to offer auto-generated captions on all videos uploaded for free by leveraging AI models like Whisper and OpenCLIP, which were previously considered expensive. They discovered GGML, an open source tensor library that optimizes CPU utilization, making it cost-effective for deployment. By using smaller model sizes, they achieved acceptable results while keeping costs low. Running inference on CPUs was found to be cheaper than using GPUs or third-party APIs. Mux's existing infrastructure for transcode jobs allowed them to efficiently schedule and run Whisper jobs alongside video transcodes, resulting in free auto captions for all their customers.