Billions and billions (of logs): scaling AI Gateway with the Cloudflare Developer Platform

Developers face significant challenges in managing multiple AI models and providers due to rapid advancements in the field. To address these issues, AI Gateway was launched as a centralized platform for efficient monitoring, usage control, and data optimization. However, initial architecture faced limitations such as retaining logs only for 30 minutes. The solution involved extending log storage capabilities from 30 minutes to being able to store billions of logs indefinitely. This was achieved by leveraging Cloudflare Workers, optimizing database schema, migrating request bodies to R2 storage, and integrating Durable Objects with SQLite for persistent logs. The new architecture also introduced an Account Manager for tracking user activities and managing storage capacity. Future enhancements include improving the Universal Endpoint with automatic retry capabilities and fallback logic, as well as expanding log storage capacities through sharded Durable Objects.