DO it again: how we used Durable Objects to add WebSockets support and authentication to AI Gateway

In October 2024, the use of AI Gateway to store billions of logs from an AI application was discussed. With over 3 billion logs processed and increasing connections, a WebSockets API was built using Cloudflare's Developer Platform and Durable Objects to manage this scale effectively. The new API allows all AI providers supported by AI Gateway to be accessed via WebSocket, maintaining a single TCP connection between the client or server application and the AI Gateway. Authentication for AI Gateway is also introduced, requiring a Cloudflare API token with permission "AI Gateway: Run". This WebSockets API is now in beta and open to everyone.