Indexing millions of HTTP requests using Durable Objects

Cloudflare has introduced a new feature that allows users to store and retrieve logs using R2, which is more cost-effective than traditional methods. The company built an API for log retrieval using Cloudflare Workers with Durable Objects and the Streams API. This approach enables customers to index and query millions of their Cloudflare logs stored in batches on R2 without needing to manage their own database or logging infrastructure. The system uses Durable Objects to maintain indexes of data stored in R2, simplifying the process for developers. Currently, RayID lookups are supported for HTTP Requests, Firewall Events, and soon Workers Trace Events. Cloudflare plans to add more complex types of filters and queries that users can run against their logs in the future.