Improving Worker Tail scalability

The Workers Tail feature was developed to provide developers with real-time information on their applications in production. It enables users to view requests, exceptions, and other crucial data, allowing quick debugging of issues reported by users. Initially limited to Workers receiving 100 requests per second (RPS) or less, the scalability limitation has been addressed by enhancing Tail's performance at high loads. The first improvement involved moving filters out of the Durable Object and into the message producer. This prevented any filtered messages from reaching the Tail Durable Object, reducing its load. The second enhancement added a safety mechanism for the Durable Object during its operation. A simple controller was created to track RPS hitting the Durable Object, sampling messages until the desired volume of 100 RPS is reached. Finally, a failsafe mechanism was introduced in case the Durable Object gets overloaded due to an extremely high volume of traffic. These improvements have made it possible for Tail to reach new limits and scale according to users' needs. The enhanced feature is now available to all users, allowing them to monitor their applications more effectively. Users can tail Workers through the Cloudflare dashboard by logging in, navigating to their Worker, and clicking on the Logs tab. For those using Wrangler CLI, a new Tail can be started by running "wrangler tail". In addition to Live logs provided by Tail, users may also push log events to additional destinations for a historical view of their application's performance with Logpush or gain more insight and control over log messages and events themselves using Tail Workers.