Geo Key Manager: Setting up a service for scale

Cloudflare's Geo Key Manager service allows customers to choose where they store their TLS certificate private keys. The system started as a research project using Keyless SSL and identity-based encryption, but faced scalability issues due to increased demand for geographical control of information. A trans-pacific voyage incident in Melbourne revealed that the head of line blocking caused by sequential processing of requests led to TLS timeouts and degradation for unrelated zones. The solution involved changing keynotto's internal task handling model to process each request concurrently, using a multi-producer, single-consumer queue. Another incident in a Midwest data center highlighted the need for better visibility into system performance and load testing of less common use cases. Cloudflare is now working on overhauling Geo Key Manager to make it more flexible and scalable.