From 0 to 20 billion - How We Built Crawler Hints

In July 2021, Cloudflare announced its intention to launch Crawler Hints as a means to reduce the environmental impact of web searches by helping search indexers make more informed decisions on when content has changed. The first iteration of Crawler Hints was made available in October 2021. By leveraging data from Cloudflare's global network, Crawler Hints generates a "content freshness score" to inform crawlers of content changes and improve efficiency for both customers and search engines. Since its launch on October 18, 2021, Crawler Hints has processed over twenty-five billion crawl signals and been opted in by more than 81,000 customers.