Crawler Hints: How Cloudflare Is Reducing The Environmental Impact Of Web Searches

Cloudflare has introduced Crawler Hints, a new feature designed to reduce energy consumption and carbon output on the internet by minimizing unnecessary web crawling. The company estimates that excessive bot traffic generates 26 million tonnes of carbon cost per year, which could be reduced by ensuring search engine crawlers only visit websites when content has changed or been added. Crawler Hints provides high-quality data to search engine crawlers on when content has been modified, allowing them to time their crawling more efficiently and reduce resource consumption for both customers' origins and Cloudflare infrastructure. The feature also aims to improve the quality of search results by providing search engines with up-to-date information about website changes.