Crawler Hints supports Microsoft’s IndexNow in helping users find new content

The article discusses the inefficiencies of the current model for indexing websites, which is based on a "pull" model where search engine bots crawl webpages at regular intervals to look for changes. It mentions that over 50% of this crawler traffic is wasted effort and highlights the environmental impact of such practices. The article proposes an alternative approach: a "push" model, where websites can ping search engines when updates are made. This would benefit all parties involved - website owners, search engines, and users. To implement this new model, Cloudflare launched Crawler Hints, which sends signals to web indexers based on cache data and origin status codes to help them understand when content has likely changed or been added to a site. On the other hand, IndexNow Protocol is a standard developed by Microsoft, Seznam.cz, and Yandex that represents a major shift in the way search engines operate. It provides a mechanism for search engines to receive signals from Crawler Hints. Microsoft has reported that millions of websites are now using IndexNow to signal to search engine crawlers when their content needs to be crawled and indexed, with about 7% of all new URLs clicked being processed through this protocol. Since its launch in October 2021, Crawler Hints has processed around six-hundred billion signals to IndexNow. The article concludes by encouraging users to enable Crawler Hints on their websites for free and mentions plans to default it on for all Cloudflare customers. It also expresses hope that Google will adopt IndexNow or a similar standard to reduce the burden of search crawling on the environment. The author emphasizes the importance of creating and supporting standards that make the internet operate better, greener, and faster.