Easily manage AI crawlers with our new bot categories

Cloudflare has introduced new bot categories and reporting features for its customers to provide better control over AI crawlers visiting their websites. The update includes Verified Bot subcategories such as Search Engine Crawler, Aggregator, AI Crawler, Page Preview, Advertising, Academic Research, Accessibility, Feed Fetcher, Security, and Webhooks. Customers can now create firewall rules based on these categories for more targeted responses to different types of bots. Additionally, Cloudflare is working on a new exclusion protocol specifically designed for AI crawlers to give website operators even more flexibility in managing them in the future. The company has also encouraged bot creators to respect robots.txt and adhere to other good practices to be considered as Verified Bots by Cloudflare. Cloudflare's new bot categories are available for all customers, including those on free plans, and can be accessed through the WAF tab in the dashboard. In the coming months, the company will also introduce more detailed reporting based on these bot categories to help customers visualize how often different types of bots visit their websites over time. The introduction of Verified Bot subcategories is part of Cloudflare's ongoing efforts to address concerns related to AI crawlers and provide better protection for its users against potential risks associated with them. The company plans to continue evolving its approach as the use of AI continues to grow on the Internet.