Company
Date Published
Dec. 10, 2024
Author
Celso Martinho, Will Allen, Nelson Duarte
Word count
683
Language
English
Hacker News points
None

Summary

Cloudflare's AI Audit dashboard now allows users to quickly see which AI services are honoring their robots.txt policies, identify non-compliant services, and programmatically enforce these policies at the network level.` This new feature provides visibility into how AI companies and services access content, helps customers define and enforce robots.txt policies, and reduces automated traffic to servers. The feature is live today for all Cloudflare customers, who can log in to their dashboard and navigate to their domain to begin auditing bot traffic from AI services and enforcing robots.txt directives.