The rise of the AI crawler

The study analyzed data from MERJ and Vercel on the behavior of top AI crawlers such as OpenAI's GPTBot, Anthropic's Claude, and others in terms of their handling of JavaScript rendering with MERJ. The results show that none of the major AI crawlers currently render JavaScript, but they can fetch JavaScript files. Googlebot leverages its infrastructure to execute full-page rendering, while AppleBot renders JavaScript through a browser-based crawler. AI crawlers prioritize content types differently than traditional search engines, focusing on HTML and images. However, their crawling behavior is marked by inefficiencies such as high 404 rates and redirects. The study also highlights the need for site owners to optimize their sites for server-side rendering, efficient URL management, and content accessibility in order to be crawled effectively or not at all. Additionally, users should consider verifying sources directly when relying on AI-provided links and expect inconsistent freshness from AI models.