AI Crawler

Training crawlers collect content for future model versions. Retrieval agents fetch a page when an assistant needs it for a current answer. If a business blocks retrieval agents, the assistant may have to rely on directories, third-party profiles, or competitors instead of the business's own pages.

The named agents worth knowing are few. OpenAI runs GPTBot for training, ChatGPT-User for live fetches on a user's behalf, and OAI-SearchBot for its search index. Anthropic runs ClaudeBot and Claude-User. Perplexity runs PerplexityBot. Google's Google-Extended governs Gemini training but explicitly does not affect Google Search crawling or ranking — a separation many site owners get wrong in both directions.

Sites can block retrieval unintentionally through copied robots.txt rules, a security plugin's "block AI scrapers" setting, or a web application firewall that rejects unfamiliar user agents. A 403 response prevents the agent from reading the page even when the business has current profiles, reviews, and service content.

Audit both declared and enforced access. Review robots.txt for each named agent, then request representative pages with the relevant user agent and record the status code. A rule that says "allow" does not help if the server, CDN, or security layer still returns 403, 429, a challenge page, or an empty response.

Choose separate policies for training and retrieval crawlers. Blocking training controls one use of published content. Blocking retrieval can prevent the site's pages from supporting current answers. Check the existing configuration before changing it. The free AI readability check shows which agents the site currently permits and whether they can fetch its pages.