AI Crawler

The distinction that matters is training versus retrieval. Training crawlers ingest content for future model versions; retrieval agents fetch a page the moment an assistant needs it to answer someone. A business that blocks retrieval agents cannot be read as evidence when an assistant is deciding who to recommend — the assistant simply builds its answer from whoever it can read, often a directory profile or a competitor.

The named agents worth knowing are few. OpenAI runs GPTBot for training, ChatGPT-User for live fetches on a user's behalf, and OAI-SearchBot for its search index. Anthropic runs ClaudeBot and Claude-User. Perplexity runs PerplexityBot. Google's Google-Extended governs Gemini training but explicitly does not affect Google Search crawling or ranking — a separation many site owners get wrong in both directions.

That confusion is the usual cause of accidental invisibility. Blocking rules arrive by copy-paste from a blog post, by a security plugin's "block AI scrapers" toggle, or by a WAF rule that treats unfamiliar user agents as threats. None of these announce what they cost. A business can hold an excellent profile, real reviews, and a well-written services page, and still be structurally unquotable because a checkbox somewhere returns 403 to the agent that would have read it.

Deciding is a business judgment, not a technical default. Blocking training crawlers is a defensible position on how your words get used. Blocking retrieval agents is a decision to be absent from the channel — sometimes correct, rarely intentional. Check what your site currently allows before choosing: the free AI readability check shows which agents you permit today, alongside the other technical signals that decide whether machines can use your pages.