ClaudeBot

ClaudeBot is the user agent Anthropic uses to crawl the public web for training data. It honours robots.txt and identifies itself clearly, so it can be allowed or disallowed by name. Anthropic runs two further agents with separate purposes: Claude-SearchBot, which indexes pages for Claude's search features, and Claude-User, which fetches a page when a user asks Claude about it during a conversation.

The split matters when you decide what you want. A publisher that sells content may legitimately block ClaudeBot (training) while allowing Claude-SearchBot (retrieval and citation). A local service business usually wants all three allowed, because being in the training data is how the model learns that the business exists, and retrieval is how it finds current details.

As with other AI crawlers, a permissive robots.txt can be undone by a CDN or firewall rule, by rate limiting that returns errors, or by content that only renders in JavaScript, which most crawlers do not execute.

Seoptist tracks Claude as one of its engines through the official Anthropic API with web search enabled, and its audit checks that ClaudeBot and the related agents can actually reach your key pages.