How to rank in Claude: getting cited when Claude searches the web

An afternoon's Anthropic checklist

1. Audit your Anthropic rules. Search your live robots.txt for Anthropic agents. To stay citable while opting out of training, the groups look like this:

# Keep Claude able to find and read pages for answers
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

# Opt out of training data collection
User-agent: ClaudeBot
Disallow: /

Older tokens such as anthropic-ai and Claude-Web are deprecated; leaving them does no harm, but they no longer control anything. Remember that each subdomain needs its own file, so docs. or shop. subdomains need checking too.

2. Remove bot challenges from content pages. Anthropic says its bots respect anti-circumvention measures and do not try to get past CAPTCHAs. A page behind a challenge is invisible to Claude even when robots.txt allows it. Check your CDN or security plugin logs for challenged Anthropic requests.
3. Search your core questions on Brave. Run ten of your target queries at search.brave.com and note where your pages appear. If they are missing there, they are less likely to reach Claude.
4. Run the same questions in Claude with web search on, and log the cited domains.
5. Add claude.ai to your AI referral channel in analytics.

Content Claude can use well

People often give Claude in-depth jobs: research a market, compare vendors, explain a standard. On those jobs it may search repeatedly and finish with a lengthy source list. That favors pages with substance over pages built around a single keyword:

• Technical and reference material in text. Code samples, configuration values and error messages as real text, not screenshots. Version numbers and dates on anything that changes.
• Documentation split by task, one task per page with a descriptive title, so a fetch returns exactly the relevant answer.
• Explanations that show their reasoning. When a page says why, not only what, it gives Claude a passage to quote when a user asks "why".
• Primary sources. Changelogs, policies, pricing and specifications from the organization that owns them are the natural citation for factual questions about that organization.

Keep the main content in the server-rendered HTML. Claude-User reads what your server sends; a single-page app that renders content only in the browser may give it little to work with.

See what a Claude fetch receives

A quick way to catch problems is to request a page with the Claude-User token in the user agent and check that your answer text comes back:

curl -s -A "Mozilla/5.0 (compatible; Claude-User/1.0)" https://example.com/your-page/ \
  | grep -c "a phrase from your answer"

A count of zero, a challenge page or a 403 response means a rule keyed to the user agent is in the way. This test cannot reveal blocks based on IP address, because your request comes from your own machine, so a clean result here still needs a look at your firewall logs for real Anthropic traffic.

Measuring Claude citations

Sessions referred by claude.ai are people who clicked a cited link inside Claude's own apps. In server logs, Claude-User requests show which pages are being read for answers, and Claude-SearchBot requests show crawling for search. The RankWave entries for Claude-SearchBot and Claude-User describe each agent.

One complication: developers can give Claude a web search tool through Anthropic's API, and Claude models run inside other companies' platforms. A citation there sends visitors with the host product's referrer, or with none, so checking which agent fetched the page tells you more than the referral report does. See the measurement page for a full routine.

Things Claude decides for itself

Whether Claude searches at all, which results it opens and which it credits are choices Anthropic keeps unpublished. The providers behind its search may change over time, and being credited in today's conversation predicts little about tomorrow's. If you publish an llms.txt file, for example with Hydrogen SEO's llms.txt editor, treat it as a courtesy to any system that reads it, not a lever: Anthropic has not said Claude's search or fetch agents use it. What you can guarantee is that nothing on your side, whether robots.txt, a firewall rule or a CAPTCHA, stops Claude reading a page once it chooses to.