About RankProofBot
RankProofBot is the RankProof crawler. It audits websites on behalf of their owners for SEO and GEO weaknesses. If you see RankProofBot in your server logs, someone with access to your domain requested an audit.
We only crawl after verification
Before a website is audited, ownership must be proven — by DNS TXT record, meta tag, or a file under /.well-known/. Without that proof not a single page is fetched, so RankProof cannot be used to load someone else's website.
How we behave
- We respect your robots.txt, including crawl-delay.
- At most two concurrent requests per host, with pauses between them.
- We only download HTML. Images, video and fonts are skipped.
- The number of pages audited is capped per plan.
How to block us
Add this to your robots.txt:
User-agent: RankProofBot Disallow: /
We honour it even when the owner has requested an audit. The audit then fails with a message saying so.
AI crawlers we check for
Part of our audit checks whether your robots.txt blocks these crawlers. We are none of them, and we do not use your content to train models.
| Bot | Vendor | Purpose |
|---|---|---|
| GPTBot | OpenAI | model training |
| OAI-SearchBot | OpenAI | search |
| ChatGPT-User | OpenAI | on-demand retrieval |
| ClaudeBot | Anthropic | model training |
| Claude-User | Anthropic | on-demand retrieval |
| Claude-SearchBot | Anthropic | search |
| PerplexityBot | Perplexity | search |
| Perplexity-User | Perplexity | on-demand retrieval |
| Google-Extended | model training | |
| Applebot-Extended | Apple | model training |
| CCBot | Common Crawl | model training |
| meta-externalagent | Meta | model training |
| Bytespider | ByteDance | model training |
Contact
If RankProofBot is causing you trouble, get in touch. Our User-Agent points at this page: http://localhost:5180/bot