AI Crawler List
Understand crawler purpose, category, robots.txt token and recommended policy.
| Bot | Company | Category | Token | Recommended action |
|---|---|---|---|---|
| AhrefsBot | Ahrefs | seo crawler | AhrefsBot | Optional: allow for SEO tools, block if server load or data access is a concern. |
| Claude-SearchBot | Anthropic | ai search crawler | Claude-SearchBot | Allow if you want Claude search features to discover your pages. |
| Claude-User | Anthropic | user triggered agent | Claude-User | Allow if user-triggered Claude access should work. |
| ClaudeBot | Anthropic | ai training crawler | ClaudeBot | Block if you want to restrict Anthropic training crawler access. |
| Applebot | Apple | search crawler | Applebot | Usually allow if you want Apple search and assistant integrations to discover your pages. |
| Applebot-Extended | Apple | product token | Applebot-Extended | Block if you want to signal restrictions for Apple AI-related use while treating Applebot separately. |
| CCBot | Common Crawl | archive crawler | CCBot | Block if you do not want your pages included in Common Crawl datasets. |
| Google-Extended | product token | Google-Extended | Block if you want to opt out of certain Google AI product training/use signals while keeping Google Search crawling separate. | |
| Googlebot | search crawler | Googlebot | Do not block unless you intentionally want to remove regular Google crawling. | |
| MJ12bot | Majestic | seo crawler | MJ12bot | Optional: allow for link data, block if crawler load is not wanted. |
| Bingbot | Microsoft | search crawler | Bingbot | Do not block unless you intentionally want to remove Bing crawling. |
| ChatGPT-User | OpenAI | user triggered agent | ChatGPT-User | Allow if users should be able to ask ChatGPT to fetch your pages. |
| GPTBot | OpenAI | ai training crawler | GPTBot | Block if you want to signal that your public content should not be used for AI training. |
| OAI-SearchBot | OpenAI | ai search crawler | OAI-SearchBot | Allow if you want visibility in OpenAI search products. |
| Perplexity-User | Perplexity | user triggered agent | Perplexity-User | Allow if user-triggered Perplexity access should work. |
| PerplexityBot | Perplexity | ai search crawler | PerplexityBot | Allow if you want discovery in Perplexity search; block if you do not want this crawler. |
| SemrushBot | Semrush | seo crawler | SemrushBot | Optional: allow for SEO tools, block if server load or data access is a concern. |