AI Robots.txt Generator

Create a clear robots.txt file for AI training crawlers, AI-search crawlers, user-triggered AI agents and regular search bots.

Website URL

Sitemap URL optional

Suggested sitemap: Enter a website URL first

Policy mode

Blocks common AI training and dataset crawlers while keeping regular search engines and AI-search crawlers open.

Custom blocked paths

One path per line. These paths are added as User-agent: * disallow rules.

Include Content Signals Adds experimental preference signals. It does not replace User-agent rules.

Advanced custom bot rules

Allow explicitly allows this crawler. Block disallows it from the whole site. Inherit keeps the selected preset/default behavior.

AhrefsBot Ahrefs · seo crawler SEO crawler used by Ahrefs.

MJ12bot Majestic · seo crawler Crawler used by Majestic link intelligence.

SemrushBot Semrush · seo crawler SEO crawler used by Semrush.

Claude-SearchBot Anthropic · ai search crawler Anthropic crawler associated with search retrieval.

OAI-SearchBot OpenAI · ai search crawler OpenAI search crawler used for search result retrieval and linking.

PerplexityBot Perplexity · ai search crawler Crawler used by Perplexity for indexing/search retrieval.

Claude-User Anthropic · user triggered agent User-triggered agent used when Claude accesses pages for a user request.

ChatGPT-User OpenAI · user triggered agent User-triggered agent used when a user requests page access through ChatGPT.

Perplexity-User Perplexity · user triggered agent User-triggered agent used when a user requests a page through Perplexity.

ClaudeBot Anthropic · ai training crawler Anthropic crawler associated with model training and data collection.

GPTBot OpenAI · ai training crawler OpenAI web crawler used for model improvement and training-related crawling.

Applebot Apple · search crawler Apple crawler for search and related experiences.

Googlebot Google · search crawler Main Google Search crawler.

Bingbot Microsoft · search crawler Main Bing Search crawler.

Applebot-Extended Apple · product token Apple robots.txt token for extended AI-related use preferences.

Google-Extended Google · product token Product token used in robots.txt, not a normal standalone HTTP user-agent.

CCBot Common Crawl · archive crawler Common Crawl crawler used to build public web datasets.

Your robots.txt result

Fill the form and generate a file. The result will appear here without reloading the page.

# robots.txt will appear here

After generation, upload the file to the root of your website and test it with the checker.

AdSense placeholder