Glossary
Short explanations of robots.txt and AI crawler terms.
What does Allow mean?
Allow can be controlled with robots.txt when the crawler respects the protocol. Use a specific rule, check the result, and avoid blocking Googlebot or Bingbot unless that is your real goal.
What does Disallow mean?
Disallow can be controlled with robots.txt when the crawler respects the protocol. Use a specific rule, check the result, and avoid blocking Googlebot or Bingbot unless that is your real goal.
What is a Content Signal?
Content Signal can be controlled with robots.txt when the crawler respects the protocol. Use a specific rule, check the result, and avoid blocking Googlebot or Bingbot unless that is your real goal.
What is a Sitemap Directive?
Sitemap directive can be controlled with robots.txt when the crawler respects the protocol. Use a specific rule, check the result, and avoid blocking Googlebot or Bingbot unless that is your real goal.
What is a User-Agent?
User-Agent can be controlled with robots.txt when the crawler respects the protocol. Use a specific rule, check the result, and avoid blocking Googlebot or Bingbot unless that is your real goal.
What is an AI Crawler?
AI crawler can be controlled with robots.txt when the crawler respects the protocol. Use a specific rule, check the result, and avoid blocking Googlebot or Bingbot unless that is your real goal.
What is an AI Search Crawler?
AI search crawler can be controlled with robots.txt when the crawler respects the protocol. Use a specific rule, check the result, and avoid blocking Googlebot or Bingbot unless that is your real goal.
What is an AI Training Crawler?
AI training crawler can be controlled with robots.txt when the crawler respects the protocol. Use a specific rule, check the result, and avoid blocking Googlebot or Bingbot unless that is your real goal.
What is Crawl-delay?
Crawl-delay can be controlled with robots.txt when the crawler respects the protocol. Use a specific rule, check the result, and avoid blocking Googlebot or Bingbot unless that is your real goal.
What is robots.txt?
robots.txt can be controlled with robots.txt when the crawler respects the protocol. Use a specific rule, check the result, and avoid blocking Googlebot or Bingbot unless that is your real goal.