Your cart is currently empty!
How to write robots.txt to avoid AI bots
I’ll leave this as a memorandum.
It seems that you can avoid crawling by AI bots by putting the following in robots.txt. That said, there are apparently some AI bots that ignore robots.txt, so I can’t say for sure. If you really don’t want the AI to patrol your site, the only option is to use the Simple Membership plugin or something to put your content behind a login wall.
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: AdsBot-Google
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ICC-Crawler
Disallow: /
User-agent: bingbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: omgili
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-ai
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: YouBot
Disallow: /
Comments
Featured products
-
[WAV] Hiphop and Rap Music – Genx Beats
$7.00 or more -
[WAV] Hiphop and Rap Music Vol. 2 – Genx Beats
$7.00 or more -
[WAV] Hiphop and Rap Music Vol. 3 – Genx Beats
$7.00 or more
Leave a Reply