How to write robots.txt to avoid AI bots

avatar
Share This:

I’ll leave this as a memorandum.

It seems that you can avoid crawling by AI bots by putting the following in robots.txt. That said, there are apparently some AI bots that ignore robots.txt, so I can’t say for sure. If you really don’t want the AI ​​to patrol your site, the only option is to use the Simple Membership plugin or something to put your content behind a login wall.

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: AdsBot-Google
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ICC-Crawler
Disallow: /

User-agent: bingbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: omgili
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-ai
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: YouBot
Disallow: /

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

3 × 4 =