It fetches a page's raw HTML and robots.txt, then checks whether known AI crawlers - such as GPTBot, ClaudeBot, Google-Extended, and PerplexityBot - are allowed to access it and whether the page's content is present without JavaScript.
AI Training Scrapers
Bots like GPTBot, ClaudeBot, and Google-Extended crawl site content specifically to train foundation LLM models. Blocking them prevents model training without directly harming your Search Engine visibility.