AI crawlers, search bots, and training agents.

User-triggered fetches when ChatGPT opens a page to answer a person.

User-triggered fetches when Claude opens a page to answer a person.

User-triggered fetches for Perplexity answers and source citations.

Google user-triggered fetcher used by AI or product experiences.

Google fetcher associated with NotebookLM-style answer workflows.

Google fetcher associated with read-aloud and assistant experiences.

Google user-triggered fetcher used by AI or product experiences.

MistralAI-User is tracked as a user-triggered fetcher from Mistral.

User-triggered Microsoft/Copilot fetches for AI answers.

User-triggered Amazon fetches for fresh answers in products such as Alexa.

DuckDuckGo real-time crawler for AI-assisted answers with citations.

xAI-SearchBot is tracked as a user-triggered fetcher from xAI.

Grok-DeepSearch is tracked as a user-triggered fetcher from xAI.

Meta fetcher used when a person requests or shares a specific URL.

Kimi-User is tracked as a user-triggered fetcher from Moonshot AI.

Qwen-User is tracked as a user-triggered fetcher from Alibaba.

OpenAI crawler used to surface sites in ChatGPT search results.

Anthropic crawler used to discover and refresh content for Claude search.

Perplexity crawler used to keep its answer and search index fresh.

Google-InspectionTool is tracked as a search or answer-index crawler from Google.

GoogleOther is tracked as a search or answer-index crawler from Google.

Classic Google crawler used for Search indexing and discovery.

MistralAI-Index is tracked as a search or answer-index crawler from Mistral.

Microsoft crawler used for Bing Search indexing.

Legacy Microsoft crawler used for search indexing.

Apple crawler used for Apple search and assistant surfaces.

Amazon crawler used to make content eligible for Amazon search experiences.

Kimi-SearchBot is tracked as a search or answer-index crawler from Moonshot AI.

TikTokSpider is tracked as a search or answer-index crawler from ByteDance.

Baiduspider is tracked as a search or answer-index crawler from Baidu.

YouBot is tracked as a search or answer-index crawler from You.com.

OpenAI crawler for collecting public content that may improve future models.

Anthropic crawler for public content that may be used to improve Claude.

Robots token controlling whether Apple may use crawled content for AI training.

Amazon crawler used to improve Amazon products and services.

Meta crawler for indexing or improving Meta products and AI systems.

KimiBot is tracked as a public-content crawler from Moonshot AI.

Bytespider is tracked as a public-content crawler from ByteDance.

ERNIEBot is tracked as a public-content crawler from Baidu.

QwenBot is tracked as a public-content crawler from Alibaba.

ChatGLM-Spider is tracked as a public-content crawler from Zhipu AI.

DeepSeekBot is tracked as a public-content crawler from DeepSeek.

cohere-ai is tracked as a public-content crawler from Cohere.

cohere-training-data-crawler is tracked as a public-content crawler from Cohere.

Allen Institute crawler used to find documents for AI research systems.

Common Crawl crawler that builds open web crawl datasets.

OpenAI crawler associated with ad or landing-page fetch workflows.

Google-CloudVertexBot is tracked as other ai bots from Google.

GrokBot is tracked as other ai bots from xAI.

xAI-Bot is tracked as other ai bots from xAI.

xAI-Grok is tracked as other ai bots from xAI.

xAI-Web-Crawler is tracked as other ai bots from xAI.

Grok is tracked as other ai bots from xAI.

FacebookBot is tracked as other ai bots from Meta.

Doubaobot is tracked as other ai bots from ByteDance.

YiyanBot is tracked as other ai bots from Baidu.

TongyiBot is tracked as other ai bots from Alibaba.

AliyunBot is tracked as other ai bots from Alibaba.