Understand what bots do on your website
DataFast now has a dedicated Bot traffic card for tracking crawler activity from AI assistants, search engines, and model training bots.
This helps you see which companies crawl your site, what kind of crawler requested the page, and which URLs they visited. For example, you can spot when ChatGPT fetches a page to answer a user, when Google indexes your content, or when an AI training crawler requests public pages.
What we analyze
- ✨ AI answers - user-triggered fetches from assistants like ChatGPT, Claude, Gemini, Perplexity, Copilot, and similar tools.
- 🔎 Indexing - crawlers that discover or refresh pages for search engines and AI search surfaces.
- 🧠 Training - crawlers that collect public web content that may be used for datasets, model training, or product improvement.
For each crawler, DataFast shows the provider, crawl type, verification confidence, request volume, and the pages crawled. This makes it easier to understand what AI tools and bots are trying to find on your product, including repeated requests to important pages or missing URLs.
You can see the full public list of crawlers we detect in the crawler directory.
Server-side setup required
Bot traffic is not part of the normal DataFast web analytics script. The browser script tracks human analytics like visitors, pageviews, goals, conversions, and attribution.
Crawler tracking needs a separate server-side setup because many AI and search crawlers do not reliably run frontend JavaScript. Install the server-side tracker to capture these requests from your backend, middleware, edge function, or worker.
Read the Bot traffic setup guide to enable it for your website.
Install @datafast/ai-crawl and add server-side Bot traffic tracking for my backend. Follow https://datafa.st/docs/bot-traffic-tracking and use my DataFast website ID.
For your AI coding agent
