To use this, enable server-side Bot traffic tracking. It is included in your DataFast plan.
See bot traffic by page
Bot traffic now has a Pages view.
Instead of only grouping crawler activity by company, you can flip the card and see the URLs that AI assistants, search engines, and training crawlers request most. Click the top-right view button to switch between company view and page view.
This makes it easier to answer questions like:
- Which pages are ChatGPT, Claude, or Gemini opening to answer users?
- Are bots repeatedly requesting missing pages or old URLs?
- Which public pages are most interesting to search and training crawlers?
You can also search for a specific page, like /llms.txt, even if it is not one of your top crawled URLs.
Discovery files are tracked too
The latest server-side bot tracker also keeps crawler-facing files visible when known bots request them.
This includes:
/robots.txt/llms.txt/llms-full.txt- sitemap XML files
These files matter more now that AI agents, search bots, and model infrastructure use them to discover what exists on your website.
Update @datafast/ai-crawl to the latest version to track crawler-facing discovery files.
