Bot traffic API

Use the Bot traffic API to answer questions such as:

How many AI answer requests reached my site last week?
Which pages did ChatGPT, Claude, or Google request?
Which companies fetched my /llms.txt file?
Which requests were IP verified?
Which companies or individual user agents are currently excluded from ingestion?
When does my account allowance reset?

The REST API and MCP tools call the same route handlers as each other, so authentication, permissions, validation, and results stay identical.

Access

Analytics endpoints accept an owner-created df_ website key, or an owner dft_ account token with analytics:read. With dft_, add websiteId to the query.
Allowance, settings, and token endpoints require an owner dft_ account token. Settings reads need settings:read; changes need settings:write.
The same account-access check as the rest of the DataFast API applies. Accounts without API access receive a 403. Team members and viewers cannot use these owner-only Bot traffic routes.

Endpoint reference

Open an endpoint below for its parameters, response fields, errors, and a ready-to-run example.

Purpose	Method	Path
Request totals, exact crawlers, and timeseries	GET	`/api/v1/analytics/bot-traffic`
Requested pages and HTTP statuses	GET	`/api/v1/analytics/bot-traffic/pages`
Account allowance and billing-cycle usage	GET	`/api/v1/admin/bot-traffic/usage`
Read settings and crawler catalog	GET	`/api/v1/admin/websites/{websiteId}/bot-traffic/settings`
Update display, verification, auth, or ingestion settings	PATCH	`/api/v1/admin/websites/{websiteId}/bot-traffic/settings`
Read token status	GET	`/api/v1/admin/websites/{websiteId}/bot-traffic/token`
Create a request-auth token	POST	`/api/v1/admin/websites/{websiteId}/bot-traffic/token`
Rotate a request-auth token	PUT	`/api/v1/admin/websites/{websiteId}/bot-traffic/token`
Delete a request-auth token	DELETE	`/api/v1/admin/websites/{websiteId}/bot-traffic/token`

Query Bot traffic

curl "https://datafa.st/api/v1/analytics/bot-traffic?startAt=2026-08-01&endAt=2026-08-31&category=answer_fetch&verification=ip_verified" \
  -H "Authorization: Bearer df_xxx"

Omit the dates for the last 30 calendar days. Supported query parameters:

Parameter	Values
`startAt`, `endAt`	Both together: `YYYY-MM-DD` or ISO 8601 with `Z`/UTC offset
`timezone`	IANA timezone; defaults to the website timezone
`interval`	`hour`, `day`, `week`, or `month`
`category`	`all`, `answer_fetch`, `search_index`, `training`, or `ai_crawler`
`verification`	`all`, `ip_verified`, or `user_agent_only`
`company`	Exact company returned by the API, such as `OpenAI`
`crawlerId`	Exact ID returned in `crawlers[].id`, such as `OpenAI::ChatGPT-User::answer_fetch`

The summary response includes direct fields such as totalRequests, aiAnswerRequests, indexingRequests, trainingRequests, ipVerifiedRequests, crawlers, and timeseries. An agent does not need to reinterpret chart labels to answer a normal question.

Query requested pages

curl "https://datafa.st/api/v1/analytics/bot-traffic/pages?startAt=2026-08-01&endAt=2026-08-31&crawlerId=OpenAI%3A%3AChatGPT-User%3A%3Aanswer_fetch&limit=50" \
  -H "Authorization: Bearer df_xxx"

This endpoint accepts the same filters plus pageSearch, limit (1–100), and offset. Each page includes total requests, status-code counts, and exact crawler counts.

Read allowance and reset date

curl "https://datafa.st/api/v1/admin/bot-traffic/usage" \
  -H "Authorization: Bearer dft_xxx"

This is account-wide, just like the billing card, so the token must have access to all websites. It returns accepted usage, dropped requests after the limit, remaining allowance, percentage used, the current period and reset date, paid packs, and only websites that consumed requests.

Buying or changing paid capacity remains a human-confirmed action in Billing. The API and MCP report the full state but do not charge a saved card.

Read and update settings

First read settings. The response includes crawlerCatalog, with every company, exact agent ID, category, short description, and enabled state.

curl "https://datafa.st/api/v1/admin/websites/WEBSITE_ID/bot-traffic/settings" \
  -H "Authorization: Bearer dft_xxx"

Prefer incremental ingestion changes so an agent does not overwrite an exclusion it did not intend to change:

curl -X PATCH "https://datafa.st/api/v1/admin/websites/WEBSITE_ID/bot-traffic/settings" \
  -H "Authorization: Bearer dft_xxx" \
  -H "Content-Type: application/json" \
  -d '{
    "disableCompanies": ["Meta"],
    "enableCrawlerIds": ["OpenAI::ChatGPT-User::answer_fetch"],
    "verifiedOnly": true
  }'

Supported fields:

Ingestion: disableCompanies, enableCompanies, disableCrawlerIds, enableCrawlerIds, or a full excludedCrawlerIds replacement.
Verification and authentication: verifiedOnly, requireAuthToken.
Display only: hiddenCrawlers, hideDashboardCard, groupBy, pageFilterIds.

hiddenCrawlers only changes what the chart shows. It does not stop ingestion or reduce usage. Company/agent exclusions stop future matching requests before storage, so skipped data cannot be recovered.

Request-authentication token

# Create (POST) or rotate (PUT)
curl -X POST "https://datafa.st/api/v1/admin/websites/WEBSITE_ID/bot-traffic/token" \
  -H "Authorization: Bearer dft_xxx"

The full requestAuthToken is returned only once on create or rotate. Save it in a server-side secret, deploy it in your tracker, then set requireAuthToken: true. Rotating invalidates the previous token immediately. Deleting the token also turns authentication enforcement off.

MCP tools

The same workflows are available as:

datafast_bot_traffic_summary, datafast_bot_traffic_pages, datafast_bot_traffic_usage
datafast_bot_traffic_settings_get, datafast_bot_traffic_settings_update
datafast_bot_traffic_token_get, _create, _rotate, and _delete

MCP asks for explicit confirmation before any ingestion/auth change or credential action. See the complete MCP tool reference and try the REST endpoints in the API playground.

✍️ Something missing? Suggest features.

🤖 AI agent or LLM? Read this page as markdown