Source: https://datafa.st/docs/api/website/bot-traffic
Markdown source: https://datafa.st/docs/api/website/bot-traffic.md
Description: Query AI assistant and crawler requests, inspect allowance, and manage every Bot traffic tracking control from REST or MCP.

# Bot traffic API

Use the Bot traffic API to answer questions such as:

- How many AI answer requests reached my site last week?
- Which pages did ChatGPT, Claude, or Google request?
- Which companies fetched my `/llms.txt` file?
- Which requests were IP verified?
- Which companies or individual user agents are currently excluded from ingestion?
- When does my account allowance reset?

The REST API and MCP tools call the same route handlers as each other, so authentication, permissions, validation, and results stay identical.

## Access

- Analytics endpoints accept an owner-created `df_` website key, or an owner `dft_` account token with `analytics:read`. With `dft_`, add `websiteId` to the query.
- Allowance, settings, and token endpoints require an owner `dft_` account token. Settings reads need `settings:read`; changes need `settings:write`.
- The same account-access check as the rest of the DataFast API applies. Accounts without API access receive a `403`. Team members and viewers cannot use these owner-only Bot traffic routes.

## Endpoint reference

Open an endpoint below for its parameters, response fields, errors, and a ready-to-run example.

| Purpose | Method | Path |
|---|---|---|
| [Request totals, exact crawlers, and timeseries](/docs/api/website/bot-traffic/summary) | GET | `/api/v1/analytics/bot-traffic` |
| [Requested pages and HTTP statuses](/docs/api/website/bot-traffic/pages) | GET | `/api/v1/analytics/bot-traffic/pages` |
| [Account allowance and billing-cycle usage](/docs/api/website/bot-traffic/usage) | GET | `/api/v1/admin/bot-traffic/usage` |
| [Read settings and crawler catalog](/docs/api/website/bot-traffic/settings/retrieve) | GET | `/api/v1/admin/websites/{websiteId}/bot-traffic/settings` |
| [Update display, verification, auth, or ingestion settings](/docs/api/website/bot-traffic/settings/update) | PATCH | `/api/v1/admin/websites/{websiteId}/bot-traffic/settings` |
| [Read token status](/docs/api/website/bot-traffic/authentication-token/retrieve) | GET | `/api/v1/admin/websites/{websiteId}/bot-traffic/token` |
| [Create a request-auth token](/docs/api/website/bot-traffic/authentication-token/create) | POST | `/api/v1/admin/websites/{websiteId}/bot-traffic/token` |
| [Rotate a request-auth token](/docs/api/website/bot-traffic/authentication-token/rotate) | PUT | `/api/v1/admin/websites/{websiteId}/bot-traffic/token` |
| [Delete a request-auth token](/docs/api/website/bot-traffic/authentication-token/delete) | DELETE | `/api/v1/admin/websites/{websiteId}/bot-traffic/token` |

## Query Bot traffic

```sh
curl "https://datafa.st/api/v1/analytics/bot-traffic?startAt=2026-08-01&endAt=2026-08-31&category=answer_fetch&verification=ip_verified" \
  -H "Authorization: Bearer df_xxx"
```

Omit the dates for the last 30 calendar days. Supported query parameters:

| Parameter | Values |
|---|---|
| `startAt`, `endAt` | Both together: `YYYY-MM-DD` or ISO 8601 with `Z`/UTC offset |
| `timezone` | IANA timezone; defaults to the website timezone |
| `interval` | `hour`, `day`, `week`, or `month` |
| `category` | `all`, `answer_fetch`, `search_index`, `training`, or `ai_crawler` |
| `verification` | `all`, `ip_verified`, or `user_agent_only` |
| `company` | Exact company returned by the API, such as `OpenAI` |
| `crawlerId` | Exact ID returned in `crawlers[].id`, such as `OpenAI::ChatGPT-User::answer_fetch` |

The summary response includes direct fields such as `totalRequests`, `aiAnswerRequests`, `indexingRequests`, `trainingRequests`, `ipVerifiedRequests`, `crawlers`, and `timeseries`. An agent does not need to reinterpret chart labels to answer a normal question.

## Query requested pages

```sh
curl "https://datafa.st/api/v1/analytics/bot-traffic/pages?startAt=2026-08-01&endAt=2026-08-31&crawlerId=OpenAI%3A%3AChatGPT-User%3A%3Aanswer_fetch&limit=50" \
  -H "Authorization: Bearer df_xxx"
```

This endpoint accepts the same filters plus `pageSearch`, `limit` (1–100), and `offset`. Each page includes total requests, status-code counts, and exact crawler counts.

## Read allowance and reset date

```sh
curl "https://datafa.st/api/v1/admin/bot-traffic/usage" \
  -H "Authorization: Bearer dft_xxx"
```

This is account-wide, just like the billing card, so the token must have access to all websites. It returns accepted usage, dropped requests after the limit, remaining allowance, percentage used, the current period and reset date, paid packs, and only websites that consumed requests.

Buying or changing paid capacity remains a human-confirmed action in [Billing](/dashboard/billing). The API and MCP report the full state but do not charge a saved card.

## Read and update settings

First read settings. The response includes `crawlerCatalog`, with every company, exact agent ID, category, short description, and enabled state.

```sh
curl "https://datafa.st/api/v1/admin/websites/WEBSITE_ID/bot-traffic/settings" \
  -H "Authorization: Bearer dft_xxx"
```

Prefer incremental ingestion changes so an agent does not overwrite an exclusion it did not intend to change:

```sh
curl -X PATCH "https://datafa.st/api/v1/admin/websites/WEBSITE_ID/bot-traffic/settings" \
  -H "Authorization: Bearer dft_xxx" \
  -H "Content-Type: application/json" \
  -d '{
    "disableCompanies": ["Meta"],
    "enableCrawlerIds": ["OpenAI::ChatGPT-User::answer_fetch"],
    "verifiedOnly": true
  }'
```

Supported fields:

- Ingestion: `disableCompanies`, `enableCompanies`, `disableCrawlerIds`, `enableCrawlerIds`, or a full `excludedCrawlerIds` replacement.
- Verification and authentication: `verifiedOnly`, `requireAuthToken`.
- Display only: `hiddenCrawlers`, `hideDashboardCard`, `groupBy`, `pageFilterIds`.

`hiddenCrawlers` only changes what the chart shows. It does not stop ingestion or reduce usage. Company/agent exclusions stop future matching requests before storage, so skipped data cannot be recovered.

## Request-authentication token

```sh
# Create (POST) or rotate (PUT)
curl -X POST "https://datafa.st/api/v1/admin/websites/WEBSITE_ID/bot-traffic/token" \
  -H "Authorization: Bearer dft_xxx"
```

The full `requestAuthToken` is returned only once on create or rotate. Save it in a server-side secret, deploy it in your tracker, then set `requireAuthToken: true`. Rotating invalidates the previous token immediately. Deleting the token also turns authentication enforcement off.

## MCP tools

The same workflows are available as:

- `datafast_bot_traffic_summary`, `datafast_bot_traffic_pages`, `datafast_bot_traffic_usage`
- `datafast_bot_traffic_settings_get`, `datafast_bot_traffic_settings_update`
- `datafast_bot_traffic_token_get`, `_create`, `_rotate`, and `_delete`

MCP asks for explicit confirmation before any ingestion/auth change or credential action. See the [complete MCP tool reference](/docs/mcp-tools) and try the REST endpoints in the [API playground](/docs/api/playground?endpoint=bot-traffic-summary).

## Code examples

### Get IP-verified AI answer requests

```bash
curl "https://datafa.st/api/v1/analytics/bot-traffic?category=answer_fetch&verification=ip_verified&startAt=2026-08-01&endAt=2026-08-31" \
  -H "Authorization: Bearer df_xxx"
```

### Success response

```json
{
  "status": "success",
  "data": [{
    "websiteId": "665f0b3c4d2e1a0012345678",
    "domain": "example.com",
    "totalRequests": 12840,
    "aiAnswerRequests": 12840,
    "ipVerifiedRequests": 12840,
    "crawlers": [{
      "id": "OpenAI::ChatGPT-User::answer_fetch",
      "company": "OpenAI",
      "userAgent": "ChatGPT-User",
      "category": "answer_fetch",
      "requestCount": 8840
    }],
    "timeseries": []
  }]
}
```