Source: https://datafa.st/docs/api/website/bot-traffic/pages
Markdown source: https://datafa.st/docs/api/website/bot-traffic/pages.md
Description: Return pages requested by bots, ranked by accepted request count, with HTTP status and exact crawler breakdowns. Search `llms.txt` to see which companies fetched your LLM discovery file.

# Get requested pages

`GET https://datafa.st/api/v1/analytics/bot-traffic/pages`

Return pages requested by bots, ranked by accepted request count, with HTTP status and exact crawler breakdowns. Search `llms.txt` to see which companies fetched your LLM discovery file.

MCP equivalent: `datafast_bot_traffic_pages`. Page search is case-insensitive and supports the same wildcard syntax as the Bot traffic dashboard.

> **Related:** [Bot traffic API](/docs/api/website/bot-traffic) · [Bot traffic tracking](/docs/bot-traffic-tracking)

## Request

#### Query parameters

| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `websiteId` | string | Required with dft_ | Website ObjectId used by account tokens to choose which website to query or manage. Website ObjectId. Omit with a `df_` website key; required with a `dft_` account token. |
| `startAt` | string | No* | Start of the reporting window. Use it with `endAt` to query a specific date range instead of the endpoint default. Inclusive start as `YYYY-MM-DD` or ISO 8601 with `Z` or a UTC offset. Provide with `endAt`; both default to the last 30 calendar days when omitted. |
| `endAt` | string | No* | End of the reporting window. Must be paired with `startAt` so DataFast can build the date range. Inclusive end as `YYYY-MM-DD` or ISO 8601 with `Z` or a UTC offset. Provide with `startAt`. |
| `timezone` | string | No | Timezone used to interpret dates and group analytics buckets. Defaults to the website timezone. IANA timezone such as `America/New_York`, `Europe/Paris`, or `UTC`. Defaults to the website timezone. |
| `interval` | string | No | Controls how time-series rows are grouped. For example, `day` returns one row per day and `hour` returns one row per hour. `hour`, `day`, `week`, or `month`. Inferred from the requested range when omitted. |
| `category` | string | No | `all`, `answer_fetch`, `search_index`, `training`, or `ai_crawler`. Defaults to `all`. |
| `verification` | string | No | `all`, `ip_verified`, or `user_agent_only`. Defaults to `all` accepted requests. |
| `company` | string | No | Exact company name returned by the API, such as `OpenAI`. Defaults to `all`. |
| `crawlerId` | string | No | Exact ID returned in `crawlers[].id`, such as `OpenAI::ChatGPT-User::answer_fetch`. |
| `pageSearch` | string | No | Case-insensitive text or wildcard search across hostname and path. Example: `llms.txt` or `/docs/*`. |
| `limit` | number | No | Maximum number of rows returned in one response. Use with `offset` to paginate through long result sets. |
| `offset` | number | No | Number of rows to skip before returning results. Use it with `limit` for pagination. |

## Response

Returns a JSON object with `status: "success"` and endpoint-specific fields in `data` (and `pagination` when the endpoint is paginated).

#### Response fields

| Field | Type | Description |
| --- | --- | --- |
| `data[].id` | string | Stable `hostname::path` page identifier. |
| `data[].hostname` | string | Requested hostname. |
| `data[].path` | string | Requested path, including files such as `/llms.txt`. |
| `data[].requestCount` | number | Accepted requests for the page. |
| `data[].ipVerifiedRequests` | number | IP-verified requests for the page. |
| `data[].userAgentOnlyRequests` | number | User-agent-only requests for the page. |
| `data[].statusCodes[]` | array | HTTP status codes and request counts. |
| `data[].crawlers[]` | array | Exact crawler IDs, companies, user agents, categories, and counts. |
| `context` | object | Website, range, and resolved filters used for the query. |
| `pagination` | object | `limit`, `offset`, `hasMore`, and `nextOffset`. |

## Authentication

Use a `df_` website API key for one website, or a `dft_` account token with `analytics:read`.

With `df_`, the website is inferred from the key. With `dft_`, add `websiteId` to the query and make sure the token can access that website.

The account must have active DataFast API access. Read [authentication and scopes](/docs/api/authentication) for token creation and permissions.

### Errors

**400** — Invalid filters, date range, timezone, interval, limit, or offset.

**404** — Website not found or the caller is not its owner.

**500** — Bot traffic pages could not be queried.

See [API errors](/docs/api#errors) for the standard error envelope, auth failures, validation errors, permission errors, and rate limits.

## Code examples

### Example request

```bash
curl -X GET "https://datafa.st/api/v1/analytics/bot-traffic/pages?pageSearch=llms.txt&startAt=2026-08-01&endAt=2026-08-31&limit=50" \
  -H "Authorization: Bearer df_xxx"
```

### Success response

```json
{
  "status": "success",
  "data": [{
    "id": "example.com::/llms.txt",
    "hostname": "example.com",
    "path": "/llms.txt",
    "requestCount": 42,
    "ipVerifiedRequests": 42,
    "userAgentOnlyRequests": 0,
    "statusCodes": [{ "statusCode": 200, "requestCount": 42 }],
    "crawlers": [{
      "id": "OpenAI::ChatGPT-User::answer_fetch",
      "company": "OpenAI",
      "userAgent": "ChatGPT-User",
      "category": "answer_fetch",
      "requestCount": 30
    }]
  }],
  "context": {
    "websiteId": "665f0b3c4d2e1a0012345678",
    "domain": "example.com"
  },
  "pagination": {
    "limit": 50,
    "offset": 0,
    "hasMore": false,
    "nextOffset": null
  }
}
```