Get requested pages

GET https://datafa.st/api/v1/analytics/bot-traffic/pages

Return pages requested by bots, ranked by accepted request count, with HTTP status and exact crawler breakdowns. Search llms.txt to see which companies fetched your LLM discovery file.

MCP equivalent: datafast_bot_traffic_pages. Page search is case-insensitive and supports the same wildcard syntax as the Bot traffic dashboard.

Related: Bot traffic API · Bot traffic tracking

Try on Playground

Request

Query parameters

websiteId

string

Website ObjectId used by account tokens to choose which website to query or manage. Website ObjectId. Omit with a df_ website key; required with a dft_ account token.

startAt

string

Start of the reporting window. Use it with endAt to query a specific date range instead of the endpoint default. Inclusive start as YYYY-MM-DD or ISO 8601 with Z or a UTC offset. Provide with endAt; both default to the last 30 calendar days when omitted.

endAt

string

End of the reporting window. Must be paired with startAt so DataFast can build the date range. Inclusive end as YYYY-MM-DD or ISO 8601 with Z or a UTC offset. Provide with startAt.

timezone

string

Timezone used to interpret dates and group analytics buckets. Defaults to the website timezone. IANA timezone such as America/New_York, Europe/Paris, or UTC. Defaults to the website timezone.

interval

string

Controls how time-series rows are grouped. For example, day returns one row per day and hour returns one row per hour. hour, day, week, or month. Inferred from the requested range when omitted.

category

string

all, answer_fetch, search_index, training, or ai_crawler. Defaults to all.

verification

string

all, ip_verified, or user_agent_only. Defaults to all accepted requests.

company

string

Exact company name returned by the API, such as OpenAI. Defaults to all.

crawlerId

string

Exact ID returned in crawlers[].id, such as OpenAI::ChatGPT-User::answer_fetch.

pageSearch

string

Case-insensitive text or wildcard search across hostname and path. Example: llms.txt or /docs/*.

limit

number

Maximum number of rows returned in one response. Use with offset to paginate through long result sets.

offset

number

Number of rows to skip before returning results. Use it with limit for pagination.

Response

Returns a JSON object with status: "success" and endpoint-specific fields in data (and pagination when the endpoint is paginated).

Response fields

data[].id

string

Stable hostname::path page identifier.

data[].hostname

string

Requested hostname.

data[].path

string

Requested path, including files such as /llms.txt.

data[].requestCount

number

Accepted requests for the page.

data[].ipVerifiedRequests

number

IP-verified requests for the page.

data[].userAgentOnlyRequests

number

User-agent-only requests for the page.

data[].statusCodes[]

array

HTTP status codes and request counts.

data[].crawlers[]

array

Exact crawler IDs, companies, user agents, categories, and counts.

context

object

Website, range, and resolved filters used for the query.

pagination

object

limit, offset, hasMore, and nextOffset.

Authentication

Use a df_ website API key for one website, or a dft_ account token with analytics:read.

With df_, the website is inferred from the key. With dft_, add websiteId to the query and make sure the token can access that website.

The account must have active DataFast API access. Read authentication and scopes for token creation and permissions.

Errors

400 — Invalid filters, date range, timezone, interval, limit, or offset.

404 — Website not found or the caller is not its owner.

500 — Bot traffic pages could not be queried.

See API errors for the standard error envelope, auth failures, validation errors, permission errors, and rate limits.

✍️ Something missing? Suggest features.

🤖 AI agent or LLM? Read this page as markdown

{ "status": "success", "data": [{ "id": "example.com::/llms.txt", "hostname": "example.com", "path": "/llms.txt", "requestCount": 42, "ipVerifiedRequests": 42, "userAgentOnlyRequests": 0, "statusCodes": [{ "statusCode": 200, "requestCount": 42 }], "crawlers": [{ "id": "OpenAI::ChatGPT-User::answer_fetch", "company": "OpenAI", "userAgent": "ChatGPT-User", "category": "answer_fetch", "requestCount": 30 }] }], "context": { "websiteId": "665f0b3c4d2e1a0012345678", "domain": "example.com" }, "pagination": { "limit": 50, "offset": 0, "hasMore": false, "nextOffset": null } }