Get Bot traffic settings

GET https://datafa.st/api/v1/admin/websites/{websiteId}/bot-traffic/settings

Return a website's Bot traffic display, verification, authentication, and ingestion settings, plus the complete company and agent catalog. Read this first before changing individual crawlers.

MCP equivalent: datafast_bot_traffic_settings_get. hiddenCrawlers changes the chart only; excludedCrawlerIds stops future matching requests before storage.

Try on Playground

Request

Path parameters

websiteId

string

required

Website ObjectId used by account tokens to choose which website to query or manage. Website ObjectId. Get it from the dashboard or the List websites API.

Response

Returns a JSON object with status: "success" and endpoint-specific fields in data (and pagination when the endpoint is paginated).

Response fields

data[].websiteId

string

Website ObjectId used by account tokens to choose which website to query or manage.

data[].domain

string

Tracked domain.

data[].settings.hiddenCrawlers

string[]

Crawler IDs hidden from dashboard charts only.

data[].settings.excludedCrawlerIds

string[]

Crawler IDs excluded from future ingestion.

data[].settings.verifiedOnly

boolean

Whether only IP-verified crawler matches are accepted.

data[].settings.requireAuthToken

boolean

Whether incoming requests must include the website token.

data[].settings.hideDashboardCard

boolean

Whether the Bot traffic card is hidden.

data[].settings.groupBy

string

crawlers or pages.

data[].settings.pageFilterIds

string[]

Selected page identifiers used by the dashboard.

data[].crawlerCatalog[]

array

Companies with enabled, disabled, or partial status and their agents.

Authentication

Requires an owner dft_ account token with settings:read. Website API keys (df_) cannot call this endpoint.

The token must be allowed to access the website in the path, and that website must belong to the token owner.

The account must have active DataFast API access. Read authentication and scopes for token creation and permissions.

Errors

400 — Invalid website ID.

404 — Website not found or the caller is not its owner.

See API errors for the standard error envelope, auth failures, validation errors, permission errors, and rate limits.

✍️ Something missing? Suggest features.

🤖 AI agent or LLM? Read this page as markdown

{ "status": "success", "data": [{ "websiteId": "665f0b3c4d2e1a0012345678", "domain": "example.com", "settings": { "hiddenCrawlers": [], "excludedCrawlerIds": ["Common Crawl::CCBot::training"], "verifiedOnly": true, "hideDashboardCard": false, "groupBy": "crawlers", "pageFilterIds": [], "requireAuthToken": false }, "crawlerCatalog": [{ "company": "OpenAI", "status": "enabled", "enabledAgentCount": 3, "totalAgentCount": 3, "agents": [{ "id": "OpenAI::ChatGPT-User::answer_fetch", "userAgent": "ChatGPT-User", "category": "answer_fetch", "description": "Fetches pages to answer a ChatGPT user.", "enabled": true }] }] }] }