Source: https://datafa.st/docs/bot-traffic-tracking
Markdown source: https://datafa.st/docs/bot-traffic-tracking.md
Description: Track AI assistants, search crawlers, and model-training bots from your server so you can see which pages crawlers request.

# Bot traffic tracking

See when AI assistants, search engines, and model-training crawlers request pages on your website.

![DataFast bot traffic analytics](/blog-ai-bot-traffic-analytics.jpg)

## Get started

Install the server-side package, add one tracking call in your backend, then deploy. This is separate from the normal DataFast browser tracking script.

> See [@datafast/ai-crawl on NPM](https://www.npmjs.com/package/@datafast/ai-crawl)

> Bot traffic tracking is included in your DataFast subscription. It is not a separate paid add-on.

### 1. Install the package

~~~bash
npm install @datafast/ai-crawl
~~~

### 2. Add it to your server proxy

Here is the most common setup for a Next.js app hosted on Vercel (more examples below):

~~~ts
// proxy.ts
import {
  NextResponse,
  type NextFetchEvent,
  type NextRequest,
} from "next/server";
import { trackAICrawlerRequest } from "@datafast/ai-crawl";

export function proxy(request: NextRequest, event: NextFetchEvent) {
  // 👇 DataFast AI crawler tracking starts here 👇
  trackAICrawlerRequest(request, event, {
    websiteId: "dfid_******",
  });
  // 👆 DataFast AI crawler tracking ends here 👆

  return NextResponse.next();
}

export const config = {
  // Optional: avoids running proxy for obvious API/static asset requests.
  // Keep crawler-facing files trackable: robots.txt, llms.txt, and sitemaps.
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};
~~~

> This is server-side tracking. In modern Next.js, put it in <code>proxy.ts</code> (previously called <code>middleware.ts</code> / <code>middleware.js</code>), or use your backend, edge function, or worker. Pass the runtime context, such as <code>event</code> or <code>context</code>, so the package can use <code>waitUntil</code> internally. ❌ Do not await <code>trackAICrawlerRequest</code>; call it, then return your response.

### 3. Deploy and check your dashboard

After deployment, open your DataFast dashboard and look for the **Bot traffic** card. You can filter by AI answers, indexing, training, and IP verification confidence.

Using Cloudflare, Express, Hono, or another backend? Use the platform examples below.

## What this tracks

AI tools and search crawlers now read your site before many users do. The pages they request show what they are trying to answer, index, or learn from, including missing URLs they expected to find.

DataFast groups bot traffic into three main categories:

| Category | Practical example | Why it matters |
| --- | --- | --- |
| AI answers | A user asks ChatGPT about your product, and ChatGPT requests your pricing or docs page to answer accurately. | Shows which pages AI assistants fetch when users ask questions. |
| Indexing | Googlebot, Bingbot, or PerplexityBot requests your pages to update search or answer indexes. | Shows which companies are discovering and refreshing your content. |
| Training | Anthropic's ClaudeBot, OpenAI's GPTBot, Applebot, Google Cloud Vertex Bot, Bytespider, or another training crawler requests public content. | Shows which crawlers are collecting public pages that may be used for model training or large-scale datasets. |

<bot-traffic-icons></bot-traffic-icons>

See the public [crawler directory](/crawlers) for the full list of crawler user agents, providers, categories, and verification sources DataFast tracks.

If a crawler repeatedly requests <code>/free-trial</code>, <code>/docs/get-started</code>, or another path that does not exist, that can be a useful content signal. It may mean users, agents, or crawlers expect that page to exist.

### Crawler-facing files

DataFast also tracks crawler-facing files when known bots request them:

- <code>/robots.txt</code>
- <code>/llms.txt</code>
- <code>/llms-full.txt</code>
- <code>/sitemap.xml</code> and sitemap XML files
- content files such as <code>/docs/setup.md</code> or <code>/startup/example.md</code>

These files matter because AI assistants, search engines, and training crawlers often request them before crawling the rest of your site. Seeing those requests helps you know whether bots are discovering your AI/SEO instructions and structured content.

## How the package works

The package runs in your backend, middleware, edge function, or worker. For each request, it quickly ignores obvious static assets, API routes, framework internals, and normal human browser traffic. If the request looks like bot traffic, it sends a small event to DataFast.

Crawler-facing discovery files are intentionally not treated like static assets. Requests to <code>robots.txt</code>, <code>llms.txt</code>, <code>llms-full.txt</code>, sitemap XML files, and markdown content can appear in Bot traffic when they come from known crawlers.

DataFast then classifies the provider, crawler type, confidence, and IP verification on the server. This keeps crawler lists and IP ranges up to date without asking every customer to upgrade the npm package each time a crawler changes.

**The tracking request is best-effort and should not slow down your site.** On Vercel, Cloudflare, and other runtimes with <code>waitUntil</code>, the package uses it internally when you pass the runtime context. Call <code>trackAICrawlerRequest</code>, return your response immediately, and DataFast finishes in the background. Do not await it in proxy/middleware.

> **Bot traffic tracking runs server-side** because AI crawlers often request raw HTML and skip frontend JavaScript.

## Platform examples

### Next.js / Vercel Proxy

Create or update <code>proxy.ts</code> (previously called <code>middleware.ts</code> / <code>middleware.js</code>):

~~~ts
// proxy.ts
import {
  NextResponse,
  type NextFetchEvent,
  type NextRequest,
} from "next/server";
import { trackAICrawlerRequest } from "@datafast/ai-crawl";

export function proxy(request: NextRequest, event: NextFetchEvent) {
  trackAICrawlerRequest(request, event, {
    websiteId: "dfid_******",
  });

  return NextResponse.next();
}

export const config = {
  // Keep robots.txt, llms.txt, and sitemap files reachable by this proxy.
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};
~~~

Vercel provides <code>event.waitUntil</code>, so the DataFast request is scheduled in the background. Middleware runs before the final page response, so status code is usually stored as unknown.

### Cloudflare Pages

Create <code>functions/_middleware.ts</code>:

~~~ts
import { trackAICrawlerRequest } from "@datafast/ai-crawl";

export async function onRequest(context) {
  trackAICrawlerRequest(context.request, context, {
    websiteId: "dfid_******",
  });

  return context.next();
}
~~~

Cloudflare Pages provides <code>context.waitUntil</code>. The package uses it internally, so Cloudflare can continue returning the HTML response while the DataFast request runs in the background.

For better Cloudflare billing and performance hygiene, exclude obvious static assets from Pages Functions invocation when your framework does not generate this automatically. Add <code>_routes.json</code> to your build output:

~~~json
{
  "version": 1,
  "include": ["/*"],
  "exclude": [
    "/assets/*",
    "/static/*",
    "/_next/*",
    "/favicon.ico",
    "/*.css",
    "/*.js",
    "/*.png",
    "/*.jpg",
    "/*.svg",
    "/*.ico",
    "/*.woff2"
  ]
}
~~~

The package still filters static assets locally, but <code>_routes.json</code> prevents Cloudflare from invoking your Function for obvious asset requests. Do not exclude crawler-facing files such as <code>/robots.txt</code>, <code>/llms.txt</code>, <code>/llms-full.txt</code>, or <code>/sitemap.xml</code> if you want to see crawler activity on those files.

### Cloudflare Workers

Wrap your Worker handler with <code>withAICrawlerTracking</code>:

~~~ts
import { withAICrawlerTracking } from "@datafast/ai-crawl";

export default {
  fetch: withAICrawlerTracking(
    async (request: Request, env: Env, ctx: ExecutionContext) => {
      return fetch(request);
    },
    {
      websiteId: "dfid_******",
    },
  ),
};
~~~

This version can capture status code because the wrapper sees the response your handler created. It still uses <code>ctx.waitUntil</code> when available, so tracking is best-effort and non-blocking.

### Custom Docker, Cloud Run, or reverse proxies

Most backends expose the public request URL automatically. If your runtime instead gives the package an internal hostname such as <code>localhost</code> or <code>0.0.0.0</code>, set the public origin explicitly:

~~~ts
trackAICrawlerRequest(request, context, {
  websiteId: "dfid_******",
  publicOrigin: "https://example.com",
});
~~~

DataFast preserves the requested path and query string and still validates the resulting hostname against your website configuration.

### Express

~~~ts
import express from "express";
import { createExpressAICrawlerMiddleware } from "@datafast/ai-crawl";

const app = express();

app.use(
  createExpressAICrawlerMiddleware({
    websiteId: "dfid_******",
  }),
);
~~~

The Express middleware calls <code>next()</code> immediately. It attaches a <code>finish</code> listener and sends the bot traffic event after the response has already been sent, so your app does not wait for DataFast before continuing.

### Hono

~~~ts
import { Hono } from "hono";
import { trackAICrawlerResponse } from "@datafast/ai-crawl";

const app = new Hono();

app.use("*", async (c, next) => {
  await next();

  trackAICrawlerResponse(c.req.raw, c.res, c.executionCtx, {
    websiteId: "dfid_******",
  });
});
~~~

Use this when your Hono runtime gives you access to both the final response and an execution context.

### Generic Request / Response handler

If your backend gives you a standard <code>Request</code> and <code>Response</code>, track after your app creates the response:

~~~ts
import { trackAICrawlerResponse } from "@datafast/ai-crawl";

export async function handleRequest(request, context) {
  const response = await yourAppHandler(request);

  trackAICrawlerResponse(request, response, context, {
    websiteId: "dfid_******",
  });

  return response;
}
~~~

If your backend only gives you the request before the response exists, use request-only tracking:

~~~ts
import { trackAICrawlerRequest } from "@datafast/ai-crawl";

export function middleware(request, context) {
  trackAICrawlerRequest(request, context, {
    websiteId: "dfid_******",
  });

  return next();
}
~~~

Request-only tracking is enough to know which page the bot tried to crawl. Response-aware tracking only adds status code when it is easy to get.

## Optional request authentication

You can add a website-specific Bot traffic token without interrupting an existing integration:

![Bot traffic request authentication setting](/bot-traffic-request-authentication-settings.jpg)

1. Open the **Bot traffic** card settings and create a token.
2. Add it to your server-side package configuration.
3. After the new configuration is deployed, enable **Reject unauthenticated requests**.

~~~ts
trackAICrawlerRequest(request, event, {
  websiteId: "dfid_******",
  authToken: process.env.DATAFAST_BOT_TOKEN,
});
~~~

Keep the <code>dfbot_...</code> token in a server-side environment variable. Never expose it in frontend JavaScript, a public repository, logs, or a URL. Authentication is optional and enforcement is off by default, so token validation does not affect whether requests are accepted until you enable **Reject unauthenticated requests**. If you rotate the token, the previous token stops working immediately. Deleting the token automatically turns request authentication off.

## Use without Node.js (PHP or any backend)

You do not need Node.js or the npm package. Any backend that can send an HTTPS POST request can report a crawler request directly.

~~~http
POST https://datafa.st/api/ai-crawls
Content-Type: application/json
Authorization: Bearer dfbot_******
~~~

The <code>Authorization</code> header is optional unless you enabled **Reject unauthenticated requests** for this website. Create the token in the Bot traffic card settings and keep it only on your server.

The endpoint does not use your general DataFast <code>df_...</code> API key. Your <code>dfid_...</code> website ID is a public tracking identifier, just like the ID in the normal browser tracking script. Never put an account API key, password, cookie, or another unrelated secret in this request.

### Request body

~~~json
{
  "websiteId": "dfid_******",
  "domain": "example.com",
  "href": "https://example.com/docs/get-started",
  "ai": {
    "userAgent": "Mozilla/5.0 ... ChatGPT-User/1.0",
    "ip": "203.0.113.10",
    "statusCode": 200,
    "source": "server_middleware"
  }
}
~~~

| Field | Required | What to send |
| --- | --- | --- |
| <code>websiteId</code> | Yes | Your public DataFast website tracking ID, starting with <code>dfid_</code>. |
| <code>domain</code> | Yes | The hostname that received the crawler request, such as <code>example.com</code>. |
| <code>href</code> | Yes | The absolute public URL the crawler requested. The hostname must belong to this DataFast website. |
| <code>ai.userAgent</code> | Yes for direct integrations | The original request's complete <code>User-Agent</code> value. DataFast uses this to classify the crawler on its servers. |
| <code>ai.ip</code> | Recommended | The original crawler's source IP as observed by your server or trusted proxy. This allows DataFast to compare it with published crawler IP ranges. |
| <code>ai.statusCode</code> | Optional | Your response status as an integer from <code>100</code> to <code>599</code>. Omit it if the response is not available yet. |
| <code>ai.source</code> | Yes for direct integrations | Use <code>server_middleware</code>. |

Do not send <code>provider</code>, <code>agent</code>, <code>category</code>, or a verification result. Those values are derived again by DataFast instead of being trusted from the integration.

The JSON body must be smaller than 16 KB. A tracked or safely ignored request normally returns <code>200</code> with <code>{"success":true}</code>. Invalid requests return a <code>4xx</code> response, including <code>429</code> when the caller is sending too quickly. Tracking is best-effort: use a short timeout and do not delay or fail your website response when DataFast is unavailable.

### PHP example

Run this from server-side PHP after your application has decided what response to return. The local user-agent check is only a bandwidth pre-filter; DataFast performs the final classification.

~~~php
<?php

function trackDataFastCrawler(string $websiteId, ?string $authToken = null): void
{
    $method = strtoupper($_SERVER['REQUEST_METHOD'] ?? 'GET');
    if (!in_array($method, ['GET', 'HEAD'], true)) {
        return;
    }

    $userAgent = $_SERVER['HTTP_USER_AGENT'] ?? '';
    $crawlerHints = [
        'bot', 'crawler', 'spider', 'chatgpt', 'gptbot', 'claude',
        'perplexity', 'bing', 'google', 'applebot', 'bytespider', 'ccbot'
    ];

    $normalizedUserAgent = strtolower($userAgent);
    $looksLikeCrawler = false;
    foreach ($crawlerHints as $hint) {
        if (strpos($normalizedUserAgent, $hint) !== false) {
            $looksLikeCrawler = true;
            break;
        }
    }

    if (!$looksLikeCrawler) {
        return;
    }

    $host = strtolower($_SERVER['HTTP_HOST'] ?? '');
    $host = preg_replace('/:\d+$/', '', $host);
    if (!$host) {
        return;
    }

    $isHttps = !empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] !== 'off';
    $scheme = $isHttps ? 'https' : 'http';
    $path = parse_url($_SERVER['REQUEST_URI'] ?? '/', PHP_URL_PATH) ?: '/';

    $payload = [
        'websiteId' => $websiteId,
        'domain' => $host,
        // Query parameters are deliberately omitted to avoid sending secrets or PII.
        'href' => $scheme . '://' . $host . $path,
        'ai' => [
            'userAgent' => $userAgent,
            'ip' => $_SERVER['REMOTE_ADDR'] ?? null,
            'statusCode' => http_response_code(),
            'source' => 'server_middleware',
        ],
    ];

    $headers = ['Content-Type: application/json'];
    if ($authToken) {
        $headers[] = 'Authorization: Bearer ' . $authToken;
    }

    $request = curl_init('https://datafa.st/api/ai-crawls');
    curl_setopt_array($request, [
        CURLOPT_POST => true,
        CURLOPT_HTTPHEADER => $headers,
        CURLOPT_POSTFIELDS => json_encode($payload, JSON_UNESCAPED_SLASHES),
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_CONNECTTIMEOUT_MS => 300,
        CURLOPT_TIMEOUT_MS => 1000,
    ]);

    curl_exec($request);
    curl_close($request);
}

trackDataFastCrawler(
    'dfid_******',
    getenv('DATAFAST_BOT_TOKEN') ?: null
);
~~~

This example uses <code>REMOTE_ADDR</code>, which is the safe default when PHP receives traffic directly. If your application is behind Cloudflare, a load balancer, or another reverse proxy, it may contain the proxy's IP instead of the crawler's IP. Only read <code>CF-Connecting-IP</code>, <code>X-Forwarded-For</code>, or a similar header after your infrastructure is configured to accept traffic exclusively from that trusted proxy and to remove client-supplied copies. Otherwise an attacker can spoof the crawler IP.

### Privacy and safety checklist

- Call this endpoint only from your backend. Do not add it to browser JavaScript or call it for normal human traffic.
- Send only the fields above. Never forward the crawler request's headers, cookies, authorization value, request body, or server environment variables. The only authorization value sent to DataFast should be the optional <code>dfbot_...</code> token you created for this website.
- Prefer URLs without query parameters, as in the PHP example. If query parameters are essential, remove any value that can contain a token, email address, search text, customer ID, or other personal data first.
- Treat forwarded IP headers as untrusted unless the request came through a proxy you control and trust.
- Use a short timeout, ignore network failures, and pre-filter obvious non-crawler requests so analytics can never slow down your page response.

DataFast still validates the website ID, checks that the URL hostname belongs to that website, reclassifies the user agent on the server, and records IP verification confidence separately. The dashboard shows IP-verified crawler traffic by default.

## Optional category filters

By default, DataFast tracks all relevant bot traffic categories. You can disable categories if you only care about specific crawler types:

~~~ts
trackAICrawlerRequest(request, event, {
  websiteId: "dfid_******",

  disableAnswerFetch: true,
  disableSearchCrawlers: true,
  disableTrainingCrawlers: true,
  disableOtherCrawlers: true,
});
~~~

Most websites should keep the defaults. The dashboard lets you filter the data later by AI answers, indexing, training, and verification confidence.

## Where to find the data

After installing the package, open your DataFast dashboard and look for the **Bot traffic** card. You can filter by crawler type, show only IP-verified crawlers, and inspect which pages each provider requested.

You can also browse the public [crawler directory](/crawlers) to see the crawlers, user-agent tokens, IP verification sources, and official references behind the tracking.

If you do not see data immediately, that usually means no known crawler has requested your server-rendered pages yet. Human pageviews do not appear in this card.