The Agent Web Crawler Census

Who is actually out there

The full census

Sorted by volume. Every row observed first-hand — nothing is copied from a third-party bot list. Operator links are the ones each agent published in its own UA string.

Use it on your own traffic

The same data, with a case-insensitive matcher per agent, is served free and unmetered — no key, no payment, no rate limit worth worrying about:

curl -s https://fetchgate.dev/v1/agent-census.json | jq '.agents[] | {matcher, category}'

Classify your own access log in about five lines:

const census = await (await fetch("https://fetchgate.dev/v1/agent-census.json")).json();
const rules = census.agents.map(a => [new RegExp(a.matcher, "i"), a.category]);

const classify = (ua) =>
  rules.find(([re]) => re.test(ua))?.[1] ?? "unknown";

Licensed CC BY 4.0 — use it anywhere, just link back to this page.

Method, and what this does not show

Counts come from Cloudflare zone analytics for fetchgate.dev (httpRequestsAdaptiveGroups) over the 24 hours ending 2026-08-27T20:30:00Z. Payment counts come from the endpoint’s own payment-analytics dataset, which records every x402 verify/settle attempt at a single choke point.

Honest limitations:

• This is one endpoint. A busier or differently-listed origin will see a different mix. It is a real sample, not the whole population.
• Categories are assigned from observed request paths plus each agent’s self-description. An agent that lies in its UA string is categorised by what it did, but a sufficiently well-disguised one would be missed.
• 2,981 requests in this window were a single-hour vulnerability-scanner burst using a bare curl UA against paths like /admin.pl and /_config. Those are excluded from the named-agent census above, because they are ordinary web background radiation and have nothing to do with agents.
• “Never paid” means never paid this origin. It is not a claim that these operators never pay anyone.
• A snapshot, not a live feed. It is refreshed by hand rather than by handing a Worker an account-scoped analytics token.

So what?

If you are building a machine-payable API, the practical reading is that getting discovered is solved and getting paid is not. Listing in every x402 and MCP directory works — it reliably produces crawlers, grades, uptime badges and index entries. None of that is demand. Budget your effort accordingly, and instrument the payment path itself so you can tell a price-scraper from a customer on day one rather than day five.