Can I reach it?

An access map of the agent web built from behavior, not declarations. We ask the top 1000 domains the same question as five different callers and record what actually comes back: the status, the challenge, the toll, and whether the site drew us a detour.

The finding

Being an agent is cheap. Being a recognised agent is expensive.

An ordinary browser was denied by robots.txt on 31 of 1000 domains. The same request wearing a published vendor token was denied on 103 (gptbot) and 105 (claudebot).

An agent that identified itself honestly but under a name nobody has heard of was reached 55.3% of the time — slightly better than the browser control at 53.9%. The penalty attaches to the name on the door, not to being automated.

By caller

Caller	Asked	robots denied	Reached	Rate	Challenged	Blocked
`browser` human-baseline	969	31	522	53.9%	65	26
`curl` naive-script	969	31	500	51.6%	69	42
`canireach` self-identified-agent	969	31	536	55.3%	51	26
`gptbot` vendor-token-disclosed	897	103	411	45.8%	75	50
`claudebot` vendor-token-disclosed	895	105	415	46.4%	69	40

“Asked” counts requests we actually sent. Where robots.txt denied us we did not send one, so those are excluded from the rate rather than counted as the site refusing.

Who is drawing detours

Of 1000 domains, these publish a machine-readable affordance an agent can follow instead of guessing.

File	Present	Share	Soft 404
`llms_txt`	102	10.2%	110
`agents_md`	6	0.6%	119
`wellknown_agents`	3	0.3%	101
`web_bot_auth`	7	0.7%	102

Soft 404s are sites answering 200 to a path that does not exist — counted, not discarded, because a site that says yes to everything is its own kind of unreachable.

What stopped us

Challenge	Responses
`cloudflare-challenge`	146
`akamai`	80
`recaptcha`	28
`datadome`	22
`hcaptcha`	18
`perimeterx`	15
`imperva-incapsula`	10
`aws-waf`	10

15 responses asked for money outright (HTTP 402).

What this is not

One capture is a photograph, not a trend. One capture is a photograph, not a trend. No before/after claim is published until a second comparable capture exists.
We do not impersonate. GPTBot and ClaudeBot rows are disclosed simulations, not authenticated vendor traffic; see METHODOLOGY.md at the tag target.
The instrument moved. This capture declares vantage class `unrecorded`, robots-unavailable policy `unrecorded`, robots.txt redirect policy `unrecorded` and observation slot `unrecorded`. Captures differing on any of those were taken by different instruments, and the repository withholds a cross-capture delta until every comparability dimension matches. Where a dimension reads `unrecorded` the manifest predates it, and `unrecorded` never counts as agreement with another `unrecorded`.

Provenance

Capture: 2026-08-22T0815Z (pre-2026-09-15-baseline)
Observed: 2026-08-22T08:28:13.633Z → 2026-08-22T08:40:54.831Z
Input: Tranco list 74V8X, 1000 domains
Rows: 10000
SHA-256: 6d6fe5b62eb7bd4c297fa6428653efa4b01d7abbf92792b7eee6a2cb813f4ca6
Bytes: published releases
Selection: pinned — This surface is pinned to the pre-2026-09-15 baseline so its published figures stay citable and stop moving. Captures after it were taken with a different instrument profile, so the difference between them is not all web. Every capture this project has published is downloadable with its manifest and SHA-256 from the releases page.

Recompute these numbers yourself: download the capture from the release, then node tools/aggregate.mjs <capture.jsonl>.

Comparability profile of this capture

What another capture would have to match, dimension for dimension, before the difference between it and 2026-08-22T0815Z could be called a change in the web rather than a change in us.

Dimension	Value in this capture
vantage class `vantage.class`	`"unrecorded"`
observation slot `observation_window.slot`	`"unrecorded"`
input list `input.sha256`	`"f923fe95bc629b757b8599ad0910fec1f245d5b0355458772e1cffae1b7a1453"`
row schema `instrument_policy.row_schema_version`	`"unrecorded"`
robots-unavailable policy `instrument_policy.robots_unavailable`	`"unrecorded"`
probe-target redirect policy `instrument_policy.redirects`	`"unrecorded"`
robots.txt redirect policy `instrument_policy.robots_redirects`	`"unrecorded"`
denial gate `instrument_policy.denial_gate`	`"unrecorded"`
caller set `instrument_policy.dialects`	`"unrecorded"`

"unrecorded" means this capture's manifest predates that dimension. It never counts as agreement with another "unrecorded": two captures whose policy is unknown are not thereby known to share one.