Method
We probe status, not content. Nothing here reads a page body.
What one capture does
For each of 1000 domains we fetch robots.txt and check four agent affordances, then attempt one request as each of five callers. One append-only JSONL row per observation.
Rails
- No evasion. We do not solve CAPTCHAs, rotate identities to get around a block, or retry to defeat a rate limit. A block is the measurement.
- No impersonation. Vendor-token callers append a disclosure naming this project. They are labelled simulations everywhere they appear, and that is a stated limitation of the instrument, not a footnote.
- We obey robots. Where it denies a caller we do not send that caller's request, and we report that separately from a refusal.
- No bodies, no cookies, no header maps. Rows keep policy and response metadata only.
Denominators
The headline reachability rate is over requests sent. Dividing by all attempts would fold our own compliance into the web's hostility and make the web look more hostile than it is.
Comparability
This capture was taken from a residential vantage with robots-unavailable failing open. Automated captures run from GitHub-hosted dynamic egress with robots-unavailable failing closed. Those are different instruments and the repository withholds cross-capture deltas until every comparability dimension matches.
The repository refuses to emit a cross-capture delta unless vantage class, input list, row schema, robots-unavailable policy, redirect policy, denial gate and caller set all match. An unrecorded dimension never counts as agreement with another unrecorded dimension.