The Open Web Is Closed to Agents: A Field Tally of 403s, Logins and Bot Walls
tl;dr — for human consumption
In one October afternoon, four research agents asked 21 platforms for their rules on links and self-promotion. The rules page was refused on eleven of them with a 403, a consent wall or an empty JavaScript shell, while the product pages around it served fine. Elsewhere the same month: three of twenty publisher sites refused every automated reader, six of fourteen creators had no contact an agent could reach, and a cloud agent cited subreddit rules 21 months stale while its own sample contradicted them. Bot walls do not stop agent research. They move it to sources the site does not control, and the result reads as verified unless the agent says which copy it read.
The rules page is the one that refuses
The page that tells an agent how to behave on your platform is the page it is least likely to be allowed to read. Across four projects this month, the policy page, the contact page and the install docs were refused more often than anything around them, and the refusal was never the expensive part. The expensive part was what the agent did next. Every observation below is from a Claude Code or claude.ai session between 4 and 21 October 2026, with the status line as the agent saw it.
The clearest sample came from one afternoon. Four research agents were asked, for 21 platforms, what the rules say about posting a link, a survey or a self-promotion, with one instruction: the platform’s own page first, fetched, never from memory. Here is what the platform’s own page said back.
| Platform | Page asked for | Answer |
|---|---|---|
| X | help centre, two articles | 403, both |
| Meetup | help centre | 403 |
| Quora | help centre | 403 |
| Discord | support | 403 |
| Substack | support | 403 to curl, to the fetch tool and to a reader proxy |
| Meta | community standards | 400 |
| Meta | ad policies | JavaScript only, no text |
| ads help | a Salesforce shell, no content | |
| a founder group’s rules | 400 and a cookie wall |
|
| community guidelines | consent wall, no content | |
| Indie Hackers | guidelines | 404, there is no page |
| TikTok, YouTube, Product Hunt, Bluesky, Luma, Telegram, WhatsApp, ICO | support, help, FAQ or terms | 200, the text |
Eleven refused, eight served. The product pages on the refusing platforms answered normally; it is the help centre, with its own bot protection, that an agent cannot reach, and the help centre is where the rule lives.
The pattern held wherever an agent went looking for the page that matters. Of twenty publisher sites on an outreach list, three refused every automated reader on every page. Of fourteen creators on another list, the business email sat behind a sign-in and a captcha, so the six who had not typed it into their About text were unreachable by anything but a human.
Refusal is by route, not by content
Reddit is the instructive case, because it refuses and serves the same resource depending on which door you knock on. One agent, one host, one Firefox user agent:
GET www.reddit.com/r/startups/about/rules.json -> 403, a 190 KB challenge page
GET api.reddit.com/r/startups/about/rules -> 403, same page
GET old.reddit.com/r/startups/about/rules.json -> 302 to /login
GET www.reddit.com/r/startups/about/rules/ -> 200, an 8 KB JavaScript shell
GET www.reddit.com/r/startups/about/rules.rss -> 200, a valid feed with zero entries
GET www.reddit.com/r/startups/new.rss?limit=100 -> 200, 100 full postsThe rules page is login-only, the rules feed is empty, and the posts feed hands over a hundred titles, authors and bodies. An agent that stops at the first 403 reports the subreddit unreachable. An agent that probes every representation gets a sample it can count on: 93 of 95 user posts in that room carried the phrase the rules require. The lesson for the agent is to try every door before concluding anything, and then to say which one it read. The lesson for the site is that the content is already public; only the useful route is closed.
The fallback ladder, and the rung that broke
When the page refuses, a research agent does not stop, which is why AI agents end up citing archived mirrors after a 403. The agents in these sessions used the same ladder, in the same order: a Wayback snapshot with its date, a reader proxy of the same URL, a third-party mirror marked secondary, and finally “no source says this”. Written into the memo as a source column, that ladder is honest: the operator can decide how much to trust a copy of X’s rules from 6 September.
The failure case is the ladder climbed without the dates. A nightly cloud session could not reach reddit.com at all, every route 403 from the container’s egress. It found an archive that serves live posts and stored rules from the same host, used it for both, and reported the rules as current. The rules payload carried a retrieved_utc of 1737583495, which is 22 January 2025, 21 months earlier. Nothing told the agent to treat age as a finding, so the memo wrote “no source says the phrase is required” while the agent’s own sample from the same archive showed 98 of 100 titles carrying it. The archive outranked the evidence because the archive looked like a source and the evidence looked like data. Drifted documentation is worse than none applies with full force to a mirror, and an operator reading the memo cannot detect that the agent read a mirror unless the memo says so.
Your own reader is a wall too
Not every wall is on the far side of the wire. Two of the refusals this month were the agent’s own tooling declining to hand over what it had fetched.
The harness fetch tool, asked to return a README verbatim, replied that it could not reproduce the page in full and summarised instead, twice. The summary dropped the markup and the exact wording, which is the whole payload when the task is “edit this file”, and a third call then guessed a wrong URL because the summary had paraphrased a URL pattern instead of quoting it. A tool that answers a politer question than the one asked costs more than one that fails, because the agent has to notice.
The second was a YouTube About tab. The fetch tool converted the page to markdown and reported no description, no email, no subscriber count, only navigation, a footer and a note that content had been truncated. The data was on the page, inside a ytInitialData script blob, which the markdown pass discards; a raw fetch and a grep gave every field. When a reader says it cut something, assume the thing you wanted is in the cut. The fix on the site’s side is the one this blog keeps arriving at from other directions: expose meaning, not data, and put the fact where a text reader finds it.
The email on that tab is the other half. It exists only behind a sign-in button, which for an agent means it does not exist; the creators an agent could hand to a human were the eight who had typed the address into their description. A paywall is an error with a price, and a login wall on a contact page is a paywall whose price nobody set.
Cloud agents arrive pre-spent
One more wall is invisible from a laptop. A remote session asked an open scholarly index for two results with no key, as its docs allow. The answer, in full:
Insufficient budget. This request has no API key, so it counts against the
free daily budget shared by everyone on your network's IP address, and that
budget is used up ($0 remaining; resets at midnight UTC). Use your own key
instead: add ?api_key=YOUR_KEY ... Keys are free and have their own budget.An agent on shared cloud infrastructure shares its egress IP’s quota with every other agent on it, so “no key needed” means “no key needed on a residential IP”. That is one observation from one session; the design lesson does not depend on the number. OpenAlex’s refusal names the cause, the remaining budget, the reset time, the exact fix and where to get it, in one message, which is an error written as a recovery instruction. The sibling index answered the same question with a bare 429, which cost a second request to learn nothing.
The takeaway: who is going to be quoted
The finding is not that sites block agents. It is that blocking an agent does not stop it from describing you. The agent reads a Wayback copy, a reader proxy or a third-party policy mirror, and writes the result into a memo its operator will act on. Even an MCP vendor’s own install docs did this: the page answered 403 and 404 to the agent sent to verify it, so the one row on a connect page an agent could not confirm was the row about that vendor, and the trust signal that travels through the agent arrived as “unverified”.
Designing for AI agents here is not a product. Agent-friendly policy pages are the same pages served as text to any user agent, the way the product pages already are. Print the contact address in the page text rather than behind a button. Put a date on the page, because the way to prevent agents citing cached pages is to make the live one carry an age the cached one cannot fake. This post is readable with curl, and its markdown twin is one URL away.
The agent-side fix is discipline. Write the fallback order into the research rule, not into the agent’s judgment. Probe every representation before calling a resource unreachable. Date every checked claim and carry the representation it was read from next to it, so “the rules say” becomes “the rules, as archived on 22 January 2025, said”.
A site that blocks agents does not keep its facts private. It outsources the telling of them, and the agent will cite whoever answered.
this page as your agent reads it → /posts/open-web-closed-to-agents.md