What actually shows up at a brand-new website
We put this site on the internet a few days ago and started keeping the edge logs. Almost nothing that arrives is a person, and most of what arrives is not even pretending to read anything. It is trying doors.
One day, counted
On 8 September, in a twenty-four hour window, the edge sampled 1,312 requests. Of those, 992 returned 404 — a page that does not exist.
They are not typos. They are requests for /wp-admin/, for /.env, for /actuator/, for appsettings.json — the standard list of places a badly configured server keeps its passwords. Somebody, or more accurately some large number of automated somebodies, is working through that list against every address on the internet, and this site was on it immediately.
On 4 September — the first full day this site existed at all — the pattern was starker: 3,147 of 3,479 sampled requests were 404s, and 2,987 of them came from one tool, curl, run at scale. Nothing linked to us and nobody had heard of us. It made no difference.
The ones that called themselves crawlers
Against that background, the requests carrying a search or assistant crawler's name look like the readers. On 7 September there were 84 of them. It is worth looking at what they actually asked for, because the answer is not what the names suggest.
| Claimed name | Requests | What it asked for |
|---|---|---|
| Claude-SearchBot | 38 | the sitemap (16), robots.txt (8), and fourteen different pages of the site (1 each) |
| OAI-SearchBot | 16 | /admin/_payload.json (15), robots.txt (1) |
| GPTBot | 12 | /amplify_outputs.json (12) |
| Baiduspider | 8 | /wp-json (8) |
| ClaudeBot | 8 | robots.txt (8) |
| Googlebot | 2 | robots.txt, the sitemap |
One of those six read the site. Claude-SearchBot took the sitemap, then worked through fourteen pages one at a time — which is what building an index looks like, and roughly what you would want a reader to do if the reader were a machine. Two more read robots.txt and nothing else, which is a crawler asking permission and then not taking it.
The other three never read anything. Thirty-five of those eighty-four requests asked only for /admin/_payload.json, /amplify_outputs.json and /wp-json — files that do not exist here and never have, all three of them the sort of thing that leaks a configuration if a server is careless. That is the scanning from the section above, wearing a search engine's name. Counting those rows as crawler traffic, which is what a table of names alone invites you to do, would have overstated the reading by three names out of six.
Two more that were not what they said they were
Another thirty-six requests that same day carried a well-known crawler's name and went straight to scanner paths. They claimed to be Bytespider, CCBot, Amazonbot and ChatGPT-User. A user agent is a text field that anybody can type anything into, and this is what that looks like in practice: the name of a reputable crawler on a request that is unmistakably hunting for a config file.
The other one is stranger and we like it more. Fifteen requests arrived carrying the name of a social-media link previewer — the kind of thing that fetches a page when someone pastes a link into a chat — and every one of them asked for /instance/config.py, which is where a particular Python web framework keeps its secrets. Nobody pasted that link into a chat.
What this measurement cannot tell you, which is a lot
Everything above is worth exactly as much as its caveats, so here they are rather than in a footnote.
- We cannot verify any of these names. Confirming that a request claiming to be Googlebot really is Googlebot needs a feature our plan does not include. Every count above is of requests claiming a name, which is why the two paragraphs before this one are possible at all.
- The numbers are floors, not totals. The analytics are sampled, and two of the three days quoted here hit a limit on how many groups the query returns. More happened than we counted.
- The record is destroyed nightly. This data is retained for twenty-four hours and cannot be queried over a longer range, so it exists only because something captures it each morning. The site's first day is missing entirely, because nothing was capturing yet.
- Some agent names were lost in transit into our own capture on two of these days and are recorded as unidentified rather than guessed at.
Why we bothered writing it down
Partly because it is genuinely surprising the first time: you build a thing, you put it online, and the internet's first response is several thousand automated attempts to break into it.
And partly for a reason closer to what this site is for. We spend most of our pages arguing that a number is only worth what its method is worth — that a vocabulary size depends on what you counted, that an attention score depends on a deadline somebody chose. This is the same argument, arriving from a completely different direction. "We had 3,479 visits" would be a true sentence about 4 September and a wildly misleading one, and the only thing standing between those two readings is knowing how the number was made.