The busiest visitor to my website was bots pretending to…
0 reactions
The busiest visitor to my website was bots pretending to be browsers
\*\*Source:\*\* my own website's logs, last 30 days (to 3 October 2026). Every request is classified as it arrives: known bots by user agent, with search and AI crawlers verified by reverse DNS or their published IP ranges. People are counted by a small page-view beacon. No IP addresses are stored.
\*\*Tools:\*\* the site itself (Node.js and SQLite) collects the data. The chart is HTML and CSS, rendered to an image with headless Chromium.
\*\*What the chart shows:\*\*
\- 68 different kinds of bot visited. Together they made 2,376 requests, 42% of all visits. People made 3,285 page views.
\- The biggest single visitor (1,049 requests) used a normal browser's user agent but behaved like a script: missing browser headers, no JavaScript, or far too many requests too fast.
\- Those masked visitors made 92 of the 108 requests that robots.txt forbids, and 30 of the 32 visits to a trap link that people can't see.
\- Only 26 of the 68 kinds of bot ever read robots.txt.
\*\*Caveat:\*\* a few masked visits could be people with privacy tools that block JavaScript. The ones that followed the invisible trap link were not.
The live version: https://crawlerzoo.com/leaderboard
Caveat:
The 3,285 human page views include my own browsing. The site is very new I have had to visit a lot to check changes and test. just fyi
#education
source
\*\*Source:\*\* my own website's logs, last 30 days (to 3 October 2026). Every request is classified as it arrives: known bots by user agent, with search and AI crawlers verified by reverse DNS or their published IP ranges. People are counted by a small page-view beacon. No IP addresses are stored.
\*\*Tools:\*\* the site itself (Node.js and SQLite) collects the data. The chart is HTML and CSS, rendered to an image with headless Chromium.
\*\*What the chart shows:\*\*
\- 68 different kinds of bot visited. Together they made 2,376 requests, 42% of all visits. People made 3,285 page views.
\- The biggest single visitor (1,049 requests) used a normal browser's user agent but behaved like a script: missing browser headers, no JavaScript, or far too many requests too fast.
\- Those masked visitors made 92 of the 108 requests that robots.txt forbids, and 30 of the 32 visits to a trap link that people can't see.
\- Only 26 of the 68 kinds of bot ever read robots.txt.
\*\*Caveat:\*\* a few masked visits could be people with privacy tools that block JavaScript. The ones that followed the invisible trap link were not.
The live version: https://crawlerzoo.com/leaderboard
Caveat:
The 3,285 human page views include my own browsing. The site is very new I have had to visit a lot to check changes and test. just fyi
#education
source
| 3,000 mlx | |
| 4 | |
| 0 |