AI crawlers are eating my logs alive I expected the…
0 reactions
AI crawlers are eating my logs alive
I expected the bandwidth hit to be annoying, i didnt expect the dumbest part to be the logs
I run a pretty normal content-heavy app behind Node + nginx, nothing huge, and the last couple weeks the access logs turned into soup. Same few pages over and over, weird user agents, no session behavior that looks even vaguely human, just relentless fetches that make every dashboard look like we suddenly got popular for the worst possible reason. The traffic graph looked exciting for about 9 seconds, then i checked where it was coming form
What gets me is how this wrecks the boring ops stuff. Log rotation got noisy, simple grep debugging got worse because actual user requests are buried under crawler sludge, and a couple alerts fired because the request pattern changed just enough to look like something broke on our side. I can block or rate limit some of it, sure, but now im burning engineering time teaching the stack to ignore fake interest so i can go back to dealing with actual users
The web is getting polluted by clients that want the whole internet for free and leave you with the bill for bandwidth, storage, and observability, and thats the part that feels insane. If youre running anything public rn, are you filtering this junk at the edge already or just accepting that your logs are basically landfill now
#programming #dev #technology
source
I expected the bandwidth hit to be annoying, i didnt expect the dumbest part to be the logs
I run a pretty normal content-heavy app behind Node + nginx, nothing huge, and the last couple weeks the access logs turned into soup. Same few pages over and over, weird user agents, no session behavior that looks even vaguely human, just relentless fetches that make every dashboard look like we suddenly got popular for the worst possible reason. The traffic graph looked exciting for about 9 seconds, then i checked where it was coming form
What gets me is how this wrecks the boring ops stuff. Log rotation got noisy, simple grep debugging got worse because actual user requests are buried under crawler sludge, and a couple alerts fired because the request pattern changed just enough to look like something broke on our side. I can block or rate limit some of it, sure, but now im burning engineering time teaching the stack to ignore fake interest so i can go back to dealing with actual users
The web is getting polluted by clients that want the whole internet for free and leave you with the bill for bandwidth, storage, and observability, and thats the part that feels insane. If youre running anything public rn, are you filtering this junk at the edge already or just accepting that your logs are basically landfill now
#programming #dev #technology
source
| 1,000 mlx | |
| 3 | |
| 0 |