Scraper traffic is inflating your cloud bill: how to measure it

At git.kernel.org, 14 CPU cores do nothing but render pages for scrapers, more than all legitimate access combined. How to split bots from humans in your logs first.

The cores are busy around the clock. The question is who is on the other end.

Before you approve one more instance, run the boring calculation. Take seven days of access logs, group them by user agent and by IP range, then add up CPU time and response time inside each group. On any system that serves publicly generated pages, the biggest slice of that time often belongs to reading robots rather than to customers.

The clearest example this week came from the official Linux kernel server. Konstantin Ryabitsev, who runs kernel.org infrastructure, reported that git.kernel.org burns more CPU cycles rendering commits as HTML for scrapers than it spends on every other kind of legitimate access combined, git clones included. Across five geo-distributed nodes, at any given moment, fourteen cores are doing nothing else. Simon Willison published the account on his site a couple of days ago and the thread landed on Hacker News.

Fourteen cores. No human on the other end.

The most expensive page on your site is the one nobody opens twice

Forty minutes of access logs answer what the budget meeting has been arguing about for three months.

A git web frontend works like this: every commit becomes a page rendered on demand, with the diff, the list of changed files and links into the tree at that point in history. Multiply commits by files by available views, diff, blame, raw, tree, and the address space grows by multiplication. In a repository the size of the kernel, nobody can even estimate how many pages exist.

A person opens one of those at a time, usually the same handful. A crawler opens all of them, once each. That pattern defeats caching by design, because for every address the crawler request is both the first and the last. You pay full generation cost and reuse nothing.

This is not a git problem. Willison wrote in the same piece that he worries about it because of Datasette, which serves a large number of crawlable pages by nature. The shape shows up in ordinary business software too: a catalogue with combinable filters in the URL, a calendar with one page per day, search with an indexable query parameter, a report that accepts a date range. Every time a filter combination became a public address, you opened a surface that multiplies itself while nobody watches.

"We have a robots.txt"

That is the first answer in the room, and it does not hold. robots.txt is a polite request. Whoever respected it already did. And a good share of large scale collection traffic no longer identifies itself: it arrives with an ordinary browser user agent, from residential IP ranges that rotate every handful of requests. There is no name left to block.

One detail changes the conversation: refusing costs money too. By the time your rule decides to say no, the TLS handshake happened, routing happened, some layer already parsed the headers. What changes is the order of magnitude, and the gap is huge, because returning a 403 is far cheaper than rendering a diff. That is exactly why blocking is worth it. But a rule written at the wrong layer, inside the application, after the database connection opened, gives you almost nothing back. The expensive work already ran before anybody said no.

Three measurements before you buy anything

The invoice arrives broken down by service, never by cause.

Measure first. Three numbers end the argument, and none of them requires a new tool or an observability project.

1. Share of requests and share of total CPU time by client type over the last seven days, split into browsers, self identifying bots, and traffic with no clear identity. If the third bucket is the largest, you already know where the money goes.
2. The twenty most expensive routes by total time, not by hit count. A route hit 300 times a day at 900 ms costs you more than one hit 90,000 times at 4 ms. Write the cache hit rate next to each of them.
3. How many distinct addresses your system is capable of generating. If nobody on the team can answer that out loud in a meeting, that is the answer, and it explains a good part of what you pay every month.

None of this needs budget approval. It needs someone to sit with the logs and accept that the number may be embarrassing.

What fits in one week of work

With those three numbers in hand, the decisions get short. First, serve a cheap version to whoever is not a person: same information, no heavy render, none of the queries that assemble the full view. Second, cache with the right key and a generous TTL on pages born from parameter combinations, where content changes little and generation is expensive. Third, take the combinatorics out of the index, with canonical pointing at the unfiltered version and the expensive view behind authentication when it only matters to someone already logged in. Fourth, rate limit by range and by autonomous system at the edge, before the application, where saying no is nearly free.

Now the honest limit. If your site is a brochure with thirty static pages behind a CDN, none of this is your problem and your bill went up for another reason. If your application sits entirely behind a login, same thing. What is written here applies to anyone publishing a lot of generated, crawlable pages, which happens to be the architecture of almost every product that decided it wanted to be found on Google.

The invoice is broken down by service, never by cause

The cloud bill arrives organised by resource: compute, database, egress, storage. No line says why it went up. So the meeting reaches for the most recent available explanation, which this year tends to be the AI feature that shipped last quarter. Two instances get approved, the curve behaves for six weeks or so, then climbs again. Next quarter the conversation restarts from the same place, with the same culprit and one more server on the invoice.

What prevents that meeting is instrumentation, and that is where most teams find out they have none. A health endpoint returning 200 and a colourful dashboard fed by empty data give the feeling of monitoring without delivering any. When a system lands here for rescue work, the same set shows up together almost every time: slow chained queries, logs that cannot tell who called what, and test coverage with no assertions. In those cases the first week is diagnosis rather than repair/diagnostic-sprint>), because touching production before measuring is how you replace a perfectly good valve while the leak stays where it was.

One thing this measurement does not settle. It will not tell you whether the crawler eating your CPU feeds a search engine that brings you customers, and that is a business question rather than an infrastructure one. Some robots you want. Some copy your entire catalogue every night and return nothing. The log separates them by behaviour, not by stated intent, and you are the one who decides which of the two deserves fourteen cores.

The kernel.org number is public. Yours has never been measured

The git.kernel.org story is useful because it comes from a place where infrastructure is taken seriously, run by competent people on a tight budget. If it happens there, the odds it is happening in your system, unnoticed, are high.

Open yesterday's log and sum CPU time by user agent. If the biggest slice belongs to traffic that will never buy anything from you, the instance you approve tomorrow will serve robots at slightly better latency, and the invoice line will climb again in a couple of months. Next time someone in your company says the cloud bill went up because of AI, ask one question: what percentage of last week's CPU time went to requests that did not come from a browser? If nobody knows, the server conversation can wait another forty minutes. As for how we compare the paths before recommending one/compare>) and what has already been rescued this way/cases>), the rest of the site tells it better than this paragraph.