High test coverage is where production bugs hide best

A founder sent me a 92% coverage badge the same night his checkout was down. Coverage counts lines that ran, not results anyone checked, and it is the easiest number for a vendor to inflate. What a senior squad reads before it trusts the green.

A founder in fintech texted me a screenshot at 11pm on a Thursday. It was his vendor's test report: 92% coverage, all green, the kind of badge you are proud to pin to the README. In the same screenshot, open in another tab, his product's checkout screen frozen. He is not a Revin client. He had come to me through a referral, wanting a second opinion before he renewed the contract with the shop that built the thing.

His question was honest and uncomfortable: how does 92% coverage let the checkout go down? The answer fits in one line, and it is behind half the surprises I see in production. Test coverage does not measure whether your software works. It measures how many lines ran while the tests executed. Those are different things, and the gap between them is where the bug hides.

A test can walk through this entire component, log the value to the console, and never once check whether it is right.

What test coverage actually counts

Coverage is an execution metric. The tool instruments your code, runs your suite, and flags every line, branch, and function touched at least once. Then it divides what was touched by the total and hands you a percentage. It is genuinely good for one thing: finding code no test comes near. A block sitting at 0% is a real warning.

The problem is what it ignores. The metric records that the line ran. Whether anyone checked the result of that line, it has no idea. A test that calls calculateInterest(balance) and never inspects the return marks the function covered all the same. The function ran. The value it produced went unreviewed. On the dashboard, that green looks identical to a test that checks the number to the cent.

That is roughly what we found in the fintech suite. Plenty of tests exercised the code and stopped there, never asking whether the result matched what it should be.

How does a test run, pass, and verify nothing?

There are three common ways for a test to go hollow. None of them shows up in the coverage percentage:

• Assertion-free tests: they call the function and use no expect at all. If nothing throws, they pass. The line is covered; the behavior is not.
• Over-mocking: the test stubs so many dependencies that it ends up checking its own mock. You program the answer and, three lines later, assert that the answer is the one you programmed.
• Snapshots on autopilot: the test compares output against a saved snapshot. When it breaks, plenty of people just run the update, the snapshot becomes the new truth, and the test goes green again with nobody reading the diff.

All three inflate coverage and hold nothing up. Because the number climbs, everyone feels safer. The vendor included, now quoting that 92% as if a percentage were the same thing as quality.

Picking up a test and asking 'what happens if I break this line on purpose?' tells you more than any percentage in the report.

The number the body shop loves to put on a slide

Coverage turned into a sales metric. It climbs easily, because writing an assertion-free test is fast. It fits in a round number. And it impresses anyone who does not read code. For a shop that bills by delivery, it is the perfect indicator: it grows almost on its own, and the client has no way to challenge it from the outside.

The incentive is crooked. When the contract pays for a feature shipped with tests, the vendor optimizes to hit the percentage. Preventing the production incident is separate work, duller and pricier, that nobody is paying for out loud. It is not that they are lying. They are measuring the easy thing and charging for it.

On a senior squad, a test is reviewed like production code. A pull request that adds an assertion-free test does not pass review: it comes back with a single question, what is this guaranteeing? That is how we look after client suites at Revin, and it is what separates coverage that protects you from coverage that just decorates you. If nobody has ever reviewed your tests with that kind of rigor, start with an honest audit of what is there today.

🧬 Mutation testing: who tests your tests?

There is an old, underused technique aimed squarely at this blind spot: mutation testing. The idea is almost cruel. A tool takes your code and plants bugs on purpose: it flips a > into >=, negates a condition, deletes a line, or turns a + into a -. Each broken version is a mutant, and your whole suite runs against every one of them.

The read is blunt. If the code was broken and no test complained, that test was protecting nothing: the mutant survived. What matters stops being how many lines you cover and becomes how many of those planted bugs your suite actually kills. Tools like Stryker in the JavaScript and TypeScript world, and PIT in Java, do exactly this and give you back a mutation score.

We ran Stryker on that 92% suite. The mutation score came back at 41%. Put plainly: more than half the bugs we planted on purpose sailed straight past the green tests. That number moves with the tool and with what you ask it to mutate, I will grant that. I do not treat 41% as a law of physics. But even shaving off a generous margin, the conclusion holds: the comfortable suite covered plenty and, when it came to catching a real bug, let it through. That is the kind of read we do in the first days with a client, and a few of them become a published case later.

The green that counts is the one you can break

If you are a founder, do not ask for the coverage percentage. It will come back high and tell you nothing. Ask your team or your vendor something else: what happens to the suite if someone breaks a business rule on purpose? If the answer is silence, or a 'we trust the ninety-something percent,' you already have your diagnosis.

I will be fair. Not every project needs this. A five-page marketing site is never going to set up mutation testing, and that is perfectly fine: high coverage already does the job there, and hiring a senior squad would be a waste of your money. But anything that touches money, sensitive data, or a flow that takes the whole operation down when it fails, the pretty number on its own is theater.

The fintech founder did not switch vendors out of anger. He switched because, for the first time, someone sat down with him and showed him in plain terms how much that comfortable suite was really defending his product. It was very little. If you suspect your own coverage is closer to decoration, the first move is to look hard at what your suite actually guarantees today, and that audit fits inside one conversation.