AI that picks vs AI that writes: where each belongs in your system

Closed decisions (classify, route, validate) call for a model that picks and errors you can count. Free text needs human review. How to decide and test it, with Jev as the case.

In ticket triage, the model's mistake is the wrong queue, and wrong queues can be counted

Before you put AI inside a system, answer one question: is it going to pick from options you already defined, or write the answer from scratch? Sorting a ticket into one of eight queues, sending an order to the right warehouse, checking whether a VAT number matches the company name: that is picking. It fits a model that only points at an option. Summarizing a complaint, drafting the reply to a customer, explaining an invoice: that is writing, and a language model is still the right tool there.

The difference shows up the day the model gets it wrong. When the model picks, the error is the wrong queue, and wrong queues can be counted: out of 400 hand-labeled tickets, how many did it get right, and which queue does it miss most. When the model writes, the error is a plausible sentence with one bent fact in the middle, and somebody has to read it to catch it. The first kind of error becomes a number. The second becomes human review, line by line, for as long as the feature lives.

The Jev hype, and what you can actually check

A per-queue threshold test is the assertion a picking model can actually pass or fail

Over the past few days TypeSafe's Jev has shown up in almost every developer conversation about AI. The launch post climbed past 1,800 points on Hacker News, and a widely shared recap on a Brazilian developer forum listed the claims: 20 to 200 times faster, 40 to 400 times cheaper, free output tokens, "doesn't hallucinate", "doesn't generate text".

Read those numbers for what they are: vendor claims, passed along by other people. I haven't tested Jev and I can't tell you how it behaves on your data. The part worth keeping is the recap author's point. TypeSafe didn't build a GPT that runs two hundred times faster. It changed the problem the model solves. A model that only has to point at an item on a list doesn't spend time or money assembling paragraphs. The speed and the cost come from that trade, and the trade works with or without Jev.

The objections in the thread ("it's just a classifier", "structured output already does this") aren't wrong, and the recap says so. If you're the one deciding where AI goes in your product, arguing about which drawer Jev belongs in doesn't get you far. The useful question is simpler: where should the model sit in your code?

"Doesn't hallucinate" is the wrong promise

The most repeated selling point is the least useful when you have to decide. A model that picks won't invent a fourth option when you gave it three. Fine. Picking the wrong option is still a mistake, though. The billing ticket that landed in the tech support queue is just as late as it would have been if the model had written a paragraph about it.

What changes is the shape of the mistake. With a closed choice, it fits in a three-column table: input, expected option, returned option. You can measure it before launch, measure it again every time you swap models, and write a test that fails when the rate drops. With free text, every answer is unique, and "correct" depends on who reads it and what kind of day they're having.

This is close to a pet peeve of mine: tests without assertions. The test runs, goes green, and proves nothing. High coverage numbers hide that problem all the time, and AI writing free text in production with no measure of accuracy is the same hollow shell. The workflow exists, the guarantee doesn't. A closed choice is where AI will actually accept a real assertion.

Three places where the model shouldn't write a word

Shovel or mixer, the batch comes out bad if the mix ratio is wrong

Three patterns show up in nearly every business system, and in all three the right output is an option, not a sentence:

• Classifying: the ticket, email or document goes into one of N categories that already exist in your system, and a wrong category has a known cost, whether that's a slower response or an analyst redoing the work.
• Routing: the order goes to a warehouse, a carrier or an approval queue, and the list of destinations is closed because your backend only knows how to handle those.
• Validating: the submitted data either matches the rule or it doesn't, and the only possible answers are yes, no, or "send it to a person".

In all three, the set of valid answers is already in your codebase, usually as an enum or a lookup table. If the model returns a paragraph and someone parses it afterwards to find the category, you paid for writing and used picking. When the list is closed, ask for the pick and nothing else.

There's a question that comes before even that one. A lot of the decisions getting a model today were an if statement last year. In one pricing service I opened, I counted 1,043 branches. Only 34 had changed in the previous twelve months, about 3%. A stable rule the code already handles well doesn't need a model, fast or slow. The picking model earns its place when the input is too messy for if statements (customer text, a photo of a document, an email forwarded three times) and the output is still a short list.

What it costs to check each kind of mistake

Checking whether a ticket landed in the right queue takes a few seconds, and anyone on the support team can do it. Checking whether a written reply is correct, doesn't promise a date nobody agreed to and doesn't quote a policy that changed in March, takes minutes. It also takes someone who really knows the product.

Now multiply by volume. A workflow producing 2,000 replies a week, with a sample reviewed, eats hours of senior time. The same volume of classifications can be audited with a small sample and a spreadsheet. Code review tools built on AI follow the same logic: wherever the output is open-ended, a human stays at the table, and that person's time belongs in the budget from day one.

"But my use case needs text"

Sometimes it does. Customer replies, summaries of long support threads, explanations of a charge: nobody wants a category code where a sentence should be. A language model is the right call there, and pretending otherwise would be selling a cement mixer to someone who needs a paintbrush.

What you can do is split the two jobs inside the same workflow. The decision that changes system state (refund or not, escalate or not, which queue) comes out as a pick, measured and tested. The text around it gets generated afterwards, already knowing the decision, and the damage a badly worded reply can do is far smaller than a refund approved by mistake. When the decision and the wording live in the same prompt, an error in the first one hides inside the second, nicely phrased.

How to measure accuracy before you ship

The boring part is building the reference set: roughly three to five hundred real inputs, labeled by hand by people who know the business. Without it, any accuracy figure is a guess. With it, measurement becomes an ordinary test that runs in CI like any other.

Measure per category, not just overall. A model at 94% overall accuracy can be sitting at 60% on the cancellation queue, which happens to be small and expensive. The test below fails in both cases: when the output leaves the list and when any queue drops below the threshold.

from collections import Counter
from triage import classify  # your model call

QUEUES = {"billing", "tech_support", "cancellation", "other"}
THRESHOLD = 0.9

def test_accuracy_per_queue(labeled_cases):
    hits, total = Counter(), Counter()
    for case in labeled_cases:
        predicted = classify(case["text"])
        assert predicted in QUEUES, f"off the list: {predicted}"
        total[case["queue"]] += 1
        if predicted == case["queue"]:
            hits[case["queue"]] += 1
    for queue, n in total.items():
        rate = hits[queue] / n
        assert rate >= THRESHOLD, f"{queue}: {rate:.0%} over {n} cases"

Two honest limits. Reference sets go stale: a new product, a new queue or a new kind of complaint needs fresh labels, and someone has to own that. And I'm not sure the effort pays off if you get twenty tickets a day. At that volume, one person reading everything might be cheaper than any model.

Shovel or mixer

I grew up around construction sites, and there's a line I use with founders who show up excited about AI: if you don't know how to mix concrete, it doesn't matter whether you use a shovel or a mixer, the batch will be bad. A model that picks is a faster, cheaper mixer, if TypeSafe's numbers hold up. The mix ratio is still yours: the list of options, the labeled set, the threshold where the machine hands the case back to a person.

It's the first thing we work out when a project comes in with AI already inside, before anyone talks vendors. Take the AI workflow you already run in production and answer this: if it gets something wrong tomorrow, who tells you first, a test or a customer?