Essay · Agentic systems
Thinking in Spectrums
Why binary thinking is the wrong mental model for agentic systems
Somewhere near the start of every agentic project, a team sits in a room and decides whether it is building an agent or just calling an API. The question has the shape of diligence: the hard call a serious group settles before it writes any code. Months later the demo works and the numbers do not. The cost makes no sense, the latency makes no sense, and the reason traces straight back to that room. One coarse answer, applied to every part of a system that was never uniform enough to deserve it.
A binary closes an argument, which is the source of its appeal. You pick a side and the meeting ends. But an agentic system is not one decision. It is a pipeline of many small ones, each with its own shape, and when you resolve the whole thing down to a single either/or, you end up paying the worst-case cost of the system for the sake of its most demanding part. The decisions that matter are not binary. They sit on a spectrum, and the spectrum is not vagueness. It is resolution.
Four decisions where this plays out, and four places teams reach for a switch when what the problem wants is a dial.
01 Use case selection: autonomy is a dial, not a switch
The choice was never really agent or no agent. Autonomy runs along a continuum, and each step along it buys capability at the price of surface area you then have to control:
- a single model call
- a fixed chain of calls
- a chain with a conditional branch
- a bounded loop with a hard step cap
- a tool-using agent over a fixed toolset
- an open-ended planner that writes its own steps
Two things decide where a use case belongs on that ladder: how much of the work is the model's to figure out, and how much a mistake costs. They pull in opposite directions. A step that drafts an email can sit high on the dial, because the worst case is a re-run. A step that sends the email sits low, however capable the model, because a wrong send does not come back. Autonomy should be granted in proportion to reversibility, not to how impressive the model looks in a demo.
Most teams need something around the second rung and build something around the fifth, then spend the following quarter adding guardrails to take back the autonomy they never had a use for.
02 Model selection: rent what is generic, own what is yours
Frontier versus open weight is a procurement question, though it is almost always argued as a strategic one. The decision that actually matters sits underneath it: is this sub-task something you compete on, or is it general capability you happen to need?
Rent frontier capability for the generic reasoning you could never justify building yourself. Own the parts where you have an edge: the proprietary data, the taxonomy that is yours, the definition of correct that no vendor shares. That is where a smaller, fine-tuned model pays for itself.
Getting this backwards costs you twice over. Fine-tune a model to do general reasoning and you are paying to rebuild what you could have rented. Rent a frontier model for the one narrow classification that is your actual product and you have placed your differentiator inside someone else's system, behind an eval you do not run.
So the honest answer is not one model. It is a small portfolio, with something deciding which model handles which sub-task. That something turns out to matter more than it looks, and I will come back to it.
03 Model size: fit the stage, not the system
"We are a frontier-model shop" is a sizing decision made once and then applied to everything, including the stages that never asked for it. In a single call the waste is invisible. Spread across eight steps, it becomes the character of the whole system.
Most stages do not need a large model. Triage does not. Extraction does not. Working out which branch a request belongs to does not. These are jobs a small, fast model does well and cheaply, in a couple of hundred milliseconds, and it does them all day without complaint. Keep the large model for the stages that genuinely earn it: planning under conflicting constraints, the final synthesis where fluency and judgement actually decide the quality of the answer.
Latency is the part that gets underestimated, because it never shows up in a demo of a single step. It shows up in production, compounded. Eight sequential calls at frontier latency is a system that feels heavy to use, and the fix is rarely a faster frontier model. The fix is not running the frontier model where you did not need it.
04 Evaluation: four layers, not two
Benchmarks or vibes is the crudest of the four framings, because it offers two options where the work needs four. Component, trajectory, outcome, system - each answers a different question, and each is blind to what the others catch.
- Component. Does the unit do its job? F1 for the extractor, precision for the classifier, recall for the retriever. This is what benchmarks measure. It is necessary and nowhere near sufficient.
- Trajectory. Did the system arrive by a route you would endorse, or stumble onto a plausible answer? Two runs with identical outputs can have entirely different paths, and only one of them will survive a change in inputs.
- Outcome. Did the result meet the actual goal, judged the way a person would judge it, rather than a proxy metric standing in for the goal because the goal was awkward to measure?
- System. Does the whole thing behave over time and under load? Cost per run, tail latency, failure modes, and the slow drift as models and traffic shift underneath you.
Benchmarks live at the component layer, in the abstract. Vibes live at the outcome layer, one anecdote at a time. Between them sits almost everything that actually breaks agentic systems: the sound answer reached by unsound reasoning, the pipeline that passed every check in March and is quietly failing by June. You do not choose a layer. You run all of them, and the ones that matter most run continuously, in production, not once before launch.
These four are not separate choices lined up in a row. They move together. The higher you climb the autonomy dial in the first, the more the weight of proof shifts from component up towards trajectory and system: a single call you can judge on its output; an open-ended planner you can only trust by watching how it works. Choosing a notch is also, quietly, choosing how hard the system will be to evaluate. Better to know that before you choose than after.
The spectrum at runtime
Make these four decisions with a spectrum lens and you arrive somewhere specific: not one model doing one job, but several models, of different sizes and autonomy levels, strung along a pipeline. That produces a question the decisions themselves do not answer. When a real request arrives, what sends each stage to the right handler - this model, that one, or no model at all?
That is routing. It is the machinery that carries the spectrum decisions into runtime - the something I said earlier would decide which model handles which sub-task. The four dimensions are how you choose; routing is how the choices run, one request at a time. So it is not a new topic bolted on. It is what the spectrum becomes once it has to serve live traffic, and it is easiest to see in a single system walked end to end.
An accounts-payable agent takes a supplier invoice - a PDF landing in an inbox - and drives it toward one of three ends: pay it, hold it, or escalate to a person. Seven stages, and they could hardly be less alike.
- Classify the document. Invoice, credit note, statement, or duplicate. High volume, narrow, and specific to your suppliers' formats. A tiny model you own and fine-tune.
- Extract the fields. Supplier, purchase-order number, line items, tax, totals, dates. Also yours to own and fine-tune, because reading your suppliers' idiosyncratic layouts is exactly where generic models fumble and you have an edge.
- Validate. Does the arithmetic add up, is the tax right, are the totals within tolerance, is this a duplicate? Rules, not a model. A model here would be slower, dearer, and less reliable than a few lines of code.
- Match against the purchase order and the goods-receipt note. A lookup and a comparison. Deterministic, and again no model.
- Gate. A clean three-way match under a value limit approves itself; anything else - a mismatch, a large amount, a missing receipt - routes onward. A deterministic threshold reading the numbers and the blast radius of paying, not a model's opinion of its own confidence. This is the notch on the autonomy dial, and it lives in the architecture.
- Investigate the exceptions. The one genuinely agentic stage. For a mismatch, the agent reasons across purchase-order history, contract terms, prior invoices, and the email trail, works out what went wrong, and proposes a fix. Open-ended reasoning over messy context: rent the frontier model, and give it room.
- Post or escalate. Write the approved payment to the ERP, or compile the exception into a summary a person can act on. The posting moves money and does not come back, so it stays gated and deterministic; the summary is a mid-size model's job.
Every dimension is in there, and the deterministic-versus-agentic one most of all. Six of the seven stages are deterministic or a single bounded call - the cheap, predictable spine of the system. Exactly one is agentic, and it earns the licence only because it proposes a fix rather than paying one. Model type splits the same way: you own the reading, you rent the reasoning. Size follows - four stages run no model at all, small owned models do the high-volume reading, and the frontier model is spent only on exceptions.
Evaluation is not one test laid over the top but a different question at each layer. Component metrics watch the extractor and the classifier. Trajectory checks watch the investigator, because a right-looking resolution that quietly ignored the contract is still a failure. Outcome sampling puts the final pay-or-hold calls in front of a senior clerk. And the system layer watches for the slow disaster: a supplier changes their invoice template, extraction accuracy slips a few points, and duplicate payments climb for weeks before anyone notices.
Now set the binary beside it. Run one frontier model across all seven stages and you pay frontier rates and frontier latency on every invoice, including the great majority a fine-tuned reader and a few rules would have cleared without a language model at all. Worse than the cost, you have handed arithmetic and duplicate-checking to a model that approximates, where code was exact. The single-model version is more expensive, slower, and less trustworthy on precisely the stages where trust was cheapest to earn. That is the cost of the binary, made concrete.
The router inherits the problem
In that pipeline the frontier-or-small decision sat in a deterministic gate, and that was deliberate. The tempting alternative is to let a model make the call - a learned router that reads each request and picks a model for it. It looks cleaner. It is not. It does not remove the frontier-or-small decision; it moves it to runtime and hands it to a component that has never seen this request before, and that component is a classifier, with all the properties classifiers have.
It has a precision and a recall, whether or not you are measuring them. It drifts as traffic changes. It sits on the critical path of every request as a single point of failure. Send a genuine exception down the cheap path and it gets a confident, wrong answer; send a clean invoice to the frontier model and the cost the router existed to save comes straight back. The routing decision is an agentic decision in its own right, and it needs the same evaluation as any other stage: component, trajectory, drift. A learned router you do not measure has not solved the binary; it has only hidden it. Which is why, wherever the decision can be written as a rule, the deterministic gate is the better tool.
Resolution has a price
None of this is free. Resolution buys precision and charges complexity in return. Every additional notch, model, and eval layer is one more thing to version, monitor, and debug, and a portfolio of five models carries five sets of failure modes held together by a router that has its own.
Which is why the binary is sometimes the right answer. A small system, an early build still proving it works at all, a single clear task with no real variety to exploit: decomposing any of these is premature optimisation. One capable model, one honest eval, ship it and learn. The spectrum earns its complexity only once the pipeline is doing enough genuinely different things that a single setting starts to cost you. Reach for it then. Not before.
Where to start
"Which one, where" is the question, but it is not a licence to scatter resolution across the system on day one. That is the binary's mirror-image mistake, and it ends the same way: five models maintained to serve four users.
Start at the floor. The lowest notch on the autonomy dial, the smallest model that clears the bar, the thinnest evaluation that still tells you something true. Then climb, one rung at a time, and only when a specific failure makes you: this step keeps getting it wrong, that stage is too slow, this classifier has started to drift. Let the evidence do the promoting. The spectrum is not a mandate to build everything at once. It is permission to stop pretending one setting fits the whole pipeline, and the discipline to justify every step away from the simplest thing that works.
The reason to bother is that composition punishes coarse decisions in a particular way: the errors do not average out, they accumulate. A choice that is eighty per cent right at each of eight steps is nowhere near eighty per cent right by the end. So decompose the system, decide each part on its own terms, and put the deciding in a deterministic layer rather than the model's judgement, so that where each choice applies is enforced and not merely intended.
Binary thinking gives all of that up at the first question, and calls what it lost decisiveness.