The headline result is no longer controversial. Across task-specific fine-tuning studies — extraction, classification, summarisation, domain question answering, structured generation — the pattern repeats: a 4-to-14B open model, fine-tuned on a few thousand good examples of a bounded workflow, reaches parity with and frequently surpasses a frontier model prompted for the same workflow, at one to two orders of magnitude lower serving cost.
Distillation sharpens it further: use a strong teacher to generate the training set and the student inherits capability its base never showed.
That result is usually consumed as economics and strategy. This piece is about the part that makes it true in your deployment — because the same literature contains the failure catalogue, and the failures all share one root. The evaluation was softer than the claim.
That sentence is worth sitting with, because it is not an accusation of carelessness. Every one of the three failure modes below arises from an evaluation choice that is locally reasonable, standard in the literature, and quietly invalidating for a production decision.
Death one: distribution drift between evaluation and production
The fine-tune was measured on held-out data from the training distribution. Production drifted — new document formats, new edge cases, a policy change upstream that altered what the inputs look like.
Frontier models degrade gracefully off-distribution, because breadth is precisely what you were paying for. Narrow fine-tunes fall off a cliff — and the evaluation that certified parity never saw the cliff.
The reason it could not see the cliff is structural rather than negligent, which is why this keeps happening to careful teams. A held-out set is held out from the training distribution, so it is by construction a sample of the region where the model was taught to be good. It measures the flat part of the curve. Nothing about the standard methodology is designed to sample the region the model has never seen, because in most of machine learning that region is not the point.
For a production deployment it is exactly the point, because production supplies inputs from wherever it likes. So the evaluation set has to be drawn from production rather than from the training corpus, and it has to be refreshed, because the whole failure mode is that the distribution moves.
The fix is unglamorous: golden sets refreshed from production samples, on a schedule, with drift monitored as a first-class metric rather than discovered as an incident. And the monitoring matters as much as the refresh — a quarterly refresh tells you where you are; a drift metric tells you when the ground moved between refreshes.
Death two: the teacher grading its own student
If the model that generated the training data also judges the parity test, the evaluation measures family resemblance, not quality.
The student is rewarded for inheriting the teacher's biases, including its errors. This is a particularly nasty failure because it is self-reinforcing: the closer the student imitates the teacher — including where the teacher is wrong — the better it scores, so the evaluation actively selects for inherited error. And the literature on LLM-judge bias is unambiguous that judges prefer outputs shaped like their own.
Distillation makes this trap easier to fall into, because the teacher is right there and is the strongest model in the room. Using it as judge is the obvious, convenient, apparently rigorous choice.
The minimum standard has three clauses and all three matter. The parity judge is a different strong model from the teacher, or a human panel. The comparison is blind, because knowing which output came from the small model measurably changes both human and model judges' scores. And the comparison is pairwise rather than scored in isolation, because absolute quality scores are far noisier than relative preferences.
Of the three, blinding is the one most often skipped and the cheapest to add — it is a labelling change in the harness, not a methodological rebuild. It is also the one whose absence is hardest to detect afterwards, since a non-blind evaluation produces perfectly plausible numbers.
Death three: aggregate parity hiding segment failure
Matches the frontier baseline overall routinely decomposes into wins on the common cases, loses badly on the rare, expensive ones.
The arithmetic makes this inevitable rather than unlucky. An aggregate is dominated by the common segment because it is common, so a volume-weighted average will always flatter a model that wins the easy majority. A model can lose catastrophically on three percent of cases and still report parity.
In regulated workflows the rare cases are frequently the ones that end up in front of a reviewer — the unusual claim, the edge-case entitlement, the transaction that tripped a threshold. The distribution of cases that matter is close to the inverse of the distribution of cases that occur, which is precisely the property an aggregate destroys.
So report parity by segment, with the segments defined by consequence, not by frequency. That last clause is the operative one and the one teams get wrong: segmenting by document type or by channel produces segments that are still frequency-shaped. Segmenting by what happens if this is wrong produces the segments the safety case actually needs, and usually produces a much smaller number of them.
This also determines how the golden set is sized. A production sample drawn proportionally will contain almost none of the rare segment, which means the evaluation has no statistical power exactly where the risk is. Rare segments have to be over-sampled deliberately, and the reporting has to keep them separate rather than re-weighting them back into an average.
The honest protocol
Six steps, assembled from what survives across the literature.
Freeze a golden set sampled from production, sized for the rare segments rather than proportionally, refreshed quarterly. Fix the incumbent baseline at its production configuration — not at a benchmark configuration, because what you are replacing is what you actually run, and the two are frequently different in reasoning effort, prompt and context.
Judge blind, with a non-teacher judge, pairwise. Set the promotion bar before running, because parity is a decision threshold and not a vibe — deciding what counts as good enough after seeing the number is not deciding. Report by segment.
And re-run the identical harness on every model change on either side. That last clause is the one that converts an evaluation into a control, and it is where the compounding value lives: at that point you have also solved silent version drift, since the harness that certified the small model now guards it. When the incumbent frontier model updates underneath you, the harness tells you what moved. One artefact, three governance problems.
A note on cost, since this is the objection that stops adoption. The expensive part is the golden set, not the running — and specifically the labelling of the rare segment, which cannot be sampled cheaply because it is rare. Budget for that once, honestly, and the marginal cost of every subsequent evaluation is small. Teams that try to make the golden set cheap by sampling proportionally end up with a harness that cannot see the thing it was built to see.
When small honestly loses
It does, and a piece advocating measurement should say where.
Open-ended reasoning across unpredictable domains, where the input surface is genuinely unbounded and breadth is the requirement rather than a nice-to-have. Workflows whose specification changes weekly, where retuning outpaces its own payback — the fine-tune is a snapshot, and if the target moves faster than the tuning cycle you are permanently chasing. And long-tail input surfaces, which is the same argument as the cliff, stated as a design constraint rather than as a measurement failure.
There is a fourth that gets less attention: workflows where you cannot assemble a good training set because good examples do not exist in volume. Distillation helps, but a teacher generating examples of a task nobody has done well is generating plausible fiction, and the student will learn it faithfully.
Small models are a class in the portfolio, not the portfolio. The point of honest measurement is precisely that it tells you which workflows belong in the class — instead of the current industry method, which is deciding first and benchmarking after.
The fourth death nobody names: the baseline was never fixed
Three failure modes are well described in the literature. A fourth shows up constantly in practice and is almost never written down, because it is a process failure rather than a statistical one.
The parity comparison is against an incumbent, and the incumbent moves. Between the evaluation that certified the small model and the decision that deployed it, the frontier baseline shipped an update — or the team quietly changed the prompt, or the reasoning-effort setting, or the retrieval configuration feeding it. The comparison that justified the swap was against a system that no longer exists.
This cuts both ways, which is why it is easy to miss. Sometimes the incumbent improved and the parity claim is now false; sometimes it regressed and the small model looks better than it is. Either way the number on the slide is describing a comparison nobody could reproduce.
The fix is the fix-the-baseline step in the protocol, and it needs to be more literal than it sounds: record the incumbent's model version, prompt, effort setting and retrieval configuration as part of the evaluation artefact, and re-run the whole comparison when any of them changes. A baseline described as the frontier model is not a baseline. A baseline described as this model version at this configuration on this date is.
This is also the clause that connects the harness to model-risk practice, because it is the same discipline as version pinning: what makes a validation meaningful is that both sides of the comparison are pinned, and what makes it a control rather than a one-off is that the pin is checked.
Three markets, three different reasons to build it
The harness is the same everywhere. What motivates it differs enough to change how it gets funded.
In North America the driver is supervisory: validation and ongoing monitoring are established expectations for models in consequential use, and a parity harness is the most direct way to satisfy them for a model swap. The argument that funds it is not cost saving — it is that the swap is not approvable without it.
In India the driver is the economics, and they are compelling enough that the swap will happen regardless. The harness is what makes it safe, and the specific risk in a fast, price-pressured market is that the golden set gets sampled proportionally to save labelling effort — which produces a harness that cannot see the rare-segment failure it exists to catch. Spend the money on the rare segment or do not build it.
In the Gulf, sovereignty is frequently the motivation for moving to open-weight models that can be hosted locally, which means the decision to swap is often made on grounds other than capability. That makes honest measurement more important rather than less: when the reason to move is strategic, the evaluation is the only thing standing between a sovereignty decision and a quality regression nobody planned for.
The strongest objection: this is too much process for a model swap
The counter-argument is real and comes from teams shipping successfully.
It runs: the small-model result replicates so consistently that demanding a six-step harness before capturing it is gatekeeping. Teams that just swap the model and watch their metrics do fine — production monitoring catches regressions, the cost saving is immediate and large, and the harness is a quarter of engineering effort spent proving something the literature already established. Ship it, watch it, roll back if it degrades.
Watch-and-roll-back is a legitimate strategy and I would not argue against it universally. It works when three conditions hold: the failure is visible in production metrics, it is visible quickly, and the damage during the detection window is acceptable.
The three deaths above are precisely the cases where those conditions fail. Drift-cliff failures appear when the distribution moves, which may be months after deployment and will not correlate with the change. Teacher-bias failures are invisible in production metrics entirely, because the output looks like what the teacher would have produced and nothing flags it. And segment failures are, by construction, rare — so they are slow to accumulate signal and the damage per instance is high, which is the worst combination for detect-and-correct.
So the honest scoping is the same one that runs through this whole series: match the discipline to the consequence. A summarisation workflow feeding a human reader can be swapped and watched. A workflow whose output reaches a customer or a record should have the harness, because the failure modes it catches are specifically the ones production monitoring does not.
Where this argument is weakest
Two admissions.
First, I have described the parity literature as consistent, and it is — but it is also heavily weighted toward tasks that are easy to evaluate automatically, which is a selection effect that probably overstates how well the result generalises to workflows with fuzzy success criteria. The tasks where parity is easiest to demonstrate are not a random sample of the tasks enterprises want to automate.
Second, the protocol is more demanding than most organisations will adopt, and a protocol nobody runs is worth less than a lighter one they do. If only one step survives contact with your organisation, make it the blind non-teacher judge — it is the cheapest, and it is the one whose absence most reliably produces a confident wrong answer.
The deeper point
In production AI, the scarce capability is rarely the model. It is the harness around it.
A team with a disciplined parity harness can safely capture the small-model dividend, the reasoning-budget dividend, and every future dividend the model market ships — because it can measure its way through each transition. A team without one is betting the workflow on a vendor's benchmark slide, every time, forever.
Build the harness once. Collect the dividends indefinitely. Measured honestly, the small-model result is the best deal in applied AI. Measured lazily, it is next quarter's quiet incident.