How to decide your LLM route with real traffic evidence

The company picked the best model on Tuesday. On Thursday the price changed, the provider degraded p95 latency, and nobody could say whether the route was still the right one, because the original decision never had a number to defend it. Choosing an LLM route by lab intuition is easy. Changing a route with evidence is what almost nobody organizes.

The model price table moved predictably over the past year: Anthropic released Opus 5, OpenAI published GPT-6, and each release repositioned the relationship between cost per token and perceived quality. A routing decision made in March can be wrong in September without a single chart being looked at.

This guide does not discuss which model is better. It describes how a company decides its own route with evidence: duplicating a slice of live traffic, judging answers blind, and promoting the change only when measurement on the company's own traffic authorizes it. The method fits in five steps and fits the infrastructure the company already has.

What is shadow evaluation and why does it decide better than a benchmark

Shadow evaluation is running the candidate route in parallel with the incumbent route, over real traffic, without it answering the user. The team duplicates a sample of the calls, sends them down both paths, stores both answers and judges the difference blind, before any promotion.

The advantage over a public benchmark is simple to state and hard to argue with: the benchmark measures the model, while the shadow measures the model on your traffic, with your prompts, your task distribution and your tails. A model can win in aggregate and still lose exactly on the case that appears most in your base. The difference between the two measurements usually decides the bill at the end of the month.

A benchmark is a photo. A shadow is a video.

The routing decision is, at bottom, an accounting decision dressed as engineering. And accounting is not decided by vendor reputation, it is decided by your own number.

The error the shadow corrects is static evaluation. Most companies test a new model on a few dozen hand-built prompts, like the result and promote the route. That measures the impression of whoever wrote the prompts, not performance in production. The shadow trades the showcase prompt for the call the user actually made.

Prerequisites before the first step

Before duplicating anything, three pieces have to exist, and without them the shadow becomes cost with no reading: every call carries a route, tenant and task identifier, a store keeps each route's answer under the same identifier, and a strong judge evaluates the pairs against a written rubric. Missing any one of them, the experiment does not start.

Without a trace, there is no verdict.

• Identifiable traffic. Every call carries a route, tenant and task identifier, so the sample can be segmented later.
• Answer logging. A store that keeps each route's answer under the same call identifier.
• A trustworthy judge. A model strong enough to evaluate, isolated from the route under test, plus a rubric of criteria.

If the task identifier is missing, the experiment measures an average that guides no one. An aggregate quality gain hides a regression in an entire class of calls, and it is the regression that hurts in production.

What does it cost to run both routes at the same time?

The cost of the shadow phase is the cost of the candidate route over the duplicated slice, plus the cost of the judge over the pairs. It is not double the invoice, and the exact number depends on how much traffic you mirror.

The temptation is to mirror everything. Mirroring 100% of traffic doubles inference spend and turns a routing decision into a budgeting project. The recommended practice is to mirror a fraction that produces enough sample for the decision, and leave the rest alone.

Doubling a sample is not the only way to control the bill. Complexity-based routing, which sends each call to the cheapest model capable of solving it, cuts the cost of the candidate route before the shadow even starts.

The comparison below summarizes what the team gains in each evaluation approach. The two columns measure different things, and that is why they complement each other instead of competing.

Step 1: define the hypothesis and the promotion criterion before measuring

The first step is not technical, it is method. Before turning on the shadow, write the hypothesis and the criterion that authorizes promotion, because without that pre-registration the experiment ends in argument and any result starts to serve any position. A proper endpoint evaluation and routing policy starts from exactly that discipline.

A hypothesis without a criterion is a guess with a stamp.

The hypothesis has to name what changes and what is expected. "The candidate route reduces cost per task by 20% without lowering the judge's approval rate by more than two percentage points" is a hypothesis. "Testing the new model" is not.

The promotion criterion is the line that separates approve from do not approve. It is written now, not after seeing the number, because a criterion written with the result on the table is a justification, not a criterion.

Two parameters deserve explicit pre-registration:

1. Minimum sample size. How many paired calls before any reading, so the result is not one afternoon's noise.
2. Cost ceiling. The value per thousand calls above which the candidate route fails even with equal quality.

The practical floor sits on the order of a few dozen pairs per cycle. Below that, the cost of building the judge does not pay off and direct human review is more efficient than the automatic scoreboard.

Novelty bias is the silent enemy of this step. Teams tend to prefer the answer from the newer, better-known model when the judgment is not blind, and the shadow exists precisely to remove that shortcut.

Step 2: duplicate a representative slice of live traffic

The second step is turning on the shadow over a sample that represents the real distribution of calls, and the word that carries the step is "representative", because a biased sample produces a confident and wrong decision. Duplicating only the most common call measures the easy task and leaves out the tail, where the surprises live.

The naive path is to duplicate everything. The correct path is to duplicate by segment. Separate the calls by task class, by relevant tenant and by context band, and mirror inside each segment. If you mirror only the most common path, the result does not hold for the tail, and the tail is where the surprises live.

The choice of splitter also matters. The splitter sits at the gateway layer, at the point where the call has already been normalized and has not yet been dispatched. Putting duplication inside the application forces every service to know about the experiment, and the experiment dies in the first refactor.

There is an isolation requirement: the candidate route cannot answer the user. If it answers, you are running a canary, not a shadow, and you have taken on the risk of exposing someone to a route that has not been approved.

It is worth measuring LLM provider performance on your own load before trusting any ranking. A provider's p95 latency on your distribution can be far worse than the average it publishes, and the shadow is the instrument that reveals that difference.

Step 3: judge the answers blind with an LLM judge

The third step is turning answer pairs into a verdict. An LLM judge evaluates the two answers without knowing which route produced them, following a written rubric, and the secrecy of the labels is what gives the judgment value.

The judge must be strong enough to evaluate and independent of the two routes under test. If the judge is the same model as one of the routes, it carries that route's bias and the experiment measures the judge's preference for itself.

The rubric has to be specific. "Quality" is not a criterion. A useful rubric separates at least three dimensions:

• Factual correctness, when the task has a verifiable answer.
• Format adherence, when the consumer is a parser and not a human.
• Usefulness for the task, evaluated against the call's declared objective.
• Verbosity control, so the judge does not prefer the longer answer by default and reward padding.

A judge without length control tends to crown the longest answer, which favors precisely the new and more verbose route. Either length is controlled in the pair, or the rubric penalizes padding. One of the two, written before measuring.

The judge also has cost and error. A badly calibrated judge approves almost everything or fails almost everything, and the team reads that as a result of the route. It is worth measuring agreement between the judge and a human-judged sample before trusting the automatic scoreboard.

Blind judgment solves the problem that review by personal taste does not. There is no way for the engineer to know which answer came from the new route when the order of the answers is randomized, and it is that ignorance that produces an honest scoreboard.

Step 4: apply the statistical criterion and read the result by segment

The fourth step is looking at the number. Here the decision stops being opinion and becomes a reading of a result measured against the criterion pre-registered in Step 1, segment by segment, because the aggregate average only hides the loss that matters.

The reading starts at the aggregate and does not end there. An average quality gain can hide a severe loss in a small and critical segment. Read the result by task class, by context band and by tenant, and treat the average as a summary, never as a conclusion.

A small difference inside the noise does not authorize promotion. If the confidence interval of the paired gain by segment crosses zero, the honest thing is to say the experiment found no difference, not to pick the side that pleases. A team that promotes based on noise will promote and revert in a cycle, and each reversion costs more than the experiment.

Three readings condemn the candidate route, even when the aggregate pleases:

1. Cost per task blew through the pre-registered ceiling.
2. A relevant segment regressed beyond the tolerated margin.
3. Tail latency worsened in a way the user will feel.

The counterintuitive point is that the statistical criterion does not exist to prove the new route is better. It exists to prevent promotion of the new route when the evidence is weak. The burden of proof stays with whoever wants to change.

Step 5: promote the route and keep reading after promotion

The fifth step is promoting, and promotion is a configuration change, not a project. The candidate route becomes the incumbent because measurement on real traffic enabled it, and the old route remains available as a fallback, so the change stays a reversible decision and not a long-term commitment.

Promotion has to be reversible. Promoting without a way back turns a bet into a commitment. The previous route stays in the configured fallback, and the reversion trigger is written with the same care as the promotion criterion.

After promotion, measurement does not stop. The traffic distribution changes, the model price changes and provider quality changes. A route promoted in September deserves a new reading in November, because evidence has an expiration date.

The promotion gate fits in a simple reading. The new route is promoted when judged quality is equal to or better than the incumbent, cost per task stays under the ceiling, and no critical segment regressed beyond the margin. Outside that, the incumbent stays and the cycle restarts with a new hypothesis.

How do you verify the shadow is working?

Verification is direct and does not depend on faith. The pipeline is correct when the answer pairs reach the store with the same call identifier, when the judge receives both answers without the route labels, and when the scoreboard by segment matches the mirrored sample.

A healthy shadow leaves three traces.

The sign that the infrastructure is wrong appears fast. If the paired-answer rate is low, the splitter lost calls. If the judge agrees with almost everything, the rubric is loose. If the cost of the shadow phase touches the cost of production, the sample is too large for what you intend to decide.

The LLM call trace is the instrument that closes verification. Every duplicated call has to be auditable from the prompt to the judge's verdict, with the route identifier preserved across the whole chain, because without that complete chain the judge's scoreboard becomes a black box that nobody can contest afterwards.

Common mistakes and how to fix them

The mistakes below show up repeatedly in real deployments, and each one has a specific fix that no amount of traffic solves. The common pattern among them is that the routing decision was made without a number of its own to defend it, and there is no volume of traffic that fixes the absence of a criterion written before measurement.

A route error is almost never a model error.

Judgment without blindness. When the team knows which answer came from the new route, preference for the trendy model contaminates the scoreboard. The fix is to randomize the order of the answers and strip the labels before sending them to the judge.

Biased sample. Mirroring only the highest-volume calls measures the easy task and ignores the tail. The fix is to segment duplication by task class and by context band.

Weak or single judge. A judge from the same provider as the candidate route carries built-in bias. The fix is to choose a strong judge and, when possible, judge with two judges and measure the disagreement.

Criterion written after the result. Promoting based on a criterion born after seeing the number is justifying, not deciding. The fix is the pre-registration of Step 1 and the discipline of not moving it.

Promotion without observability. A route promoted without continuous reading becomes silent debt. The fix is to keep segment measurement after promotion, with the same panel that decided the change.

Frequently asked questions about deciding the route with real traffic

Does shadow evaluation replace the public benchmark? No, it complements it, and the division of labor between the two is precisely the point. The public benchmark filters candidates fast and discards what does not even deserve an expensive test, while the shadow decides on your traffic; the efficient sequence is to use the benchmark to choose what is worth testing and the shadow to authorize promotion.

How much traffic needs to be duplicated? Enough that the reading by segment stops depending on a handful of calls, because a small but well-segmented sample is worth more than mirroring everything with no criterion. The cost of the shadow grows directly with the mirrored fraction, so the decision is always a balance between statistical confidence and the bill at the end of the month.

Is an LLM judge reliable enough to decide a route? Reliable enough when the rubric is specific, the judgment is blind and agreement with human judgment is measured on a sample before the scoreboard becomes a decision. A judge with no written rubric produces a scoreboard nobody can audit afterwards, and the route ends up promoted by a number that does not survive a meeting.

When do you revert a promoted route? When continuous reading shows regression beyond the margin that authorized the original promotion. The reversion trigger is written together with the promotion criterion, never after the problem appears, because reverting in the middle of a crisis forces the team to improvise exactly when there is no calm left to decide.

Does the shadow work for cost routing or only for quality? It works for both, as long as the criterion combines the two dimensions in the same gate and does not treat cost as a tiebreaker. A cheaper route that worsens quality in a critical segment fails the same test as an expensive route that improves nothing.

References and Further Reading

• How an endpoint evaluation changes a company's routing policy
• Complexity-based routing: quality without waste
• How to measure LLM provider performance
• LLM call trace: what it is and why to audit it
• Google SRE Book, the chapter on gradual release and canaries: sre.google/sre-book/release-engineering
• NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0): nist.gov/itl/ai-risk-management-framework

The next step is instrumenting the decision, not choosing a model

The right route today is not the right route in three months. What survives the change in price, latency and release is the method that turns a routing decision into an experiment with a criterion written before measurement, because choosing the best model is easy and instrumenting the change separates whoever decides from whoever bets.

This is where the Nexforce Router comes in as infrastructure, not as a ready answer. A single API, one key, 300+ models, with intelligent routing, automatic failover and configurable fallback for when the incumbent route goes down. The trace of every call sustains the shadow with the route identifier preserved. The budget per key, per agent or per project keeps the mirroring phase from blowing the ceiling without warning. And the savings of up to 50% in cost per token reported by Nexforce itself, with an invoice in BRL, changes the math the candidate route has to beat.

The method is yours. The layer that runs it can be a single one.