Does the AI agent benchmark show that the harness changes the cost?

When the model is the same and the result changes, the variable is outside the model

A benchmark published by Ora in August 2026 ran the same task, with the same models, in two agent harnesses and measured differences of 7% in steps, 2x in native success and 9% in valid endpoints. Then, fixing a prompt cache defect cut total cost by about 15%. The AI agent benchmark that Ora operates exists to answer exactly this question: how much of your agent's cost and success depends on the environment that runs the model, not on the model itself. The bill changes. For companies that pay per inference, the answer changes what is worth measuring.

Context: Ora measures whether the web is ready for agents

Ora is a platform that runs agents against real customer websites with the task of signing up for a product, integrating with it and paying for it, and it records the cost, the latency and the number of steps of each journey. The company's own estimate is that 99% of the web is not ready for this kind of agent, a number that Ora states, not an independent measurement.

For the company buying inference, the effect is direct. Two teams can choose the same model, from the same provider, with the same published token price, and end the month with very different costs per task, because the execution around the model is not the same. The invoice does not break that difference down: it shows tokens consumed and nothing else. It is the same pattern already discussed when the real cost of an AI application goes beyond the token price, and the Ora case gives it attributable numbers.

The detail that matters to the B2B buyer is the test design. An agent has two parts: the model, which reasons, and the harness, the software that provides tools to the model and guides it from step to step. When an agent fails in a signup flow, the difference may lie in the model, in the harness or in the site, and without instrumentation the result is a grade with no attributed cause. That is why Ora runs a separate runtime for each harness and records the trace of every step. Without a trace, there is no cause.

How Ora's AI agent benchmark works

The documented case compared two harnesses across hundreds of real journeys, in multiple domains, with an identical task for both: integrate with a product. Both sides ran the same models. This harness benchmark keeps the model fixed, and that is what gives the test its value: any observed difference comes from the harness and the infrastructure around it, not from a model swap.

Each journey records three dimensions that Ora publishes: steps to complete the task, native success on the customer's site, and the quality of the endpoints found. The instrumentation is granular enough to show, when an agent gets stuck in a signup flow, which step stalled and what the agent tried.

Results: the harness changes the bill with the model fixed

The numbers published by Ora, always attributed to their test:

None of these numbers describes a better model. The models were the same. The numbers describe execution.

The last number in the table deserves separate attention. During testing, one round exposed a prompt cache problem. The harness team fixed the defect and the next round measured about 15% lower total cost. The source does not publish the controls of that subsequent round: what it documents is the sequence of a defect exposed by the test, a fix applied, and lower total cost in the next measurement.

There is an honest limit that the case itself imposes: the numbers compare two harnesses on integration journeys across customer sites. The models were the same. The numbers describe execution: how many steps the harness spent to reach the goal, how often it completed the task on the customer's site without resorting to web search, and how usable the endpoints it found along the way were. They do not say that one harness is universally better than another, nor that the observed difference would replicate in other task types. What the numbers transfer is the method, not the scoreboard. In an inference budget, each of these dimensions becomes money: steps are calls, fallback is latency and extra cost, an invalid endpoint is a billed retry.

What the case teaches about measuring agents

1. Compare with the model fixed. Differences in steps, success and cost between configurations only mean something when the model is the same on both sides. Swapping model and harness at the same time leaves the result without a cause.
2. Measure per journey, not per token. The AI agent cost that matters is the one of completing the task, which includes billed retries, fallbacks and wasted steps, not the token price multiplied by volume.
3. Instrument before optimizing. The 15% cost gain only appeared after the execution trace exposed the prompt cache problem. Without a trace, the waste stays buried in the invoice.
4. Prompt caching is an audit item, not a configuration. It degrades silently and the symptom shows up as cost, not as an error.
5. Success has layers. Completing the task on the customer's site without falling into web search is a qualitatively better outcome than completing with a fallback, and both count as "success" in naive dashboards.

How to apply this in your operation

For anyone routing models in production, the case reinforces a discipline we already apply in cost per task in model routing: the unit of decision is the completed task, with the trace that shows where the cost happened. The same reasoning applies to measuring LLM provider performance, because variation between endpoints with the same model has already been documented in same-model variability per endpoint.

Three transferable steps, in the order of the case:

1. Fix the model and vary one factor at a time: harness, endpoint or caching policy.
2. Record the trace of every journey. Steps, latency, cost and success type, in the format that the LLM call trace makes auditable. The trace comes first.
3. Audit prompt caching before negotiating price or switching models, because it is a direct fix that is easy to overlook.

It is also worth noting the difference between native success and fallback, because it shows up in production silently. An agent that finishes the task by searching the web completed it, in the report. It failed at what the customer contracted: executing on the customer's own system. The trace that separates the two outcomes is the same LLM call trace that allows auditing request by request, and without it the dashboard's aggregate success hides half the story.

Anyone who already calculates the real cost of an AI application beyond the token price recognizes the pattern: the cost that kills the margin is rarely in the model's marketing number.

Frequently asked questions

What is the harness of an AI agent? It is the software that provides tools to the model and guides it from step to step until the task is complete. In Ora's benchmark, each harness ran with the same model, which isolated the difference in outcome to the harness and the infrastructure.

Is Ora's AI agent benchmark independent? It is a commercial Ora platform, which sells exactly this measurement, and the numbers cited here come from the case the company published on 2026-08-21. The estimate that 99% of the web is not ready for agents is Ora's own, not an independent measurement.

How did prompt caching reduce the cost by 15%? A test round exposed a prompt cache problem. After the harness team fixed it, the next round measured about 15% lower total cost, according to Ora. The source does not detail the internal mechanism of the defect nor the controls of the measurement round.

What should I measure first in my agent's cost? Start with the per-journey trace: steps, latency, cost, and whether the task completed on the target or with a fallback. With the trace in hand, audit prompt caching, a direct fix, and only then consider switching models.

Does switching models solve an execution problem? Not necessarily. The Ora case shows differences in steps, success and cost with the model fixed, which means part of the cost may lie in the harness, the cache or the infrastructure. Switching models without instrumentation moves the variable without identifying it.

References

• How Ora evaluates AI agents, case published on 2026-08-21

Where to start

The trace comes before the vendor decision. Before any model swap or contract renegotiation, produce the per-journey trace of the current traffic: steps, latency, cost and success type, with prompt caching audited. The Ora case suggests the order. First instrumentation, then execution fixes, and only then the vendor decision. Nexforce Router operates at this layer with routing, logs and per-call tracing, keeping measurement consistent when models and endpoints change behind the application.