Throughput per megawatt: self-host or API for AI?

A GPU can produce more tokens per watt and still lose on the cost of a request. Throughput per megawatt measures capacity under a defined workload, not the cost of idle infrastructure. Dedicated capacity deserves preference when its measured total cost is lower under equivalent requirements; an API wins when it meets those same requirements for less.

Artificial Analysis published the AA-AgentPerf results on October 15, 2026. The largest reported multiple reaches 5.7x in a specific comparison of throughput per unit of GPU power. It is neither a promise of savings nor a comparison of invoices. A purchase decision requires comparing the full cost of the alternatives, a calculation the benchmark does not provide.

What does throughput per megawatt measure?

Throughput per megawatt relates accounted tokens per unit of time to measured power, in megawatts, under a defined workload. The measurement boundary matters. In AA-AgentPerf, power covers GPUs and their memory, not the whole facility; the measure describes capacity under test conditions, but does not reveal total energy use or cost per request.

The source compares Vera Rubin NVL72 with the range of GB300 configurations described by Artificial Analysis, including GB300 NVL72. The table brings together the largest approximate advantages found at specific per-user output speeds, not the ratio between each system's peak throughput.

The comparator matters. At the Pro maximum, the GB300 line uses smaller configurations built by Artificial Analysis, not a full rack. At the Flash maximum, the Vera Rubin configuration itself is smaller than a full rack. Output limits also differ for Flash, and the source reports that this modestly favors Vera Rubin throughput. The topologies are not interchangeable.

There is another boundary. According to the AA-AgentPerf methodology, throughput includes new input tokens, input tokens reused from cache, and output tokens. It does not represent only tokens reprocessed or billed by an API. Output speed excludes the wait for the first token. For Pro, the conclusion changes when that wait is included in the end-to-end speed metric.

CPU, networking, and cooling are outside the benchmark's power measurement. A team that treats the denominator as total datacenter consumption starts the calculation with an invisible expense.

How does prefix reuse change processing?

Prefix caching allows a system to reuse the processing state of an earlier input when the prefix is compatible and the KV state is available to the new request. Repetition creates an opportunity, not an automatic hit. The portion of the prefix with a compatible state available can be reused; the rest requires prefill.

During prefill, the model processes input tokens and produces intermediate attention states known as KV. If a later request starts with the same prefix and can access the compatible state, the system can reuse that work and process the new portion. The output still has to be generated for the request. This differs from response caching, which reuses an already generated response.

In the trajectory set used to describe the AA-AgentPerf workload, inputs reach 132,000 tokens and, on average, 97% of input tokens repeat the previous request. These values describe that trajectory set. They are not a cache hit rate, financial savings, or a characteristic of any other application.

State retention matters as much as textual similarity. If the cache expires, is evicted under memory pressure, or is inaccessible to the process serving the request, the repeated prefix does not yield the expected reuse. The team should check compatibility, retention, and the observed route together, without assuming that any gateway provides session affinity or KV-aware routing.

A hit rate can be calculated by dividing eligible tokens served from cache by eligible tokens. The proportion of requests with any hit answers a different question.

How can throughput per megawatt be converted into cost per request?

There is no direct conversion from throughput per megawatt to cost per AI request. To decide between self-host or API, the comparison requires dividing total attributable expense in a window by the logical requests completed in that same window. Power and throughput indicate capacity; utilization, energy, billing, and successful completion determine the bill.

The proposed formula is simple, but its boundary must be explicit:

Cost per request = total attributable cost in the window / logical requests completed in the window

A logical request is the unit the operation defines for delivering an accepted response. If a failure or retry occurs, the associated expense stays in the numerator; the denominator does not turn every attempt into a new completed request. A task can involve multiple requests, so cost per task is a different measure. The real cost of an AI application helps distinguish the expense of a task from the advertised price per token.

For inference on self-hosted infrastructure, the numerator includes contracted capacity or capital cost allocated to the period, attributable facility energy, networking, operations, and other relevant expenses. Idle capacity stays in the calculation. Energy must cover both active and idle periods within the chosen boundary, without counting costs already included in another component twice.

For an API, include the attributable invoice, including input, output, any cache charges, failures, and billed retries, as well as additional networking or operational expenses absent from the invoice. The finance team uses the same currency and period for both alternatives. The platform team confirms the workload, the same model or verifiably equivalent quality, and the acceptance criterion.

Power is not energy. Average power multiplied by time gives energy when the boundary and window match. With power in kilowatts and time in hours, energy is expressed in kilowatt-hours; multiplied by the tariff in currency per kilowatt-hour, it yields energy cost. Measuring the GPU does not automatically account for attributable facility energy used by CPUs, networking, and cooling.

Caption: Average power multiplied by time, within the same boundary and window, gives energy. GPU power in the benchmark is not attributable facility energy. Alt: Diagram compares the components of cost per request for self-hosted infrastructure and an API, with power and energy shown separately.

Accounted tokens divided by time yield throughput; throughput divided by measured power in MW yields throughput per MW. This indicator does not reveal how many requests were completed: input and output length, concurrency, hits, and usable capacity must be observed.

Which costs does a per-token comparison leave out?

A token price alone does not capture reserved and idle capacity, facility energy, operations, or additional costs outside an API invoice. The comparison must attribute each component to the same window and identify its source, avoiding both omitted expenses and charges counted twice.

Idle cost stays in the calculation.

The guide on reducing inference costs discusses operating expenses. In this comparison, the team needs to measure how many completed logical requests the paid capacity delivers and how much the GPU and other components cost.

The response must also meet the service requirement. The model router benchmark metrics help assess the quality, latency, and cost of routes. None should be confused with the GPU power used in the AA-AgentPerf denominator. The cheapest route that fails the quality criterion is not an equivalent alternative.

When does dedicated capacity tend to make sense?

Dedicated capacity tends to make sense when it delivers completed logical requests at a lower measured total cost than the API alternative, with equivalent requirements. Utilization and actual cache reuse, where present, enter the assessment. A recurring workload alone does not guarantee the result. The choice depends on expenses across the entire window and on quality and service requirements.

GPU utilization must be observed during both active and waiting periods. Measuring only the interval when the queue is full can make dedicated capacity look better in the spreadsheet by omitting idle periods. The comparison window must include that idle time, as well as attributable maintenance and operating expenses.

Where caching is present, the team measures whether the prefix is compatible, whether the state remains available, and whether the route reaches the process holding it. A repeated input sent to another process without access to that state may require prefill again. A similar history alone does not produce a hit.

When the result meets the requirements and the full cost is below that of the API measured under the same workload, dedicated capacity has an economic case. If fixed expense dominates the window or the expected cache reuse does not materialize, the same benchmark no longer supports the purchase. Advertised capacity does not pay for idle time.

When does routed consumption tend to make sense?

Routed consumption through an API tends to make sense when dispersed or irregular demand would leave dedicated capacity idle and when the effective invoice, plus additional costs, is lower under equivalent requirements. The decision requires checking available routes, delivered quality, and the billing rules of the specific contract.

Workloads vary. Models or service limits can differ between request classes; separating those classes prevents comparisons between different requirements or an expensive class from disappearing into the average.

The API's cache policy belongs to the contract in use. The provider may offer special treatment for reused input, but the team must verify eligibility, billing, and recorded usage; it should not assume the same rule applies to every route. Expenses for failed attempts and operations remain in the numerator even when the main charge looks simple.

An API is not automatically cheaper because it avoids buying hardware. If the workload recurs and dedicated capacity has a better full cost at equivalent quality, the evidence points to self-hosting. The case reverses when the effective invoice and additional costs favor the API. The Artificial Analysis study decides neither case.

How can the decision be tested on the actual workload?

The test compares the alternatives in the same window, with request classes, quality criteria, and financial unit defined beforehand, while the team records prefixes, hits, capacity, energy, expenses, and completed logical requests. The result is a verifiable calculation by class and for the whole workload, not an extrapolation from the public benchmark.

1. Define the window, request classes, the same model or verifiably equivalent quality, and what constitutes an accepted response. Record total requests dispatched, completions, failures, and retries without confusing attempts with logical requests.
2. Record input, output, and repeated prefix by class, preserving the distribution of requests.
3. Measure eligible tokens served from cache, retention time, and availability of the compatible state. Distinguish a partial hit from full reuse.
4. Measure allocated capacity, power, attributable energy, idle time, and operating expenses. Document boundaries and units.
5. Obtain the effective API invoice and add attributable costs not included in it. Check charges for input, output, cache, and attempts.
6. Divide each alternative's total cost by completed logical requests and compare quality and service requirements by class. Revisit the decision if the workload, contract, or cache availability changes.

Caption: Available compatible state allows full or partial prefix reuse; the remainder requires prefill. Alt: Flow distinguishes full reuse, a partial hit, and the absence of compatible state in the KV cache.

The procedure for deciding on an LLM route with real traffic complements this assessment with observed traffic. The financial measure remains separate: a technically acceptable route must still make economic sense in the chosen unit.

Frequently asked questions

Higher throughput does not guarantee lower cost; the calculation depends on utilization, energy, and expenses across the window. A repeated prefix is not a cache hit either: reuse requires compatible, available state. For an API, billing depends on the contract and recorded usage.

Does higher throughput mean lower cost?

Not necessarily. Higher throughput per unit of power can indicate more capacity for the workload and boundary tested, but cost per request also depends on utilization, facility energy, allocated capacity, operations, and completed requests. A financial advantage appears only when the full cost is measured under equivalent requirements.

Is prefix repetition the same as a cache hit?

No. Repetition describes the input content; a cache hit requires compatible state to be available and found when the request arrives. In AA-AgentPerf, the 97% represents average input-token repetition in that workload, not the proportion of tokens served from cache or any savings.

Does an API always charge for a reused prefix in the same way?

The contract defines the charge. Treatment of reused input depends on the contract and the rules of the route used. The team should check eligibility, recorded usage, and the effective invoice, including any cache charges and attempts. Without those data, there is no basis for assigning a price or savings to reuse.

Is a repetitive workload enough to justify dedicated capacity?

No. Repetition can create an opportunity for reuse, but the choice depends on measured full cost, utilization, cache reuse actually observed, and equivalent requirements, even if there are no hits. The comparison must include idle periods and verify that quality and service requirements are equivalent to those of the API.

References and further reading

Artificial Analysis is the source of the throughput and power results and methodology, but the study does not compare invoices or cost per request between self-host and API. The formula is a proposal. To use the financial measure, the operation must measure expenses in the same window and divide them by completed logical requests.

• AA-AgentPerf: Vera Rubin NVL72 results, Artificial Analysis, October 15, 2026.
• AA-AgentPerf methodology, Artificial Analysis.

What to measure before choosing a route

The choice does not come from the benchmark. The decision should start with a representative window, attributable costs, and logical requests completed at accepted quality, compared with an API that meets the same requirements. The alternative with the lower measured cost in that window has an economic case.

The next step is to assign responsibility for measuring energy, invoices, and quality before negotiating capacity. Nexforce Router is a gateway for accessing and routing models through an API, with configurable criteria and consumption observability. The choice among routes actually available remains subject to measurement, with no promise of savings or prefix caching.