Tokens per second: what changed in LLM comparison

The rate on the Artificial Analysis dashboard changed base on 03/11/2026: version 3.0.0 now includes time to first token, replaced the 72-hour window with five days and changed token counting and failure rules. Nexforce ran no benchmark of its own.

Key facts Event: Artificial Analysis methodology 3.0.0, on 03/11/2026. What changes: time to first token enters the rate, the window becomes five days and an endpoint failure counts as zero. Source: methodology 3.0.0, consulted on 12/11/2026; no Nexforce measurement.

What changed in the measurement of tokens per second?

At Artificial Analysis, output speed now runs from submission to the last block with a token, including the initial wait, and the window moved from 72 hours to five days on 03/11/2026. Two readings separated by the revision are not a comparable series.

The unit is still tokens per second, and the metric is not the same one. No reliable conversion exists between published aggregates using only dashboard numbers; per call, recalculation depends on the marks and counts. Clock boundary, window and aggregate population all changed in one interval, so comparing providers requires one definition on both sides. Treating distinct dashboards as if they measured the same thing yields a conclusion the data does not support. The honest output is declaring the definition behind each number. Two captures cannot be added: a median of one load and an average of medians over five days describe different populations. The history does not attribute a model's variation to the method changes, and neither does Nexforce. The size of the effect was not measured.

What happened

Version 3.0.0, dated 03/11/2026, redefined the output speed, window, load set and failure handling of the Artificial Analysis dashboard. The history describes each change without attributing a model's variation to it.

Artificial Analysis measures speed from submission to the last block with a token, an interval that includes the wait for the first token and reasoning the provider accounts for. With t0 at submission, t1 at the first token and tf at the last block, the rate is response and reasoning tokens over tf - t0. In the variant starting at t1, the first token falls outside the interval and the numerator. The complete historical formula, its numerator and any exceptions were not verified here. The numerator is not visible text alone: reported reasoning enters, and without API reporting the count comes from the text. In sessions with several calls, tool time stays out, so it does not measure task duration. An endpoint error counts as zero speed, and an incomplete session considers only the responses obtained.

Figure: the top bar runs from submission (t0) to the last block (tf); the bottom bar runs from the first token (t1) to the last block (tf), with the t0-t1 stretch marked. Time intervals, no scale. Source: methodology 3.0.0, 03/11/2026.

Why this matters for whoever buys inference

The change prevents treating readings from different regimes as one series. Whoever compares providers must declare boundary, window and population of each number, because a rate that includes the initial wait is not interchangeable with one starting at the first token.

The history records the window swap: the main number used to be the median of the 10 thousand input token load over the last 72 hours; now it is the average of per-load medians over five days. This is not a speed gain. The correct reading is definition, not improvement. Five days are 120 hours, 48 more than the previous window, 66.7% of additional period. The aggregate also changed: different loads now enter the main number, and a load without measurement for that endpoint stays out. Division does not produce a comparable rate, since dividing two aggregates published on different dates does not reconstruct a call. A p95 over successes hides who was left out, as the article on gateway latency and long context details.

What changes in practice

What changes for whoever reads the dashboard is the ruler, not the supplier. The table separates the previous regime from the current one across five points that alter comparisons between dates, and the last column records the effect.

Four rows describe changes documented by methodology 3.0.0 and one describes a gap in the current source. There is no retroactive conversion. Whoever builds a comparison redoes the measurement on their own load or accepts that the dashboard number is worth only the definition published that day, which shifts the renewal question: not which endpoint has the biggest number, but which definition of speed the operation needs. The effect column gives the direct consequence of each change, not a performance measure. A serious comparison fixes boundary, window and population before looking at the result, and the guide to model router metrics details criteria for that choice. Whoever records date, version and configuration compares what was measured, and conversion between regimes stays with whoever measures.

What to do now

Five decisions reduce the risk of buying inference on a dashboard number that does not measure what the operation needs. They apply to renewal, route change and the first test of a new provider, and none depends on a supplier.

The test behind a routing decision repeats the same tasks on the candidate endpoints in alternating order, recording concurrency, region, version and date. Time to first token, total duration and success percentiles enter separately, each with declared population and the failure count in the same window, because a p95 over successes hides the timeouts. Task-based evaluation covers the choice by quality, and performance measurement guides the comparison of loads. Neither replaces the test on your load, and the useful result is a table with boundary, window and numerator declared. Repeating the load at different times exposes variation, and the queue share only appears with instrumentation that measures the wait; without that record, comparison becomes opinion.

1. Fix the boundary in the contract. Declare boundary, numerator and window.
2. Ask for p50 and p95 per metric. Require TTFT, total duration and error rate with a declared population.
3. Measure on your own load. Repeat the tasks, with alternating order and region.
4. Preserve the failures. Error over attempts, and success-only percentiles kept apart.
5. Record the tokenizer. Unreported reasoning enters as a limitation, never as equivalence.

Frequently asked questions about tokens per second

Tokens per second measures the output rate of a request, and the number changes with the clock boundary, the window and what counts in the numerator. At Artificial Analysis, the initial wait began to enter that divisor in version 3.0.0, of 03/11/2026.

The Artificial Analysis page shows recent readings under methodology 3.0.0, published on 03/11/2026, and the captures from 19/10 and 09/11 of 2026 were not reproduced. Because formula, window and population changed in one interval, the difference between an old reading and the current one does not separate method from performance. The page publishes averages per endpoint and load in the five-day window, and does not prove performance in the buyer's operation. The samples behind the average did not appear in the consultation, so the aggregate cannot be rebuilt. Measuring an endpoint requires repeating the load and recording definition, window and date used. The aggregate describes its own definition, not your load's latency, and a complete operational comparison asks for attempt population, failure rate and percentiles over successes.

Do tokens per second include time to first token?

In the current Artificial Analysis methodology, yes. Duration runs from request submission to the last block with a token, so the initial wait enters the divisor. A rate calculated from the first token declares a different measurement boundary, and the two numbers answer different questions about the same call.

Is TTFT the same as time to first response token?

Not necessarily. In a model that reasons, the first token to arrive may be a reasoning token and can come before the visible response one. When the protocol does not distinguish the two, record the limitation and do not treat the two providers' timings as equivalent.

Does p95 replace the failure rate?

No. The p95 describes a distribution of times for the defined population, while the failure rate uses the total number of attempts as its denominator. If only successes enter the percentile, errors and timeouts disappear from the result, and a route that fails looks stable.

Does a lower speed on the dashboard prove that the endpoint got slower?

Not on its own. The comparison requires verified captures, equivalent configurations and knowledge of the changes in formula, load and aggregation. Without that basis, the difference does not separate method from real performance, and the path is to repeat the load in your own operation.

References and Further Reading

The main reference is the Artificial Analysis performance methodology, version 3.0.0, published on 03/11/2026 and consulted on 12/11/2026. Every definition cited here comes from it, and the sources below allow checking boundary, loads and aggregation rules.

The performance methodology describes the request boundary, the loads and the aggregation and failure rules behind the findings. The K2 Horizon 375B A23B page and the providers dashboard are dynamic sources, not captures. The math is a window calculation: adding readings from distinct windows does not reconstruct a call. Conversion between windows is arithmetic, 120 hours minus 72, divided by 72. Reproducing an endpoint's value would depend on the individual data, loads, configuration and runs that make up the aggregate, plus failure rules and chosen window. A dynamic reading does not confirm an old capture, so preserve the consultation date and flag scope changes in each reading published. The LLM call trace complements the reading on individual records.

What to watch in the coming days

The next version of the Artificial Analysis methodology is the event to follow, not the difference between two dashboard numbers. Until then, each published reading is a dated portrait of a specific definition, with source, explicit limits and date.

Three signals are worth watching: a new version of the history, a change in the set of measured loads and the way the dashboard presents failures. Nexforce Router offers testing of one prompt across multiple models, call tracing, logs and metrics, and serves as instrumentation for a routing policy based on comparable measurement, subject to what is available and validated in the environment itself. No dashboard number becomes an SLA because of a public page. LLM observability helps plan the collection behind that decision. Each dated version defines the rule. While it does not change, compare only readings under the same definition, with the number next to the boundary it uses, because the public page does not recreate the old capture.

The buyer's decision is to measure equivalent loads, quality and failures before renewing a route, with Router as the instrument.