Skip to main content

Decision model evaluation: 97% is not enough in 2026

Rafael Torres
Rafael TorresNovember 17, 202614 min. read
Decision model evaluation: 97% is not enough in 2026

A decision model evaluation starts with a difference in the denominator. In the launch of Decision Index v2, by Artificial Analysis, on November 8, 2026, the AA-Paperclip evaluation, with 30% of the index, showed a model that got 97% of individual answers right and only 63% of documents. The first number is not the approval rate of the second.

That distinction changes a purchase. To compare candidates, a team examines correctness, confidence and consistency, then measures cost per accepted task in its own operation. A correct classification can arrive with a wrong extraction or with an action that requires review. The benchmark selects candidates. The internal test verifies full acceptance.

Decision frame: key facts

  • The 97% and 63% example uses individual answers and documents as different units.
  • The Decision Score weighs correctness and confidence; consistency answers another question.
  • Average cost per task in the index is not cost per accepted task in the operation.
  • The final choice requires your own cases, a record of attempts, and an acceptance rule defined in advance.

What are the main findings?

Decision Index v2 combines six evaluations into a weighted average and compares different model families. The final score aggregates correctness, confidence and stability; separating those three is work the reader does, not a property of the index. The aggregate result does not prove that an agent completes a real operation. Reading starts with the unit of each measure.

  1. In AA-Paperclip, the example of one model combines 97% of individual answers correct and 63% of documents with every answer correct. Each document requires five decisions and ten extractions.
  2. In v2, 45% of the index comes from private sets, against 25% in the previous version: those are the two private evaluations, weighted at 30% and 15%. The source adopts private data to get closer to real utility and to make optimization against the test harder. The cost of that choice is a narrower path to independent reproduction.
  3. The Decision Score weighs correctness and confidence; AA-Symmetry examines stability under equivalent wordings. One result does not replace the other.
  4. The index estimates cost per task from token prices. That account does not include, by definition, the acceptance of the task inside a company's process.

The practical conclusion is not to abandon the index. It is to stop a score from carrying a decision it was not built for, such as choosing an agent for a task that requires permissions, tool execution and acceptance under the company's rules. A team that measures only accuracy per answer knows little about the work that reaches the end without correction.

How was the data collected?

Artificial Analysis published the v2 scope on November 8, 2026, gathering tests of decision, extraction and choice among possible actions. AA-Paperclip holds 1,500 documents, with 15 required answers per document. The evaluator itself ran the measurements. This is not an original Nexforce study.

ElementPublished scopeLimit for interpretation
AA-Paperclip1,500 documents; five decisions and ten extractions per documentA document passes only when all 15 answers are correct
Private sets in the index45% in v2, against 25% in the previous versionThe private share limits independent inspection of the cases
Decision ScoreWeighs correctness and assigned confidenceDoes not measure the success of a complete workflow on its own
AA-SymmetryObserves stability across equivalent wordingsA stable answer can be wrong or can declare that it does not know

The AA-Paperclip description states that documents and answer keys are private. That matters to anyone who intends to reproduce the result: the published unit of evaluation can be understood without access to the full set that would allow the test to be run again. Test privacy does not make the result useless. It makes independent validation narrower. The published scope fixes the numbers of the test: 1,500 documents, each with five decisions and ten extractions, 15 required answers per document, and a document counts as approved only when all 15 answers are correct. The version matters too. The dated article presents v2, while the methodology history already records v2.1 and a change in AA-Symmetry. An older result has to travel with its version and its date. Swapping the method mid-comparison while keeping the same caption produces a series that looks continuous without necessarily being comparable.

Why do 97% correct answers not mean 97% approved documents?

In the published AA-Paperclip example, 97% is the share of individual answers that were correct, and 63% is the share of documents that passed the full criterion. The second measure requires that the five decisions and the ten extractions of the same document all be right. The numbers describe the same test, but not the same unit.

Measure in the exampleUnitPublished resultCriterion
Individual accuracyAnswer97%Each answer is evaluated separately
Full accuracyDocument63%All 15 answers in the document have to be correct
inline-01.png

A wrong extraction fails the document. It can accumulate several correct answers and still fail the full criterion. The individual average does not show how many documents pass without correction, but the published breakdown shows where the criterion breaks: for the same model, the five decisions came out correct in 71% of the documents and the ten extractions in 88%. The error that pulls the document down sits more in the decision than in the extraction, and the example carries no error rate per individual field. In your own operation, acceptance can require correct information, a permitted action and a record for review. The test has to separate a wrong answer, an intervention and an accepted task. Completing the run is not enough: a completed task is accepted only when it also meets the acceptance criterion defined for the process. The AA-Paperclip result does not predict the acceptance rate of another process.

Do confidence and consistency measure the same thing?

No. The Decision Score described by Artificial Analysis weighs correctness together with the confidence assigned to the answer. AA-Symmetry examines whether the decision holds when the wording is equivalent. A confident answer can be wrong, and a consistent answer can repeat the same error or the same state of not knowing.

MeasureQuestion it answersWhat it does not show on its own
CorrectnessDoes the answer match the evaluation key?That the declared confidence tracks the correct answer
CalibrationDoes the assigned confidence track the observed frequency of correct answers?That equivalent decisions hold across wordings
ConsistencyDoes the decision hold across equivalent wordings?That the decision is correct or useful

Confidence is not a guarantee. When a model declares high confidence on wrong answers, the team has to identify those cases. In an internal evaluation, it pays to store the confidence next to the result and compare bands against the observed correct answers. Without an answer key or a reliable review, calibration itself has no basis. Consistency has another job. A team can resubmit equivalent cases and check whether the classification or the decision changes, keeping a record of the model version, the instructions and the test conditions. It can also repeat runs of the same case to observe variation across rounds. These internal tests complement AA-Symmetry. They must not be presented as if they were the same published measurement. The launch article notes that a model can be consistent in declaring that it does not know. That can be an appropriate answer when data is missing, but it does not prove the task was solved. The evaluation has to record when abstention prevents an improper action and when it blocks the conclusion of a case that had enough information.

What does the benchmark not prove about an agent in production?

The index tests capabilities in evaluations selected and run by Artificial Analysis, not the complete execution of an agent connected to the company's systems. Choosing the right answer about a tool differs from running it with the correct permission, handling a failure and delivering a result the operation accepts.

Test or situationWhat is observedWhat stays outside that test
When2CallChoice among candidate answers, including calling a tool or asking for informationActual execution of the tool and the effect on the company's system
AA-PaperclipDecisions and extractions scored per answer and per documentApproval of a real business task after execution and review
The team's own workflow testResult against the criteria and permissions the team definedAutomatic generalization to other processes or volumes

The When2Call page describes the evaluation as a choice among candidate answers: answer, call a tool, ask for information or recognize that no answer is available. That measures a relevant decision. It does not show, on its own, that the call happened, that access was authorized or that the target system accepted the operation. The difference shows up in the trace. For each task, the team has to inspect the decision taken, the data available, the execution attempts and the reason for an intervention. A correct final answer can hide an earlier improper attempt; an incomplete final answer can be the visible consequence of a permission that was correctly denied. The article on agent evaluation beyond the final answer details why path and permissions deserve their own measurement. That observation does not disqualify a model test. It defines its reach. The external evaluation helps build a shortlist of candidates; the real workflow decides whether any of them meets the company's execution and acceptance rules.

How to measure cost per accepted task?

The measure proposed for the operation divides the total cost of the evaluated runs by the number of tasks accepted in that same set. The numerator has to include failures and retries, and declare which components it counted. Acceptance criteria and cases are defined beforehand and held equal across candidates, though the number of accepted tasks can vary. Contingency counts too, when it is consumed.

Cost per accepted task = total cost of the evaluated runs / number of accepted tasks is a measurement proposal, not a Decision Index result. The launch presents estimates per one million tasks, and the index page defines average cost per task as a weighted measure across evaluations, calculated at token prices. Billing shape and input repetition affect the comparison. The published cost must not be divided by the success rate of another test to produce a supposed cost per accepted task. Keeping token price and cost per task apart holds the measures separate. In the operation, the first step is to declare the perimeter of the account. Inference, retries and contingency resources enter when they are consumed; human review can enter when its cost is measured. If a component is unavailable, the comparison has to mark it as missing instead of treating it as zero. A cost per attempt record helps structure that evidence.

The second step separates autonomous acceptance from acceptance after intervention. Both can count as accepted tasks in the total, provided the criterion is stated, but a single average does not reveal how many tasks required human work. The rate of each mode and its costs have to accompany the quotient.

If no task is accepted, the division is undefined; there is no calculable cost per accepted task for that set.

A comparable test uses the same cases, criteria and measurement window for each candidate. The team records the cost of every attempt tied to the task, rather than the last successful call alone. That stops a failure from disappearing from the account just because another attempt delivered the result.

How to choose a model for an agentic workflow?

The choice starts with a bounded task and ends with evidence of acceptance, not with a position in a ranking. For each candidate, the team observes correctness per task, confidence, repetition, failures and cost on the same set of cases. The external index serves as triage; approval belongs to the operation's test.

  1. Define acceptance before execution. Record which decisions, extractions and actions are mandatory, who can approve them, and when a human intervention changes the result from autonomous to assisted.
  2. Build cases with a verifiable answer key. Select tasks that represent the permitted volume, including cases with insufficient information and situations where the action must be refused. Protect data and permissions during the evaluation.
  3. Run and repeat under recorded conditions. Keep the model version, the instructions and the configuration identifiable. Run the same cases across candidates and repeat cases to observe consistency, without confusing repetition with correctness.
  4. Inspect failures by stage. Separate a decision error, a wrong extraction, an inadequate tool choice, a permission block and an execution failure. Keep the cases that end without a conclusive answer as well.
  5. Calculate and decide. Present accuracy per task with confidence, autonomous and assisted acceptance, cost per accepted task and relevant failures. Approve only the candidate that meets the limits defined for that process.

The error threshold has to reflect the consequence of the decision. To compare a reversible failure in an internal classification with an action that changes a business record, the team weighs the concrete impact and the cost of reversing each case. The criterion can require human approval for one class of actions and allow autonomous execution for another, as long as the team tests both routes and records the difference.

After the choice, the evaluation does not end. Changes in the model, the instructions, the tools or the distribution of cases justify a new round with the same documented criterion. Without that control, the team changes the measurement conditions and credits the model with a difference it has not isolated.

Frequently asked questions

Decision Index v2 offers an external reference for comparing decision capabilities, but each measure carries a unit and a purpose. The answers below separate the published result from the evaluation a team has to run in its own operation, with its own data and its own acceptance rules.

What is Decision Index v2?

It is an index from Artificial Analysis that gathers model evaluations on decision tasks. The November 8, 2026 launch describes measures of correctness, confidence and consistency, plus estimated cost per task. Its results belong to the evaluator's tests and do not represent guaranteed performance in a company's agent.

What does getting 97% of answers right mean?

In the cited AA-Paperclip example, it means individual answer accuracy for one of the evaluated models. In that same measurement, 63% of documents had all 15 answers correct. They are not two estimates of the same rate, and they are not a performance promise for another model or another document set.

Does calibration replace a consistency test?

No. Calibration compares the assigned confidence with the observed correctness; consistency examines the stability of the decision under equivalent conditions or across runs. A system can repeat a wrong decision or declare that it does not know in a stable way. For a production choice, the team has to examine both signals next to the task result.

How to calculate cost per accepted task?

The team divides the cost of the evaluated runs by the number of tasks accepted in the same set, including failures and retries in the declared cost. To interpret the result, it separates tasks accepted autonomously from tasks accepted after intervention and states which cost components were measured, without treating missing components as free. With zero accepted tasks, the quotient is undefined.

References and Further Reading

The external results belong to Artificial Analysis, in the v2 scope of November 8, 2026. The version matters. The article does not state an update cadence. When citing the percentages, the team names AA-Paperclip, the unit of each measure and the date, and checks the version history before comparing with another edition.

Next step

The team defines one business task and its acceptance criterion before choosing the model. Nexforce Agents is the development and implementation unit for B2B agents; Nexforce Work offers a work environment with approvals and permissions for agents. Those capabilities do not replace measurement: the team has to test the workflow and record its results.

The purchase decision gets clearer when the report shows, for the same set of cases, what was accepted without help, what required intervention and what all the attempts cost. The benchmark opens the conversation. The final choice confronts acceptance, interventions, failures and cost with the limits defined for the operation.

Nexforce

Deploy Work and Code Agentswith zero software licensing costs

Automate operational tasks and code writing autonomously with dedicated agents integrated into your systems

Free Trial

Related articles