Evals in CI: the gate that passes the pull request and fails in production

A support agent started telling customers the refund window was 30 days. The policy said 14. The cause was one line in the system prompt, and the build went green, because the pipeline checks syntax, types and software tests, and none of that reads what the model answers. The first person to see the wrong answer was the customer.

The case opens a technical guide published on 1 October 2026 by a model aggregation platform, and it explains why the eval gate exists: to turn one prompt line into an event that breaks the pull request. The mechanism has four small holes, all of them implementation, and each one delivers the same result by a different path: a green check that measured nothing.

The gate protects the prompt, not the route. It compares the model against a fixed set of cases, inside an environment you control, and it stops there. The production route, with the cheapest provider of that week, stays outside the gate until someone measures it.

What an evals gate in CI needs before it exists

An evals gate in CI is a job that runs a fixed set of cases on every relevant change and fails the pull request when the pass rate falls below a measured threshold. Before the workflow, three prerequisites exist: the case set, the scoring rule with the threshold, and enough repetition to trust the rate.

A test case is an input and an acceptance rule, nothing more: the rule can be a text check or a check of what the agent did with the tool; what matters is that a machine runs it and that it sits in the repository, not in the head of whoever reviews.

The set lives in the same repository as the prompt and changes only through a reviewed pull request. Edited straight on the main branch, it protects nothing: the next person in a hurry adjusts the case instead of the agent. The guide starts with three cases and grows by a single rule, one new case for every failure in production. For the initial set, the recommendation of 20 to 50 simple tasks taken from real failures comes from Anthropic. Start small.

What to measure is already mapped in evaluating AI agents beyond the final answer and agent evaluation metrics in the pilot. The question here is another one: what the pipeline does with that set.

Prerequisites before the first hole

Three things have to exist before any workflow: a repository with CI on pull requests, an API key stored as a secret, and permission to edit branch protection. Without the third, the gate is born with no power to block a merge, and a gate that does not block is an expensive report.

• A repository with CI on pull requests, so the eval can be a job.
• The key as a secret, with a plan for the pull request from a fork, which receives no secret at all.
• Permission on the branch protection rules, because that is where the check becomes mandatory.

The third item is the one that most often drops off the list, because it usually belongs to someone else. Without it, nothing is mandatory. Whoever writes the workflow rarely administers branch protection, and the result is a gate that runs beautifully and blocks nothing.

The runtime of the agent under test only has to be callable from a job. Nexforce Code runs headless in automation and CI; the eval harness is your script.

Hole 1: the filter and the required check decide whether a gate exists

The trigger has to be a path filter at the job level, and the job has to be a required status check in branch protection. Getting the first wrong produces a green check with no eval; getting the second wrong produces a red eval that the merge button ignores.

A filter at the workflow level looks economical and is the worst arrangement: when the path does not match, GitHub creates no job at all, the required check stays pending and the pull request never becomes ready to merge. The eval that does not run becomes a permanent lock.

At the job level the behaviour deceives in the other direction. A job that does not match the condition does not execute, and GitHub reports it as a passing check. The merge goes through, the eval never existed, and the green documents only that the condition was false.

The path list has to include the workflow itself and the eval script itself. Without them, a pull request that breaks the scoring logic does not trigger the eval, enters untested, and the next regression is attributed to the model.

The required check is what is missing, and it is where most teams stop. Without marking the job as mandatory, a red eval appears as an X and the pull request stays mergeable, because GitHub informs and does not block. Worse: if the job that decides whether something relevant changed fails, GitHub skips the eval job, and a skipped job counts as approved. That is why both checks are mandatory.

Concurrency with cancel-in-progress cancels the previous run when someone pushes again to the same branch, and without it every push pays for an eval nobody reads. A pull request from a fork receives no secrets and fails on the missing key: running the eval on the push to the main branch as well covers the gap.

Hole 2: with no pinned provider, you are measuring the router

The same model slug can be served by different providers in two runs, and the difference shows up in the report as a prompt regression. Pinning a single provider in the eval call, with no fallback, keeps the route constant so that the measured variable is the prompt.

The slug identifies the model, not the place it answered from. Two endpoints with the same label return different answers, and the report, which only knows the label, records the difference as if the prompt had got worse.

The result is the worst kind of false positive: it blocks a correct pull request and teaches the team to distrust the gate.

In the eval call, one provider. And no fallback. In the API of the platform that published the guide, that is a provider order and a flag that disables substitution (provider.order and allow_fallbacks); fields of that API, not a market standard, and in another gateway the equivalent pins the call to a single endpoint.

Pinning the provider closes the endpoint and leaves the review open. The slug also does not identify which build of the model answered, so a new build served under the same slug and the same provider reproduces exactly the false positive of this hole, with the pinning already in place and no field in the report to record the difference. It is a limit of the instrument, not a configuration detail.

The limit goes on the record: pinning the provider inside the eval is measurement control, not route coverage. It keeps the route constant so that the measured difference belongs to the prompt, and the production route stays outside the gate until someone measures the route, with another case set. Endpoint evaluation and routing policy deals with that side: there, the threshold is the cut that decides where the request goes; here, the threshold decides whether the pull request enters.

In production, the route is a policy: model choice by cost, latency and context, failover, spend ceiling per key and a trace per call. Nexforce Router is where that policy runs in the agents a company puts into operation, and that is why the eval cannot confuse the policy with the prompt. Whoever chooses the model is measuring something else, in three agentic evaluations to choose a model.

Anyone running an agent in production changes provider for price in the same month they touch the prompt, and the eval reports a prompt regression when what changed was the route.

Hole 3: a threshold picked by eye blocks noise

Picking the threshold by eye is the most common mistake, because the number sounds reasonable: a high threshold sounds like a serious requirement. The same set, run six times without touching anything, does not return the same rate on every run. The floor is the worst of them. A low floor has a wrong reflex and a right order. The reflex is to lower the threshold. The order starts far from the number: the assertion, then the determinism parameters, and last the sampling. The threshold is the last thing to move.

The first check is the acceptance rule itself. An assertion that demands the exact word fails every time the model writes the synonym, and the prompt has no way to prevent that. The rate falls, the floor falls, and the number in the report measures the fragility of the test, not the quality of the answer. Lowering the threshold before looking at the assertion turns that fragility into permanent policy: the case that failed on a bad assertion can no longer fail anything, because the threshold was calibrated to accommodate it.

The second is determinism. temperature and seed only help in models that declare both in supported_parameters; sending them without require_parameters: true makes the provider receive and ignore the values, which is worse than not sending them, because it produces a feeling of control. There are models that declare neither.

The third is sampling, and the path that works on any model is to repeat the run with majority voting. It costs more and does not depend on the provider. What the majority buys is stability against noise; what it delivers in exchange is sensitivity. A case that fails in a minority of the samples counts as approved, so an agent that got worse and started failing in a minority of them stays green, and only shows up when the failure becomes the majority. The intermittent regression, the hardest one to see in production, stays out of the report by construction.

With the three checks done, measuring is what remains: six runs on the same branch with no change, and the threshold cannot sit above the floor.

With three cases and one of them unstable across six runs, the floor is 0.67, and a threshold of 0.9 blocks a pull request with no regression at all. The problem is not the 0.9, it is the size. With three cases the rate only takes four values, and a threshold at the floor has a resolution of one case: it fails when two cases break, not when one breaks. A single case can get entirely worse without the pull request stopping. The resolution of the gate is the inverse of the size of the set, and that is what decides which drop the pull request can see.

The conclusion is to grow the set, not to live with a loose threshold. The guide reaches the same conclusion and points to 20 cases or more; the range of 20 to 50 simple tasks taken from real failures, for the initial set, is the Anthropic recommendation cited at the start. A small set with a lowered threshold is the definition of a decorative gate.

Two failures still have to be separated, and the script already separates them if you let it. The first code covers "the agent got worse"; the second, "it was not possible to measure".

If both paint the same red, the team learns to ignore red, and an ignored gate costs more than no gate at all.

Hole 4: a test that can be loosened is not a test

CODEOWNERS on its own asks for a review, it does not block a merge. Without ticking the option that requires approval from the code owner, the cheapest path to unblock a red pull request is to edit the case that failed, and the gate starts protecting the merge instead of production.

Consider the incentive before the tool. The person has a red pull request at six in the evening, the case that failed points to an answer she considers acceptable, and there is one line in the set that can be adjusted in thirty seconds. Adjusting the prompt takes an hour and touches something she did not write.

In that arrangement, the test set loses. Not through bad faith, by design.

The defence is to treat the set as production code. Branch protection is what requires the file owner's approval; without that flag, the review request is advisory. With it, loosening the case requires the signature of someone who understands that the line is the agent's contract with the customer. The edit stays in the diff, and the reviewer sees the expectation move from 14 days to 30.

How to know the gate is working

On the first prompt change after installing the gate, open the run log and confirm that the eval ran. A wrong path pattern leaves the job green without executing a single case, and the proof that the gate exists is in the log, not in the check.

The green check is the weakest evidence in the system. A ten-second test that never called the API and a four-minute test that ran the whole set produce the same symbol on screen.

Three checks settle it. Read the log and find the list of cases with the pass rate; if the log only holds the success message, the eval did not run. Break a case on purpose and watch the pull request turn red. And confirm the eval job as pending while it runs, because a check that never goes pending is not mandatory.

It is worth repeating the test at every change to the workflow. A path filter is the kind of configuration someone adjusts on a Friday to unblock a merge, and the next week's prompt change no longer triggers anything.

What it costs to run this

The cost is the number of cases multiplied by the samples per case and by the frequency of pull requests that touch the prompt. The script prints the cost of each run from the usage.cost field, which is a field of the API of the platform that published the guide; in another gateway it does not exist, and the cost comes from tokens multiplied by price. That number is what decides whether the gate fits the budget. Without it, faith is what remains.

The bill grows linearly with the samples, and whoever raises samples to stabilise the measurement multiplies the cost by the same constant; the calibration of the threshold shows up on the invoice, not on the pipeline clock.

A script with a threshold check is enough on its own. Eval platforms add reporting, run history or ready-made metrics on top of the same exit-code pattern, and the ones cited by the guide are DeepEval, Braintrust, Arize and Galileo. The trade is an operations decision, not a quality one.

What to do on Monday

Five steps, in this order, each leaving one artifact in the repository: the test set, the workflow with the filter at the job level and the required check, the owners file, the threshold with the six runs recorded, and the cost per run printed in the log.

1. Write three cases from real failures of the agent and commit them to the same repository as the prompt. Artifact: the versioned set.
2. Create the eval job with a path filter at the job level, including the workflow itself and the script itself in the list. Artifact: the workflow.
3. Mark the filter job and the eval job as required status checks in branch protection. Artifact: the protection rule, recorded in the runbook.
4. Run the set six times against the main branch without changing anything, record the worst rate and set the threshold below it. A low floor is fixed in the assertion and in the sampling first, and the set grows before the threshold comes down. Artifact: the number documented, with the date.
5. Turn on the cost report in the log and review the figure in the first month. Artifact: the cost per run.

The step teams skip is the fourth, because it produces nothing visible and delivers a number nobody wants to find low. Nothing on the list requires buying anything, and the artifacts stay in the repository or in the runbook. The production AI agent platform checklist covers what comes next: the gate is the first measurement layer.

Frequently asked questions

Does an eval in CI replace unit tests? No. A unit test verifies the code you wrote; the eval verifies the behaviour of the model against an input. The two live in the same pipeline and fail for different reasons: one points to a broken function, the other to an answer that changed. The gate covers the layer the unit test does not reach: it does not check your orchestration code directly; it tests what the agent answers and which tools it called.

Do I need an eval platform to start? No. A script that runs the cases, applies the scoring rule and exits non-zero below the threshold solves the gate's problem. Platforms add reporting and history on top of the same exit pattern, and the trade is justified when the team needs to compare runs over time, not before that. The script comes first.

How do I keep the gate from making the pull request slow? A path filter at the job level, so the eval runs only when the prompt, the test set or the gate itself changes, and concurrency with cancel-in-progress to cancel the previous run when someone pushes again to the same branch. Repeated sampling costs time, and the calibration defines how many samples are needed to avoid blocking on noise. A slow gate is a gate the team learns to work around.

The eval failed and the agent looks the same. What now? Look at the exit code first: if it is the code for an eval that did not run, the problem is the key, the provider or a broken case, not quality. If it is the regression code, run the set on the main branch without changing anything and compare with the recorded floor. A difference inside the noise is a high threshold, not a worse agent. Compare the two numbers before touching the prompt.

References

• Technical guide on gating pull requests on LLM evals, published on 1 October 2026 by a model aggregation platform (read on 21 October 2026).
• Recommendation of 20 to 50 simple tasks in initial sets: Anthropic, Demystifying evals for AI agents, cited by the guide (read on 21 October 2026).
• Behaviour of the pending check and of the skipped job: GitHub Docs, Skipping workflow runs and About protected branches (read on 21 October 2026).
• Evaluating AI agents beyond the final answer
• Agent evaluation metrics in the pilot
• Endpoint evaluation and routing policy
• Three agentic evaluations to choose a model
• Production AI agent platform checklist

The hole that remains after the gate

The argument here is not that the gate is useless. It works, and it works cheaply, as long as it is a calibrated instrument, with an owner and a known margin of error.

What it does not do is look at the route. The route stays out of the report. An agent in production answers through a policy that changes from week to week, for price, for latency or for provider availability, and none of those changes passes through a pull request.

The next measurement, therefore, is not of the prompt. It is of the route: which endpoint answered, with what latency, at what cost, and whether the answer the customer received came from the model the policy promised. Until that is in the same report, the gate stays green and production stays the place where the truth shows up.