How to evaluate and choose an LLM gateway for your company

A company that already runs multiple models in production is not choosing an LLM gateway. It is choosing between three things that are not the same: building an internal layer, contracting a managed service, or handing the problem to a model router. All three collect the same decision at a different point on the balance sheet, and most companies pick the first without ever learning the third existed. This post is the tiebreaker.

Why "choosing a gateway" is the wrong question

Choosing an LLM gateway starts wrong because the word "gateway" names at least two things with different costs. A single-endpoint proxy redirects calls to an API; a router with intent classification decides, per request, which model serves which workload. The first question is an engineering decision; the second is margin, accounted for month after month.

The awareness post on the theme, already published on the blog, explains what a gateway does and why a company running AI in production arrives at it. Here the question changes subject: not "what is it," but "which one of the several goes in, and how much does the wrong choice cost." And the wrong choice is rarely the vendor. Almost always, it is the category.

Look at what happens in Brazil when the decision ignores effective cost. A US$100,000 token bill leaves the bank account as up to US$155,000 after IRRF, CIDE, PIS, COFINS, ISS, IOF, and the FX spread. No latency benchmark captures that. The criterion that should decide the selection is billing in local currency, and it enters as the last line of the spreadsheet, when it enters at all.

What separates a router from a proxy

A single-endpoint proxy does one thing: it takes the request and puts it in front of an API, sometimes several, and returns the response. A router does another: it normalizes the request, classifies the intent, picks the model by cost, latency, and context, distributes the load, and normalizes the response. The two sell under the same name, "gateway," and the final production bill is completely different between them.

That is where the error lives. A platform team that adopts a proxy believing it bought routing discovers at the first traffic spike that static redirection sends the heaviest load to the most expensive model, all the time, because nobody there made a routing decision. None of them was made. It is exactly what the article on the model router as a middleware layer at scale describes in detail: routing intelligence is what separates infrastructure that scales from infrastructure that merely forwards.

The difference is clear in a table, and it is the table that should open any evaluation:

The most dangerous name in the market is precisely that one: "gateway" used to describe all three things in the same conversation. An evaluation that does not separate proxy from router ends up comparing the price of redirection with the price of a routing decision, and then the winner is always the cheapest on day one, never the cheapest in the quarter.

The seven evaluation criteria

Seven criteria are enough to separate a real router from a proxy wearing a gateway costume. They apply before any POC, and each one answers a cost question, not a feature question. The order matters less than the presence: a layer that fails any of the seven should leave the list, because the failure turns into a bill at the end of the month.

1. Smart routing. The layer classifies the intent of each request and picks the model by cost, latency, and context, in real time. Without it, model selection is made once, in code, by someone who does not look at the balance sheet, and the decision rots in three weeks, once the price and performance ranking of models has already moved.

2. Multi-layer failover. A vendor outage should migrate traffic in milliseconds, with configurable fallback and retry with exponential backoff. A gateway without that safety fold lets a model failure become a product failure, and the middle-of-the-night postmortem is always the same: nobody owned the failover.

3. Per-key and per-project cost governance. Spend caps by API key, by project, and by agent, with real-time consumption. It is the difference between a budget someone reviews at the end of the month and a limit the system enforces before the cost breaks.

4. Complete observability. Logs, metrics, tracing on every call, alerts, and dashboards. Without the auditable trail of each call, the savings routing promises cannot be proven, and a dedicated blog post covers how to measure the operational cost of an AI gateway in production.

5. API normalization. A single OpenAI-compatible API, swapping one endpoint for another without reintegration. It is what unlocks model change without rewriting code, and what turns a cost decision into a one-variable change.

6. Local-currency billing with an invoice. A bill in Brazilian reais, with an invoice and import tax included. In Brazil this criterion outweighs any benchmark, because the effective cost of a token in foreign currency carries up to 55% in taxes and FX combined, a number that performance-based selection never sees.

7. Migration time. What it costs to switch gateways if the decision proves wrong. A router is swappable without reintegration; a gateway as a service holds the contract and the lock-in; an internal layer holds engineering hours the business already spent.

Those seven criteria are also the list that splits the decision. The first four measure whether the layer routes or merely forwards; the last three measure whether the decision survives the first quarter of real production, when the cost in foreign currency and the cost of undoing the choice enter the accounting.

Build, buy, or route

The question that decides everything is not about the vendor, it is about where the gateway layer lives in the company's structure. The category decides. There are three paths, and the answer varies by size and use case, not by technical opinion.

Building the layer internally makes sense for a dedicated platform team that already maintains its own model grid and needs fine control over routing. The real cost is not the code, which is written in a quarter; it is carrying availability, failover, price updates for hundreds of models, and cost governance, month after month, forever.

Buying a gateway as a service makes sense for a team without platform engineering that wants the problem handled by a third party. The cut is effective cost: the service charges the subscription in foreign currency, and in Brazil that bill carries the same tax and FX load it was trying to avoid. The argument of escaping the pain of infrastructure runs into the pain of the remittance.

Routing delegates the decision to a model router as the AI infrastructure layer. It is the third path, the one most companies did not know existed when the evaluation started. Instead of building routing intelligence or buying its management, the company adopts the layer that already routes, already governs cost by key, and already invoices in reais with a local invoice. The article on proving model router savings in production details how that saving materializes in numbers, not in promise.

The tree below sums up the tiebreaker in three lines:

The cost math nobody does before the POC

Before any proof of concept, there is one calculation almost no evaluation makes, and it decides more than the other six combined. The other six lose. It is the effective cost of a token in Brazil, where a dollar bill crosses IRRF, CIDE, PIS, COFINS, ISS, IOF, and the FX spread before it becomes real expense.

The number is specific, and the Nexforce Router economics deck makes it explicit: a US$100,000 token bill costs up to US$155,000 disbursed on the direct path, because remittance taxes and conversion add up to 55% on top. The 10% CIDE applies to SaaS as a technical service, per SC Cosit 191/2017 and 99/2018. The §1°-A exemption of art. 2° of Law 10.168 applies only to a pure software license without technology transfer, a category distinct from SaaS.

The consequence is that local-currency billing stops being a convenience detail and becomes a selection criterion. A layer that invoices in reais, with an invoice and tax included, changes the entire math of an LLM gateway: the same account that blows up in dollars becomes a predictable expense, with a tax credit the direct path rarely captures. That is why item six of the list above is not a procurement afterthought, but the question that reorders the entire spreadsheet.

For anyone who has not yet measured the operational cost of the layer they already have, the post on how to measure the operational cost of an AI gateway in production offers the calculation method. The trap is assuming the cost of the gateway is the price of the layer, when it is the price of the layer plus the cost of every token it lets through on the wrong route, in the wrong currency.

How to run the evaluation in 30 days

The evaluation fits in thirty days, and the roadmap is a POC checklist, not a tender. The rule is short. Measure little, measure in real production, and abort early when a criterion fails.

First, fix the seven criteria into a scorecard before looking at any vendor, so you do not end up comparing a proxy price with a router price. Then run the same real load, not a synthetic benchmark, against the selected layers, and measure what routing decides per request, what failover recovers in seconds, and what governance cut in cost by the end of the week.

The test that the layer is a router, not a proxy, is observable: change the price of one of the models mid-test and see if the route adjusts on its own. If the answer is that the adjustment requires changing code, you are looking at a single endpoint with a grand name, and the POC has already answered. Abort on day twelve, not day thirty.

Finally, run the effective-cost calculation in reais with the real invoice, not the estimate. A layer that does not issue an invoice in reais should lose criterion six even if it won the other six, because it exports to the vendor the cost the company was deciding to absorb. The final decision is about margin, not engineering.

FAQ

Is an LLM gateway the same thing as a model router?

No. A single-endpoint proxy redirects calls; a model router classifies intent and picks the model by cost and latency on every request. The two are sold under the common name "gateway," but the production bill and the routing intelligence are different.

When is it worth building an LLM gateway internally?

It is worth it when the company maintains a dedicated platform team, needs fine control over routing, and accepts carrying forever the cost of maintaining failover, availability, and the price and performance updates for hundreds of models. Below that profile, it is not worth it. The cost of keeping the layer outweighs the cost of adopting it.

What is the cost of adopting no layer at all?

The most expensive model serves the heaviest load by default, a vendor outage brings the product down, and the dollar bill carries up to 55% in taxes and FX. The cost does not disappear. The absence of a gateway simply transfers it to the token and to the middle of the night of whoever holds the incident.

How much does it cost to switch gateways after a bad decision?

It depends on the migration time that criterion seven measures. An API-compatible router changes endpoint without reintegration. An internal layer requires rewriting the integration. A gateway as a service holds the contract and the lock-in, so the cost of undoing the choice becomes the decisive argument for not making the wrong choice.

How does local-currency billing change the evaluation?

It turns effective cost into predictable reais, with an invoice and tax credit, instead of a dollar bill that crosses IRRF, CIDE, PIS, COFINS, ISS, IOF, and the FX spread. In Brazil that is the criterion that reorders the spreadsheet. Conversion adds up to 55% to the token.

References and Further Reading

• LLM Gateway: What It Is and Why Your Company Needs One is the awareness post on the theme, defining what a gateway does and why a company in production arrives at it.
• Model Router: prove AI savings in production details the economics of routing in production numbers.

What changes for those who decide with criteria

The difference between adopting an LLM gateway by inertia and choosing one by criteria does not show up on the day of purchase. It shows up in the first month the scorecard survives real production: the failover held the outage, governance cut the cost before it broke, and the invoice arrived in reais with a local invoice, without the FX surprise.

Those who decide with criteria no longer argue about vendors. They argue about category, and they can name the decision they made: build the layer, buy the service, or route through a model router. The math decides. The third option is the one that removes the routing decision from code and moves it into the AI infrastructure, where it belongs.

The Nexforce Router is the answer to the third option: an LLM gateway and router with intent classification, automatic failover, per-key cost governance, complete observability, and billing in reais with an invoice, at up to 50% savings on per-token cost. The evaluation begins with a single criterion every other one hides: how much, in reais, the wrong choice costs. Whoever runs that math before the POC has already decided half of it.