How to Cut LLM Inference Costs: 5 Engineering Techniques

The inference bill for a production system follows a predictable arc: it doubles before anyone notices it could have been cut in half. Five engineering techniques can reduce LLM inference costs by 30% to 70% without switching models. None of them requires rewriting the application. Most teams running LLMs in production are not using three of these five. What separates the bill that alarms from the bill that fits the budget is less about technology and more about the order in which the techniques reach production.

Prerequisites: What You Need Before You Start

Three things are required before step one. First, access to the APIs of the models your application consumes, or a self-hosted infrastructure with available GPUs. Without visibility into real traffic, any optimization is a bet. Second, an observability tool that logs input and output tokens per request, latency, and the model called. Langfuse, OpenLIT, and the Nexforce Router consumption dashboard all serve this function. Third, a cost baseline measured in dollars.

For companies operating in Brazil, that baseline must also exist in Brazilian reais. Brazilian enterprises pay a multiplier of up to 55% on every dollar of inference cost because of the chain of IRRF, CIDE, PIS, COFINS, ISS, IOF, and FX spread that hits every international remittance. Without that BRL number, the real savings from each technique stay undercounted.

The Nexforce Router invoices in reais with a local tax document and taxes included, eliminating the multiplier at the source and giving the team a single baseline.

The dollar number is half the story. Finance sees the local currency.

Step 1: Diagnose Where the Money Is Going

Before applying any technique, the team needs to know what share of the inference bill comes from input tokens, what share from output tokens, and what fraction of requests are repeated or nearly repeated. Most implementations discover that between 40% and 60% of input token volume is redundant: long prompts with identical system instructions, conversation context resent on every turn, RAG documents pulled in full when the model needed two paragraphs.

The diagnosis answers three questions. First: what is the ratio of input tokens to output tokens? Models charge more for output, so an application that generates long responses has a different cost profile from one that processes large documents with short output. Second: what is the request repetition rate? The higher it is, the more semantic caching delivers. Third: is the current model oversized for the actual task complexity? The answer is almost always yes. A frontier model called to classify user intent is a Town Car delivering flyers.

Tools like Langfuse and OpenLIT provide this breakdown per request. The Nexforce Router dashboard delivers the aggregate view by API key and by project, with cost already expressed in your local currency, eliminating the separate step of converting a dollar invoice into the currency finance recognizes. The LLM Benchmark for CFOs covers the financial evaluation angle; here the focus is instrumentation for engineering.

Step 2: Semantic Caching: The Savings That Live in Repetition

Semantic caching is the highest return on effort technique for most applications. Instead of sending a request to the model, the system checks whether a response for a semantically similar prompt already exists in cache. If it does and falls within the similarity threshold, the response is served with zero inference cost.

The most common implementation uses Redis as storage and an embeddings model to generate the vector for the input prompt. The system queries Redis by cosine similarity against cached prompts. If the distance falls below the threshold (typically between 0.05 and 0.15, depending on the application), the cached response is returned. The Nexforce Router offers response caching and embeddings as native functionality, without requiring the team to stand up Redis infrastructure and the embeddings pipeline.

Typical savings range from 30% to 70% of calls, but depend directly on traffic profile. Applications with high volumes of similar requests (support chatbots, documentation assistants, ticket classification systems) gain the most. Applications with highly variable prompts, such as custom code generation or unstructured data analysis, have low hit rates and the cache does not pay for itself.

The main trade-off is the risk of stale responses. Responses age quickly. If the underlying model was updated or the business context changed, the cache serves an outdated answer. The mitigation is an aggressive TTL (between 1 and 24 hours, set by experiment) combined with manual invalidation for sensitive domains.

Step 3: Prompt Compression: Less Context, Same Answer

If semantic caching attacks repetition across requests, prompt compression attacks the volume inside each request. The technique reduces input token count without losing the information the model needs to answer.

The most common target is context injected by RAG systems. A typical pipeline retrieves five documents from the vector database, concatenates the full texts, and sends everything to the model. Of those five, two are relevant and three are noise, but the model paid for all of them. Tools like LLMLingua and prompt pruning via smaller models apply selective compression: they remove low-perplexity tokens, cut paragraphs with low similarity to the query, and preserve named entities, numbers, and technical terms.

Typical savings reach 50% to 70% reduction in input tokens for RAG pipelines. In applications that do not inject external context, such as a code assistant that receives only the current file, compression has little effect and does not justify the additional latency of the compressor.

Compression is not free. Every request passes through a compressor model before reaching the main LLM.

The trade-off sits at the boundary between compression and precision loss. For complex reasoning tasks, aggressive context cutting can remove the nuance the model would have used to reach the correct answer. The recommendation: apply prompt compression first to extraction and summarization tasks, where the relevant information is localized, and test with an evaluation set before enabling it on analytical tasks.

A single paragraph stops here.

Step 4: Quantization and Self-Hosting: The Structural Path

Quantization delivers the highest absolute savings, but also demands the most infrastructure. It reduces the numerical precision of model weights (from FP16 to INT8 or INT4), cutting memory usage and accelerating inference. For teams running their own models on GPUs, savings can reach up to 80% compared to the equivalent API cost.

Tools like vLLM with AWQ (Activation-aware Weight Quantization) deliver INT4 inference with quality loss under 1% on standard benchmarks for most open-source models. Llama 3.1 70B quantized to INT4 fits on a single A100 GPU and delivers throughput comparable to double the model in FP16, at the same hardware cost.

The self-hosting decision is not purely technical. It involves acquiring or renting GPUs, maintaining infrastructure, and owning uptime and latency responsibility. For low volumes, below roughly 50 million tokens per month, the cost of idle GPUs exceeds the savings from not paying the API. For high and predictable volumes, above 200 million tokens per month, self-hosting pays for itself within weeks.

The Nexforce Router serves as a routing layer over self-hosted models and API models with the same key: the team starts via API, gradually migrates to self-hosting for the highest-volume models, and keeps the API as fallback. The switch is transparent to the application.

Step 5: Intelligent Routing: The Right Model for Every Request

Intelligent routing operates at the decision layer: for each request, the system chooses which model handles that call based on complexity, required latency, and cost. It is the technique that captures the savings the other four leave behind, because it acts on requests that are not repeated, not compressible, and do not justify self-hosting.

It is the most underestimated of the five.

The typical architecture classifies each request by intent and complexity. A sentiment classification or entity extraction goes to a lightweight, cheap model. A legal analysis or complex report generation goes to a frontier model. The differentiator is the classifier: if the system sends 5% of simple requests to the expensive model, savings drop from 60% to 45%.

Typical savings range from 40% to 60% of the inference bill for applications with heterogeneous traffic profiles, which describes most real-world applications. When all traffic is complex, routing has nowhere to divert and savings are zero. When it is predominantly simple, gains can exceed 70%.

The Nexforce Router implements intelligent routing as native functionality: intent classification, model selection by cost and performance, and automatic failover between providers. The team defines rules per API key. For instance, classification requests go to a lightweight model, reasoning requests go to a frontier model, and the budget is controlled by a spending cap per key. The post Model Router: The AI Middleware Your Stack Is Missing details the architecture; the Enterprise AI Gateway covers the security and governance layer.

How to Know It Worked: Impact Verification

The impact of each technique is measured against the baseline established in Step 1. Four metrics matter. Total cost per period, compared to the baseline in dollars and in the relevant local currency. Semantic cache hit rate, expressed as the percentage of requests served without inference. Request distribution by model tier, showing the migration from the expensive model to the cheap one over time. Input token reduction after prompt compression, measured as percentage delta over the baseline.

A team that applies the five techniques in the correct order (diagnosis, caching, compression, routing, and self-hosting only if the volume justifies it) typically sees a 50% to 70% reduction in the total bill within 90 days. The gain is not linear: the first two techniques deliver 60% of the savings with 20% of the effort.

Decision Table: Which Technique to Apply First

The order of application matters more than the choice of techniques. The table below organizes the five techniques by workload profile and estimated savings, so the team can decide where to start.

The recommendation for most teams: start with diagnosis, apply semantic caching and prompt compression in parallel, enable intelligent routing next, and evaluate self-hosting only when monthly volume justifies the hardware investment. The first three techniques require no additional infrastructure and deliver the bulk of the savings.

Start with what is free. Then invest.

What Can Go Wrong (and How to Fix It)

Three failure modes appear consistently when teams implement these techniques for the first time.

First: the cache serving stale responses. The symptom is the user receiving an answer that references an old API version or a price that changed the previous week. The fix is to reduce the cache TTL and implement event-based invalidation: whenever the knowledge domain changes (a new documentation version, a pricing policy update), the cache is selectively cleared by prompt pattern.

The second is prompt compression degrading quality on reasoning tasks. The symptom is the model delivering correct but incomplete answers, because the nuance that connected two paragraphs was removed by the compressor. The fix is to apply compression only to extraction and summarization tasks, and to use a more conservative similarity threshold on the compressor (above 0.85) for analytical tasks.

The third is the router sending complex requests to simple models. The symptom is the lightweight model responding with hallucinations or refusals on tasks that require multi-step reasoning. The fix is to calibrate the intent classifier with a representative set of examples and implement an escape rule: if the lightweight model returns low confidence or refuses the answer, the request is re-routed to the frontier model.

Frequently Asked Questions

How long does it take to implement all five techniques?

Between two and six weeks, depending on infrastructure maturity. Semantic caching and prompt compression can be in production within days. Intelligent routing takes one to two weeks if a gateway layer already exists. Self-hosting requires four to eight weeks for provisioning, configuration, and gradual traffic migration.

Which technique delivers the highest return on effort?

Semantic caching for applications with repetitive traffic. Intelligent routing for applications with heterogeneous traffic. The two combined cover most workload profiles and deliver between 50% and 70% savings with moderate effort.

Do I need to apply all five techniques?

No. Most teams achieve 50% to 70% of the possible savings with the first three: caching, compression, and routing. Quantization and self-hosting are for teams whose volume justifies the hardware investment. Continuous batching is relevant only for self-hosted models.

How does the Brazilian tax multiplier affect these calculations?

Every dollar of inference cost paid through international billing carries a multiplier of up to 55% in Brazil. The chain consists of IRRF (15% to 25%), CIDE of 10% on SaaS classified as a technical service (per SC Cosit 191/2017, confirmed by 99/2018, with the exemption under §1°-A of Article 2 of Law 10.168/2000 restricted to pure software licenses without technology transfer), PIS at 1.65%, COFINS at 7.6%, ISS at 2% to 5% depending on the municipality, IOF at 3.5%, and FX spread of 5% to 10%. A 50% savings in dollars through engineering optimization becomes a 50% savings on a base that was already 55% higher in local currency. The Nexforce Router eliminates the multiplier at the source by invoicing in BRL with taxes included.

What is the first thing a team should do tomorrow?

Instrument the application to know exactly how many tokens are being consumed, by which model, and what the request repetition rate is. Without that diagnosis, any technique applied is optimization in the dark. The Nexforce Router dashboard delivers this visibility in minutes, with cost already expressed in the local currency.

The Multiplier That Makes Every Technique Worth More

The LLM Cost Comparison 2026 showed the per-token price of each model. This article showed how to reduce the number of tokens the application consumes. The two multiply.

For companies operating across Latin America, a third factor exists that no engineering technique solves alone: the structural tax multiplier on every dollar spent on inference. In Brazil, IRRF, CIDE, PIS, COFINS, IOF, and FX spread stack up to 55% on top of the token cost. An application spending $100,000 per month on inference disburses the equivalent of up to $155,000 in local currency. Applying semantic caching, prompt compression, and intelligent routing reduces the dollar spend. Invoicing through the Nexforce Router in BRL eliminates the multiplier at the source, because the local tax document already includes the taxes and the FX conversion is resolved, so the savings from the five engineering techniques are calculated against the real cost in the currency finance actually sees.

The five engineering techniques are the how. Local-currency billing is the where. Together, they turn a bill that doubles silently into a bill that fits the budget and that finance can read.

References and Further Reading

• LLM Cost Comparison 2026: Intelligent Routing with Nexforce Router
• LLM Benchmark for CFOs: What Matters Is the Cost
• Model Router: The AI Middleware Your Stack Is Missing
• Enterprise AI Gateway: Routing and Security for LLMs
• Nexforce Router