Highlights
This blog explores a challenge many organizations are beginning to face: building an impressive agentic AI pilot is one thing, but scaling it economically is another. Swaraj Kokare describes that the real barrier to production adoption is often not AI capability, but architecture. By using Small Language Models (SLMs) for high-volume, repeatable tasks such as routing, tool selection, and data extraction, while reserving larger frontier models for complex reasoning, enterprises can significantly reduce costs, improve latency, and maintain performance. Through industry research and practical implementation guidance, the blog makes a compelling case for an SLM-first approach to improving agentic AI economics and helping more AI initiatives successfully transition from pilot to production.
I’ve watched enough agentic AI initiatives move from demo to dead-end to know the pattern by heart. The demo works. Everyone in the room is impressed. Then someone runs the cost projection at production volume, and the project quietly stops.
This is rarely a capability problem. More often, it’s an architecture problem that shows up first in unit economics. Enterprises are wiring every step of an agent’s reasoning loop — parse the request, pick a tool, format the output, validate the result — through a frontier model built for general conversation. That’s the equivalent of hiring a senior architect to file every support ticket. The work gets done. The bill doesn’t make sense.
The industry has a name for the fix: small language models, or SLMs, deployed as the default execution layer inside agentic systems, with frontier models reserved for the narrow slice of work that needs them.
This blog walks through why that shift matters for agentic AI economics, what the evidence shows, what an actual migration looked like, what it costs, and what a Monday-morning path looks like for teams stuck in pilot purgatory.
Let’s start off.
Why Most Agentic AI Pilots Never Reach Production
Gartner has forecast that over 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating cost, unclear value, and inadequate risk controls as the dominant reasons. It’s a formal, published forecast, and it’s the number I’d lead with. A separate, much-cited figure from MIT’s NANDA initiative puts pilot failure even higher, at roughly 95% — but that report is preliminary, hasn’t been peer-reviewed, and its sample size has drawn pushback on the precision of the exact figure. Treat it as directionally consistent with Gartner, not as the headline stat.
I’ve sat in enough steering committee reviews to recognize the trajectory both numbers describe. A pilot ships against a narrow, well-scoped workflow. It works. Leadership approves scaling. Then the agent starts running against production volume — thousands of invocations a day instead of dozens — and the token bill and latency profile both move in the wrong direction at the same time. The team that owns the budget starts asking questions the engineering team wasn’t ready to answer.
The instinct is usually to blame the model. In my experience, that’s rarely where the problem lives. The real issue is an assumption baked into the architecture: that every step of an agent’s loop deserves the same model as its hardest step.
The Architecture Mismatch
Frontier LLMs are optimized for broad, general-purpose reasoning — holding context across a long conversation, handling ambiguous or novel requests, producing coherent prose. Most of what an agent does in production doesn’t need any of that.
Consider a typical agent loop: read an instruction, decide which tool to call, call it, parse the response, decide the next step, repeat. Researchers at NVIDIA, joined by a Georgia Tech co-author, make the case directly in their June 2025 position paper, “Small Language Models are the Future of Agentic AI”: agentic systems perform a small number of specialized, repetitive tasks with little variation, and that profile favors small, narrowly tuned models over general-purpose giants. Their analysis puts the inference-efficiency gap at roughly 10 to 30 times, comparing a 7-billion-parameter SLM against a 70- to 175-billion-parameter LLM across latency, energy, and compute.
That’s not a marginal efficiency gain — it changes whether an agent architecture survives contact with production volume at all.
What SLMs Actually Solve And What They Don’t
The SLM argument gets oversold in some corners of the industry, and overclaiming erodes trust faster than underdelivering, so it’s worth being precise.
Where SLMs win: tool-calling and structured output, where a model resolves a natural-language instruction to a function call and returns valid JSON — narrow, learnable work that a fine-tuned 3B–9B model can handle at near-perfect accuracy, sometimes outperforming a zero-shot frontier model on the same task. Salesforce’s 8-billion-parameter xLAM-2 has topped tool-calling benchmarks ahead of GPT-4o and Claude 3.5 Sonnet. Routing decisions — which specialist agent handles a request — are classification problems, not reasoning problems. So is field extraction from semi-structured input like invoices or tickets, where Microsoft’s Phi-3-small, at 7 billion parameters, has matched language and code scores from models ten times its size.
Where frontier models still earn their keep: genuinely open-ended reasoning on novel or ambiguous problems, long-context synthesis across large documents or conversation histories, and low-frequency, high-stakes decisions — the final approval step in a workflow, or a judgment call with real business consequence — where you want the most capable model even at higher cost per call, because the call is rare and the cost of getting it wrong is high.
The NVIDIA researchers are explicit that where general-purpose conversational ability is genuinely required, heterogeneous systems — agents that invoke different models depending on the step — are the natural architecture, not a compromise.
A Real Migration, Not a Hypothetical One
Most of the SLM case for agentic AI still leans on projected savings rather than a company that actually made the switch and published results. One example closes that gap. Convirza, a call-center analytics company, moved its agent stack off OpenAI onto a fine-tuned Llama-3-8B model, serving more than 60 specialized LoRA adapters off a single base model through Predibase. The company reported a roughly tenfold cost reduction, an 8-point F1 improvement over its previous setup, and 80% higher throughput, with sub-second inference across millions of call-hours. The result shows up consistently across Predibase’s own case study, Meta’s official Llama community write-up, and an independent LLMOps benchmark database — not a single vendor’s marketing page.
A smaller, similar data point: Checkr, in background-check classification work, reported beating GPT-4 with a fine-tuned Llama-3-8B model that ran roughly 30 times faster and 5 times cheaper, as part of Predibase’s “LoRA Land” benchmark series. Neither example is a universal guarantee — both are narrow, high-volume classification and extraction tasks, exactly the profile where SLMs are strongest — but they move the argument from theory to a documented outcome.
You might also like this blog: SLM vs LLM Cost: Understanding AI Inference, Infrastructure, and Total Cost of Ownership – Nitor Infotech Blog
Latency Is the Other Half of the Story
Cost gets most of the attention, but latency is what determines whether an agent architecture is usable at all. An SLM inference step typically runs in the tens of milliseconds; a frontier-model step commonly runs in the hundreds. That difference barely registers in a single-turn chatbot. It’s decisive in an agent loop that chains ten steps together before returning a result — the gap compounds with every hop, and a workflow that would take 30 seconds end to end on a frontier model can complete in a couple of seconds when the narrow steps run on an SLM. For anything customer-facing or time-boxed — a live support interaction, a real-time fraud check — that’s the difference between an architecture that ships and one that gets shelved for being too slow, independent of what it costs per call.
Where Self-Hosting Changes the Calculus
Cost isn’t the only reason enterprises move to SLMs. Because SLMs are small enough to self-host, they enable a deployment pattern a frontier-model API simply can’t offer: keeping every request, response, and fine-tuning dataset inside your own infrastructure, with no data ever crossing to a third-party API. For healthcare, finance, and government buyers, that’s often a bigger driver than the token bill — it’s the difference between an architecture that clears a compliance review and one that doesn’t get considered.
Agentic AI Economics: The Math, Worked Through
Here’s a worked example, built on the paper’s 10-to-30x efficiency range and published API pricing as of September 2026. Treat the exact figures as illustrative — token prices move constantly, and your own step count and hosting choice will shift the number — but the shape of the result holds.
Take an ordinary 10-step agent loop: parse the request, classify intent, call two or three tools, validate each response, format the output, handle a retry, escalate when needed. Assume each step averages around 500 input tokens and 200 output tokens.
Run all 10 steps through GPT-4o, priced at roughly $2.50 per million input tokens and $10 per million output tokens, and the full loop costs about $0.0325 per completed task. Worth flagging: GPT-4o sits toward the cheaper end of frontier pricing — GPT-4.1, GPT-5, and Claude Sonnet are all priced differently — so this comparison is conservative, not a worst case chosen to flatter the argument.
Now route the 7 of those 10 steps that are narrow and repeatable — tool selection, structured extraction, routing — to a fine-tuned Llama-3.1-8B-class model hosted on Together AI, priced at roughly $0.18 per million tokens combined. Those 7 steps now cost under a tenth of a cent in total. The remaining 3 steps, which genuinely need open-ended reasoning or carry real business stakes, stay on GPT-4o.
Total hybrid cost per task: roughly $0.0106 — about a 67% reduction from the all-frontier baseline. At 5,000 completed tasks a day, a modest volume for a live enterprise agent, that’s the difference between roughly $4,875 a month and $1,595 a month — over $40,000 a year for a single moderate-volume workflow, and most enterprises running agentic AI aren’t running just one.
That savings isn’t free to capture. You’re trading a single frontier-model API key for a small fleet of fine-tuned models whose lifecycle you now own: model drift, retraining cadence, and a misrouted request that quietly degrades quality instead of failing loudly. None of that is a reason to skip the migration — the Convirza numbers show the economics still hold — but budget engineering time and MLOps maturity into the plan, not just the GPU-hours you expect to save.
Building the SLM-First Agent: A Practical Pattern
None of this works as a blanket model swap. Here’s the operating pattern I’d stand behind in a production review.
- Start by instrumenting the existing or planned agent loop — log every model invocation, its input, output, cost, and latency. You can’t right-size a system you haven’t measured. Then classify each step by complexity, not by habit: open-ended reasoning, or a bounded, repeatable task with a learnable pattern? Most steps in most production agent loops fall into the second category.
- Fine-tune SLMs on real workflow traces, not generic prompts — a small model given “think step by step” instructions underperforms, while one fine-tuned on your actual step decomposition and output format does not. Build a router rather than a rule book, and evaluate it on its own: routing accuracy is a distinct metric from task-completion accuracy, and a router that misroutes even a small share of requests will surface as silent quality degradation rather than a clean failure.
- Evaluate every fine-tuned agent in isolation against a held-out set, using a metric matched to its job — routing accuracy for the router, structured-field accuracy for extraction — and then evaluate the assembled workflow end to end, because per-agent accuracy doesn’t guarantee the handoffs between agents behave correctly. Finally, keep an explicit escalation path from every step back to a larger model when confidence is low or the input falls outside the SLM’s trained distribution. The goal isn’t eliminating frontier calls; it’s making every one of them deliberate.
What I’d Do on Monday Morning
I’d pull the invocation logs from whatever agentic pilot is closest to a scale decision and sort every call by frequency and by whether it actually required open-ended reasoning. In most production agent loops I’ve looked at, that sort alone shows that 70–80% of invocations are narrow, bounded tasks that never needed a frontier model in the first place — a range that lines up with the NVIDIA paper’s own case-study estimates, which put the share of replaceable calls at roughly 40% for the workflow-automation agent Open Operator, 60% for the multi-agent coding framework MetaGPT, and 70% for the GUI-control agent Cradle.
Then I’d pick the single highest-frequency step — routing or tool-selection, usually — and fine-tune a small model against it using real trace data. I’d build one eval suite for that agent in isolation, get it passing, and only then wire it back into the full workflow with an end-to-end eval running alongside it.
I would not attempt a full-stack migration in one pass. The teams that get this right treat it as an incremental, measured swap-out — one high-frequency step at a time, each validated before the next — governed the same way you’d govern any other production infrastructure change, using a lifecycle framework like Nitor Infotech’s ADLC (Agentic Development Lifecycle), rather than as a one-off cost-cutting experiment.
This is the pattern the section above builds toward: a router classifies each incoming step, hands high-frequency bounded work to a tier of fine-tuned SLMs, reserves the frontier tier for open-ended reasoning and high-stakes calls, and routes everything through an evaluation layer before it becomes a response — with an explicit escalation path back to the frontier tier whenever an SLM’s confidence is too low to trust.
Take a look at the diagram I have shared below:
Fig: The heterogeneous SLM-first agent architecture
A topic around Agentic AI economics can be compared to a conversation in progress, so stay tuned for more blogs expounding on these ideas. Also do write to me with your thoughts about this blog.
In The Works
We’re getting set to release an informative report centered on the state of AI economics.
This report looks at what’s really driving AI spend, how those costs change as usage grows, and what it takes to turn AI investment into sustainable business value.
Watch this space to get your copy of the report.
Contact us at Nitor Infotech to learn more about our evolving takes on Agentic AI.
