AI 101 — What is an LLM router?
An LLM router is a layer that sits between your app and a set of AI models and decides, per request, which model should answer — cheap and small for the easy question, expensive and strong for the hard one. You still have one API to call; the router picks the model behind it.
Why the question is live right now
For two years the default was one model for everything, because there was one model worth using. That is no longer true. A frontier model and a small open-weight model now differ by a wide margin in price per token and by a much narrower margin on most everyday requests — summarise this ticket, classify this email, rewrite this sentence. Paying frontier prices for those is how an AI feature's bill outgrows its revenue.
The second shift is agents. A chat app sends one request per question; an agent sends dozens per task, most of them trivial — checking a tool's output, deciding whether a step finished, choosing which function to call next. Fastino's 340M-parameter classifier is built for exactly that job, returning intent and routing labels in a single pass with no generated text at all, as we reported when it shipped. When a workflow makes fifty calls to do one useful thing, the model choice on calls forty-nine and fifty matters as much as the one on call one.
The one-paragraph mental model
Every request has a difficulty, and every model has a price and a competence ceiling. Routing is the act of estimating that difficulty before you spend the money, then matching it to the cheapest model that will clear it. Some routers are simple rules — if the prompt is under 500 tokens and contains no attachments, use the small model. Some are trained classifiers that read the query and predict which model a human would have preferred, which is what UC Berkeley's LMSYS group did with RouteLLM, an open-source routing framework built on Chatbot Arena preference data: its routers cut cost by 85% on MT-Bench and 45% on MMLU while still reaching about 95% of GPT-4's performance. Others are "cascades": try the cheap model, check the answer against a confidence signal, and only escalate to the big one when the check fails.
A concrete analogy
Think of a hospital triage nurse. The nurse does not treat anyone. She looks at who walked in, asks two or three questions, and sends each patient to the right level of care: a sprained ankle to the walk-in clinic, chest pain straight to a cardiologist. Get the triage right and the hospital runs on a fraction of the specialist hours. Get it wrong in the cautious direction and you waste consultants on paper cuts; get it wrong in the other direction and someone with a real problem waits behind a queue of minor cases.
A router is that nurse for tokens, with one difference that matters: the nurse can see the patient. A router usually sees only the text of the request, which is why the hard part of routing is not choosing a model — it is estimating difficulty from the prompt alone, before any answer exists. Model quantization and running models locally instead of through a cloud API change the price side of that calculation; routing changes which of those prices you pay.
Where routers actually run
Three placements, not exclusive. Inside your own code, as a function that picks a model name before the call — the cheapest option, and the one to try first, since it adds no vendor. Inside a gateway: a proxy that presents one OpenAI-compatible endpoint and fans out to many providers, with OpenRouter the best-known hosted example. Or inside the provider's own platform, which is the least control and the least work.
Whichever you pick, the router needs a fallback path. One that picks a rate-limited, deprecated or down model has made the request worse, not cheaper — so try the chosen model, and on failure or a failed quality check, fall back to the strongest model in the pool.
Common misconceptions
"It's a load balancer." A load balancer spreads traffic evenly for throughput. A router sends traffic unevenly on purpose: 90% of requests to the cheap model is the goal, not a bug.
"Routing makes the model smarter." It cannot. The best answer a router can produce is the answer of the best model in its pool — routing only avoids paying for that model when a cheaper one would do.
"It always saves money." Misroutes cost double: the wrong model's answer plus the retry. And cascades pay for the cheap attempt every time, so a pool where most requests are genuinely hard can cost more than sending everything to the big model. The savings figure depends entirely on your traffic mix, which is why you measure it on your own requests rather than trusting a benchmark number — our guide to cutting an LLM bill without switching models puts routing last for exactly this reason: caching, batching and shorter context usually move the number more.
"Routers are always machine learning." Swap the order of two paragraphs and a keyword rule covers a surprising share of real traffic. Start with rules, measure the misroute rate, and only train a classifier when the rules visibly fail.
"Set it and forget it." Router quality drifts. Prices change, models get deprecated, and a classifier trained on last year's preference data is judging this year's models. Re-check the threshold whenever the model pool changes.
Where to learn more
Read the RouteLLM write-up for a worked example of training a router on preference data, then look at the routing layer of whatever gateway you already use — most expose a fallback list and a cost cap before they expose anything cleverer. If you are still deciding whether to call a hosted API at all, start with what a token is, because every routing decision is ultimately a bet on how many of them the answer is worth.
Would you trust a router to decide when your request deserves the expensive model? Write to us via our contact page.
