Understanding the Router Pattern in LLM Applications
If you've worked with LLM-based applications that contain workflows, multiple agents, or tool-calling systems, you've probably come across the router pattern.
At a high level, routing is simply the process of answering:
"Given this request, where should it go next?"
The interesting part is that where it should go can mean different things.
Sometimes the router decides which LLM should handle the request. In other systems, it decides which agent, workflow, or tool should handle it.
These may look similar from the outside, but they solve different problems.
1. Model Routing
The first common use of routing is choosing the model that should handle a request.
Instead of sending every request to the same model:
User
↓
LLM
↓
Response
we introduce a routing layer:
┌──→ Small / Fast Model
│
User → Router ────┼──→ General Model
│
└──→ Reasoning / Specialized Model
The router examines the request and determines which model is most appropriate.
Why would we need this?
Different models have different trade-offs.
A powerful reasoning model might produce better results, but it could also be more expensive or slower. Sending every simple request to that model would therefore be wasteful.
For example:
"What is the capital of France?"
↓
Small / cheap model
"Summarize this 500-word email."
↓
Fast general-purpose model
"Debug this distributed systems problem."
↓
More capable reasoning model
The objective isn't necessarily to always choose the best model.
Instead, the objective is often to choose the best model for the particular request and constraints.
What Can a Model Router Consider?
A router can make its decision using many different signals.
Task Complexity
Simple requests can go to smaller models, while complex reasoning tasks can be sent to more capable models.
Simple task → Small model
Medium task → General model
Complex task → Reasoning model
Latency
Sometimes the most important requirement is speed.
If a user is interacting with a real-time voice agent, waiting several seconds for a response may not be acceptable.
The router could therefore prefer a faster model even if another model is slightly more capable.
Cost
For applications operating at large scale, model cost becomes important.
If thousands of simple requests are being processed every day, routing those requests to an expensive model can significantly increase the infrastructure bill.
A router can use a cheaper model for simpler tasks and reserve expensive models for requests that actually require them.
Model Capabilities
Different models may support different capabilities.
For example:
Text → General LLM
Image → Vision-capable model
Long document → Long-context model
Complex reasoning → Reasoning model
The router doesn't necessarily need an LLM to make these decisions. Some of them can be determined through straightforward application logic.
Availability and Failures
Routing can also be useful when a model becomes unavailable.
For example:
Request
↓
Primary Model
↓
Failure
↓
Fallback Model
↓
Response
This is closer to failover routing, but it uses the same basic idea: determine where the request should go based on the current state of the system.
Rate Limits and Quotas
A production system may also need to consider provider rate limits or internal quotas.
For example:
Model A → Rate limit reached
↓
Router
↓
Model B
So model routing isn't simply about asking:
"Which model is smarter?"
It can be a decision involving quality, cost, latency, capability, availability, and operational constraints.
User-Selected Models vs Dynamic Routing
You may have seen applications where the UI contains a dropdown:
Choose model:
[ GPT-5 ▼ ]
The user selects the model and the application sends the request there.
This is model selection, but I wouldn't necessarily call it dynamic model routing.
The important distinction is who makes the decision.
With user selection:
User → Select Model → Application → Model
With dynamic routing:
User → Application → Router → Model
In the second case, the system makes the decision automatically.
2. Intent and Task Routing
Another very common use of the router pattern is deciding which workflow, agent, or tool should handle a request.
Here, we aren't primarily choosing between models.
We're choosing between different paths through the application.
For example, imagine a customer-support system:
User
↓
Router
↓
┌────────────┼────────────┐
↓ ↓ ↓
Billing Technical Sales
Agent Agent Agent
A user might say:
"I was charged twice for my subscription."
The router identifies the relevant category and sends the request to the billing workflow.
Another user might say:
"The application keeps returning a 500 error."
That request could be routed to the technical-support workflow.
The important distinction is:
Model routing asks:
Which model should process this?
Task/intent routing asks:
Which part of the application should process this?
Intent Routing Doesn't Have to Mean LLM Classification
When people first learn about intent routing, they often assume it means writing a prompt like:
Classify the user's request into one of these categories:
- billing
- technical_support
- sales
- general
Return only the category.
And then using an LLM to make the decision.
This is certainly one approach, but it isn't the only one.
The router pattern describes the architectural decision, not the specific technology used to implement that decision.
There are several ways to build a router.
Rule-Based Routing
For predictable cases, simple rules can be enough.
if request contains "refund":
→ billing
if request contains "password":
→ account_support
This approach is extremely fast, cheap, and predictable.
It works particularly well when the routing conditions are deterministic.
The downside is that rules become difficult to maintain when the number of possible cases grows or when language becomes more ambiguous.
Embedding-Based Routing
Another approach is to use embeddings.
Instead of asking an LLM to classify the request directly, you can create representations for your possible intents or workflows.
For example:
Billing
"refund, payment, invoice, charged, subscription"
Technical Support
"bug, error, crash, API, server"
Sales
"pricing, plans, enterprise, purchase"
These descriptions can be embedded and stored.
When a new request arrives:
User request
↓
Embedding model
↓
Vector
↓
Similarity search
↓
Closest intent
↓
Workflow
For example:
"Why did I get charged twice?"
might be semantically closest to the Billing examples.
Embedding-based routing can be useful when you have many examples and want semantic matching without running a larger LLM for every request.
However, it's important to remember that similarity is not the same thing as true intent understanding.
The quality of the routing depends heavily on how the examples, descriptions, embeddings, and thresholds are designed.
Traditional Machine Learning Classifiers
You can also train a dedicated classifier.
For example:
User Request
↓
Text Classifier
↓
Billing / Sales / Support / Other
This could be a relatively small model trained specifically on your application's data.
This approach can make sense when:
- You have enough labeled examples
- The intent categories are stable
- Low latency is important
- You want predictable inference costs
- You don't need the general reasoning capabilities of an LLM
In some production systems, a specialized classifier can be a better solution than using an LLM for every routing decision.
LLM-Based Routing
Of course, you can also use an LLM as the router.
For example:
User Request
↓
Small LLM
↓
Structured Classification
↓
┌─────────┬─────────┬──────────┐
↓ ↓ ↓
Billing Sales Technical
The LLM can consider more nuanced context and make decisions that would be difficult to encode with simple rules.
You can also provide examples, descriptions, constraints, and structured output schemas.
But there is an obvious trade-off:
You're now spending an LLM call just to decide what should happen next.
For high-volume systems, that additional latency and cost may matter.
Routers Can Be Composed
The interesting part is that these patterns don't have to exist independently.
A production AI system might have multiple routing layers.
For example:
User
↓
Intent Router
↓
Technical Support
↓
Agent / Workflow
↓
Model Router
↓
┌────────────┼────────────┐
↓ ↓ ↓
Small Model General Model Reasoning
The first router determines what the user is trying to accomplish.
The second router determines which model is appropriate for executing that task.
This separation can make the architecture much easier to reason about.
Router vs Orchestrator
Another distinction worth understanding is the difference between a router and an orchestrator.
A router generally answers:
"Where should this request go?"
An orchestrator generally answers:
"What should happen, and in what order?"
For example:
Router
User
↓
Research Agent
Orchestrator
User
↓
Research Agent
↓
Search Web
↓
Analyze Results
↓
Write Report
↓
Review
↓
Final Answer
In real systems, the boundaries aren't always perfectly clean. A component can perform both routing and orchestration.
But conceptually, separating these responsibilities is useful when designing agentic systems.
The Bigger Idea
The most important thing to understand about the router pattern is that routing is an architectural concept, not an LLM technique.
You can implement routing using:
- Rules
- Traditional classifiers
- Embeddings
- LLMs
- Model metadata
- Application state
- Availability information
- A combination of several approaches
And you can route to:
- Different LLMs
- Different agents
- Different workflows
- Different tools
- Different RAG pipelines
- Human reviewers
- Fallback systems
The fundamental pattern remains the same:
Request
↓
Router
↓
┌──────────┼──────────┐
↓ ↓ ↓
Path A Path B Path C
The engineering challenge is deciding what information the router should use, how reliable the decision needs to be, and what happens when the router gets it wrong.
And that's where routing becomes particularly interesting in agentic AI: as systems become more complex, the question isn't just:
"How do I make an agent smarter?"
It's also:
"How do I make sure the right piece of the system handles the right problem?"