What Is a Routing Policy?

What Is a Routing Policy?

A routing policy is the configuration object inside an LLM gateway that defines how a request is mapped to a provider and model. It bundles a strategy (round-robin, weighted, least-latency, cost-optimized, or conditional), a list of targets, an optional condition tree, fallback chains, and timeouts into one named resource. The router evaluates the policy on every request. The policy is the configuration; the strategy is the algorithm. FutureAGI’s Agent Command Center exposes routing policies as a first-class CRUD resource on the gateway control plane.

Why it matters in production LLM/agent systems

Hard-coding routing in YAML works for a single deployment. As soon as a team has staging vs. production, multiple models, A/B tests, and per-team SLAs, routing has to be configurable at runtime. The pain shows up in three ways:

• Deploy-coupled changes. Every routing tweak. shifting weight from openai-east to openai-west, adding a new model fallback. turns into a config-PR, deploy, restart cycle.
• Untracked drift. Without a versioned policy object, “why is this user getting Claude all of a sudden?” becomes archaeology across git history.
• No multi-tenant isolation. A platform team that wants to give Team A 100% Anthropic and Team B 100% OpenAI ends up writing one big conditional rule that nobody can read.

A first-class routing-policy resource fixes all three. Policies are named, versioned, and updatable via API. Each request carries the policy_id it was routed by, so dashboards and traces explain themselves. For agent systems where one task triggers many model calls, policies are how a platform team prevents one team’s runaway agent from starving another’s.

We’ve found that the second-order win is regression evidence. When a policy update lands and cost-per-trace shifts, a versioned policy_id lets the on-call engineer pinpoint which routing decision changed and roll back without touching application code. In our 2026 evals, teams running first-class routing-policy resources cut routing-related incident MTTR by roughly 60% compared with teams reading raw YAML in git history.

How FutureAGI handles it

FutureAGI’s routing-policies resource is exposed in the Python and TypeScript SDKs and as REST on the Agent Command Center control plane. The shape:

from agentcc import AgentCommandCenter

client = AgentCommandCenter()
policy = client.routing_policies.create(
    name="prod-chat-routing",
    strategy="cost-optimized",
    config={
        "targets": {
            "gpt-4o": [
                {"provider": "openai-east", "weight": 50},
                {"provider": "openai-west", "weight": 50},
            ],
        },
        "model_fallbacks": {
            "gpt-4o": ["claude-sonnet-4", "gemini-2.0-pro"],
        },
        "conditional_routes": [
            {
                "name": "enterprise-tier",
                "priority": 10,
                "condition": {"field": "metadata.tier", "op": "$eq", "value": "enterprise"},
                "action": {"provider": "openai-dedicated"},
            },
        ],
    },
)

Internally, the gateway compiles the policy into a routing strategy plus a conditional router that evaluates $eq, $ne, $in, $nin, $regex, $gt, $lt, $gte, $lte, and $exists operators with $and, $or, $not combinators. Fields available on every request: model, user, stream, provider, session_id, request_id, metadata.<key>. Routes sort by priority (lowest first), and the first match wins.

The OTel span emitted by the router carries agentcc.routing.policy_id, agentcc.routing.strategy, and agentcc.routing.target, joining the rest of the traceAI tree. Unlike Portkey’s “configs”. which flatten policy and strategy into one blob. Agent Command Center keeps the policy object versioned and the strategy a small set of named primitives, so changes are auditable. FutureAGI’s control plane stores the full revision history.

How to measure or detect it

Track routing-policy health with:

• Policy-version drift. number of active policies vs. policies referenced in the last 24h. Stale policies should be pruned.
• Conditional-rule match rate. per named rule. A rule with 0 matches in a week is probably dead.
• Target healthy ratio. fraction of targets passing circuit-breaker checks under each policy.
• Per-policy cost. cost_usd rolled up by agentcc.routing.policy_id to confirm a cost-optimized policy is actually optimising.
• Per-policy p99 latency. confirms a least-latency policy is converging on the fastest providers.

# Update a policy at runtime: no gateway restart.
client.routing_policies.update(policy.id, config={...new shape...})

The same policy ID flows into FutureAGI’s evaluation surface, so regression evals can be filtered by the policy that generated their traces.

Common mistakes

• Treating routing-policy and routing-strategy as synonyms. The policy is the object; the strategy is one field on it.
• Putting too much logic in conditional rules. Past 10 rules, refactor into named policies and switch policies via metadata.
• Editing YAML in production instead of using the routing-policies API. YAML is for bootstrap; API is for runtime.
• Using a single policy for all teams. Multi-tenant systems benefit from per-team policies and a metadata.team conditional rule to pick between them.
• Forgetting that priority is ascending. priority: 10 runs before priority: 50.