AI Gateway Architecture Explained Through One Request

A support bot tells a customer the refund window is 30 days. That was last week’s policy, and the team changed it on Monday. When the on-call engineer opens the provider’s usage log, there is no call for that request at all. Something between the app and the model answered for it.

The team had never drawn that layer, so the engineer had no idea where to look. The gateway had stages for keys, checks, caching, routing and logs, and one of them served a stored answer. Reading the AI gateway architecture as one request path is how you find which stage did it.

Below, we follow that support-bot request stage by stage. Each stage gets its latency cost and its failure mode. Then we map the same path onto an open-source gateway, with the plugin order read from its source code rather than from a diagram.

If you want the category first, start with what an AI gateway is.

TL;DR

• The AI gateway architecture reads best as one request path: identity, policy and cache before the model, routing and translation at it, checks, logs and cost after it.
• A virtual key, never the provider key, identifies the caller and carries its model allow list, rate limits and budget.
• Cache placement relative to input guardrails is a design choice: early lookups save work, and late writes keep blocked answers out of the cache.
• Tool calls can take the same gateway over MCP, with per-key tool lists applied on discovery and on each call.
• Agent Command Center, the open-source gateway in Future AGI, is the worked example, with its plugin order read from source.

The AI gateway architecture as one request path

An AI gateway architecture is easiest to understand as a single request moving through ordered stages. The request is identified, checked against policy, looked up in a cache, routed to a provider, translated into that provider’s format, checked again on the way out, then logged and costed. Each stage can stop the request or change it.

Our request is the support bot asking “Where is order 4417?” for a customer. It leaves the client SDK as an OpenAI-shaped chat completion and enters the gateway’s data plane. Pre-path stages run first, then the provider adapter and call, then post-path stages on the way back.

Above the data plane sits the control plane. It holds what the stages read: virtual keys, guardrail policies, routing config and model pricing. The control plane changes when someone edits config, while the data plane runs on every request. Keeping them apart lets you ship a policy without restarting the serving path.

Reading the AI gateway architecture diagram top to bottom

The diagram reads top to bottom, with the control plane on the right. Solid arrows carry the request through the seven stages, and the dashed arrow carries logs and metrics off the request path. The arrow from the cache back to the client is the short-circuit on a hit, and the fork at routing points to fallback targets.

Placing the LLM gateway architecture behind an existing API gateway

Most companies already run an API gateway at the edge for TLS, a firewall, coarse rate limits and single sign-on. The usual layout puts the AI gateway behind it. The AI gateway adds what the edge cannot: token-aware limits, model policy, provider translation and cost.

The two do not duplicate auth if the edge passes identity through and the AI gateway issues its own virtual keys. Some teams run the AI layer as plugins inside the API gateway instead. That works, but provider translation, token counting and cost accounting then become plugin code you maintain.

LLM gateway vs MCP gateway paths through one pipeline

A tool call can enter the same gateway. The agent sends tools/list and tools/call to the gateway’s Model Context Protocol endpoint instead of calling each tool server itself. An agent that sends its virtual key to /mcp gets that key’s identity and tool lists, the same place the LLM path enforces policy.

The key’s allowed_tools and denied_tools lists filter what tools/list returns and gate every tools/call. Optional tool guardrails can check the arguments before the call and, with output validation on, the result after it. The MCP path skips the model-only stages: provider routing, format translation, the response cache and token cost.

For the detail, read the full MCP gateway tool-call flow.

Virtual-key auth decides who is calling

The first thing the gateway decides is identity. The client sends a virtual key that belongs to the gateway, never the provider’s API key. The gateway resolves that key to an owner, a team and a policy set: which models and providers it may call, how many requests and tokens per minute, and how much it may spend.

An IP allow list often runs before the key lookup, so traffic from an unknown network is rejected before any other work. The key lookup itself runs in memory. Role-based access can sit on top of the key, so a team role allows a set of models and the key narrows it further.

Per-key limits come in two units. Requests per minute protect the gateway and the provider from bursts, while tokens per minute protect spend. A budget or credit balance on the key caps spend over a day, a week or a month.

The provider key stays inside the gateway and never reaches application code, laptops or CI logs. When you revoke a virtual key, every gateway replica must learn about it before the next request lands. A revocation that reaches only some replicas leaves the old key working on the rest.

Virtual keys versus provider keys

A provider key is one billing account. It has no per-team limits, no idea which service sent a request, and a leak exposes the whole account until someone rotates it. Every service also breaks together the day you rotate it.

A virtual key is scoped and revocable. It carries metadata such as team and environment, and later stages use that metadata for routing rules, log fields and cost attribution. Our support-bot key allows two models, so a bug that requests a third fails at the gateway with a clear error.

Guardrail and cache checks run before any tokens are spent

Before the gateway spends a single provider token, it runs the cheap checks. Input guardrails scan the prompt for injection attempts, secrets and personal data, and can block, warn, log or mask. The cache then checks for an identical request, or a close enough one, and returns a stored answer without calling the model.

Each guardrail also needs a policy for when the check itself errors or times out. Fail-open lets the request through, which keeps traffic flowing but lets bad input pass. Fail-closed blocks it, which suits a PII check and hurts when a check times out under load.

The exact cache keys on a hash of the model, messages and parameters, while the semantic cache compares the last user message with earlier ones against a similarity threshold. A namespace can isolate exact-cache entries; do not assume it also isolates semantic matches. Streaming requests usually bypass the cache, since the response is sent before it is complete.

For a deeper read on the masking stage, see gateways that redact PII before the provider call.

Exact match and semantic cache lookups

An exact match is the fastest check and costs nothing at the provider. It also misses on any paraphrase, so “Where is order 4417?” and “where’s my order 4417” are two different entries. It never confuses two different questions, because a hit means the gateway saw this exact request before.

A semantic lookup catches paraphrases at the price of a vector per lookup. A loose threshold returns a stored answer to a slightly different question, which is how a refund question can get last week’s policy. Keep TTLs short for answers that change, such as policies and order status, and keep volatile intents out of the semantic cache.

Where the cache sits relative to input guardrails

A lookup placed early lets a hit skip every heavier stage, including model-based guardrails and the provider call. A lookup placed after input guardrails costs more per hit, but a guardrail you add today also applies to answers cached last week. Both placements work; the choice depends on how often your policies change.

The write side has a safer default. Store a response only after the output guardrails pass, so an answer that was blocked or masked is never served from cache to the next caller. Whatever order you choose, write it down, because it decides whether a new rule covers old answers.

Routing and fallback pick a provider, then translation speaks its format

With a miss in the cache, the router chooses where the request goes. It picks a provider by rule, weight, latency, cost or prompt complexity. It retries transient errors with backoff, fails over to the next target on rate limits or server errors, and trips a circuit breaker on a provider that keeps failing.

Strategy families include round robin, weighted splits, least latency, cost based, adaptive weights, racing providers, conditional rules and complexity tiers. A different family, semantic routing, picks a model by embedding the prompt and matching it to example routes. You can pair one strategy with rules on key metadata.

Once a provider is chosen, the adapter rewrites the OpenAI-shaped body into the provider’s format and maps the reply back. For Anthropic that means the Anthropic Messages API, and for others it means Gemini or Bedrock Converse. The client sees one response shape whichever provider answered.

Retries, failover and circuit breakers

These four mechanisms handle different failures. A retry sends the request to the same provider after a transient error. Failover moves it to the next provider in order on a 429, a 5xx or a timeout. A model fallback switches to a different model when one model is unavailable, even at the same provider.

A circuit breaker stops sending traffic to a failing provider for a cooldown, then lets a few trial requests through in a half-open state. Cap total attempts across retries, failover and model fallback. Otherwise a request with a 30 second timeout can spend that budget three times, and an outage turns into a retry storm.

Translating one request into each provider’s format

The adapter has to map more than the message list. System prompt placement differs between providers, as do tool and function schemas, the name of the max tokens field, stop reasons, usage fields and the shape of streaming events. Each mapping needs its own tests.

A translation bug seldom throws an error, because providers change these formats on their own schedules. It shows up as a wrong finish reason, a tool call the client cannot parse, or missing usage numbers. Missing usage then breaks cost accounting further down the path.

The response path runs checks, logging and cost accounting

The provider’s answer does not go straight back to the client. First, output guardrails read the response for leaked data or policy breaks. The cache stores it if it passed, and the gateway then logs the request, computes its cost from token usage and model pricing, and charges it to the key’s budget.

Output checks look for data leakage, content rule breaks and format problems, such as invalid JSON when the client asked for a schema. When a check blocks, the client gets an error or a withheld answer. The provider has already been paid for those tokens, so run cheap checks first.

Cost is input tokens times the input price plus output tokens times the output price, using the pricing table in the control plane. The gateway deducts that amount from the key’s budget or credit balance. Because spend is charged after the response, it is the budget check on the next request that stops a key that has run out.

Logging, metrics and trace export run off the critical path, so a slow log sink never slows the user down. Spans that follow the OpenTelemetry GenAI semantic conventions let one trace backend read requests from every provider. Gateway spans also cover requests that never reached a provider, such as cache hits.

Response headers close the loop for the client. A good gateway reports which provider answered, which model was used, whether the cache hit, how long the request took and what it cost. Those five values answer the support-bot question without a dashboard.

Streaming changes the post path

With streaming, the gateway forwards tokens as they arrive. Output checks run per chunk or after the stream ends, usage arrives with the final chunk, and cost and logging run after the last token. A check that fires late finds part of the answer already on screen.

Fallback changes too. If the gateway fails over after the first chunk has reached the client, the second provider’s stream can repeat the start of the answer. The safer design fails over only before the first chunk and surfaces mid-stream errors to the client instead of retrying them.

What each layer of the AI gateway architecture adds to latency

Most layers cost microseconds. Key lookups, rule checks and exact cache hashes run in memory. The expensive layers are the ones that call something else: a semantic cache embedding, a model-based guardrail, a retry, a fallback to a second provider, or a gateway hosted across a network hop from the app.

Everything from auth to output checks sits between the user and the answer. Cost is a quick in-memory sum before the headers go out, while logging and trace export run off the path. The risk is stacking calls before the provider, such as an embedding, a model-based guardrail and a retry, which together can rival the time to first token.

Where the gateway runs matters as much as what it runs. A gateway in the same region or cluster as the app removes a network hop that no tuning wins back. Its runtime counts too: tight CPU limits or a goroutine scheduler mismatched to a large node’s cores slow it under load.

Logging fails without an alarm. Teams attach the user ID or the prompt text as span or metric attributes, and trace storage grows with every distinct value. Keep high-cardinality values in log bodies, sample traces where volume is high, and keep metric labels few and fixed.

Future AGI Agent Command Center as the worked example

Agent Command Center is the gateway inside the Future AGI platform. It maps onto this architecture one plugin at a time. It speaks OpenAI-compatible /v1/chat/completions and /v1/responses plus Anthropic-native /v1/messages, routes across 100+ providers as of November 2026, caches by exact match or semantic similarity, and serves an MCP gateway at /mcp.

The gateway is a Go service where each stage is a plugin with a priority. Pre-path plugins run in ascending priority, and the first one that returns a response ends the pre-path. The table lists priorities from source at commit 28e729a, unchanged on main as of 2026-11-11.

Exact entries are checked first. The semantic store turns the last user message into a vector and matches it by cosine similarity against entries for the same model, held in memory or a vector store such as Qdrant. The lookup sits before input guardrails, so a hit skips them and output guardrails still check what is served.

In the memory semantic store at the cited commit, search filters by model and expiry, not namespace or tenant. A namespace header does not prevent cross-tenant semantic hits. Keep semantic caching off for private support requests unless each tenant has a genuinely isolated store.

Routing covers round-robin, weighted, least-latency, cost-optimized, adaptive and fastest (race), plus conditional routes, complexity tiers, model fallbacks, retries, failover and a circuit breaker. Virtual keys carry models and providers allow lists, RPM and TPM limits and budgets. When an agent presents the key on /mcp, its allowed_tools and denied_tools lists filter tools/list and gate each tools/call.

The docs cover each part in depth: routing strategies and fallbacks, exact and semantic caching, virtual keys and per-key limits and the MCP gateway.

The gateway README, as of 2026-11-11, publishes reproducible benchmarks with committed configs. A 4-vCPU container load test against a mock upstream held about 29k requests per second. A separate Go benchmark with an in-process mock and no network I/O measured about 66 microseconds of gateway-internal time per proxied chat completion.

The core is Apache 2.0 and self-hosts with Docker Compose or the signed Helm chart, and the Agent Command Center source holds every plugin named above.

To compare deployment options, see our guide to self-hosted AI gateways.

One request through Agent Command Center

The client is the stock OpenAI SDK pointed at a self-hosted gateway run on its own, on its default port 8080, with a virtual key instead of the provider key. A cache namespace header scopes the exact-match key. The script prints the five x-agentcc-* headers and the answer, and we checked this call shape against openai 3.49.0.

# One request through the gateway with the stock OpenAI SDK.
# not executed: needs a running gateway and a provider key
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-agentcc-...")  # virtual key, never the provider key

raw = client.chat.completions.with_raw_response.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Where is order 4417?"}],
    extra_headers={"x-agentcc-cache-namespace": "support-prod"},
)
for h in ("x-agentcc-provider", "x-agentcc-model-used", "x-agentcc-cache", "x-agentcc-latency-ms", "x-agentcc-cost"):
    print(h, raw.headers.get(h))
print(raw.parse().choices[0].message.content)

The x-agentcc-cache header answers the opening scene’s question. It reports whether the request hit or missed the cache, so a request that never reached the provider shows up as a hit. On an exact hit the gateway skips the cost and credits plugins, so no provider tokens are charged.

The routing and cache config behind it

The self-hosted config.yaml excerpt below sets up the support-bot key with two models, rate limits and tool lists for the standalone providers. It turns on exact caching with a 10 minute TTL and serves MCP on /mcp. Semantic caching stays off for this private support workflow; the 0.92 threshold is shown for an isolated deployment. Every key was checked against the gateway’s Go config structs.

# Self-hosted Agent Command Center config.yaml excerpt (keys from agentcc-gateway/internal/config/config.go)
auth:
  enabled: true
  keys:
    - name: "support-bot"
      key: "${SUPPORT_BOT_KEY}"
      key_type: "internal"
      owner: "support-team"
      models: ["gpt-4o-mini", "claude-sonnet-4-6"]
      rate_limit_rpm: 600
      rate_limit_tpm: 200000
      allowed_tools: ["search_kb", "get_order"]
      denied_tools: ["refund_order"]
      metadata: {team: "support", env: "prod"}
cache:
  enabled: true
  default_ttl: 10m
  max_entries: 10000
  semantic:
    enabled: false  # A namespace does not isolate this shared semantic store.
    backend: "memory"
    threshold: 0.92
routing:
  default_strategy: "least-latency"
  failover:
    enabled: true
    max_attempts: 3
    on_status_codes: [429, 500, 502, 503, 504]
    on_timeout: true
  retry:
    enabled: true
    max_retries: 2
    initial_delay: 500ms
    max_delay: 10s
    multiplier: 2.0
  circuit_breaker:
    enabled: true
    failure_threshold: 5
    success_threshold: 2
    cooldown: 30s
  model_fallbacks:
    gpt-4o-mini: ["claude-sonnet-4-6"]
mcp:
  enabled: true
  endpoint: "/mcp"

Routing uses least latency with failover on 429 and 5xx codes and on timeouts. It adds two retries with backoff, a circuit breaker with a failure threshold of 5 and a 30 second cooldown, and a model fallback from gpt-4o-mini to claude-sonnet-4-6. The denied_tools entry keeps refund_order away from the support bot.

key_type: "internal" is the documented standalone key type for provider credentials in config.yaml. Without it, a key without org-scoped provider credentials gets a 403 instead of using those providers. Internal keys use the standalone provider catalog rather than a per-key provider allow list; keep them with trusted services, not tenants.

Tracing your own request through the gateway

Back to the support bot that quoted last week’s refund policy. The response header shows a cache hit, and the old configuration explains why: semantic caching with a loose threshold and a long TTL. The example above leaves semantic caching off for that private workflow. Expire the stale entries and shorten the TTL for policy answers before the next request.

Start with your own diagram. Draw the seven stages for one real request, mark the ones that sit between the user and the answer, and note where your cache sits relative to input guardrails. Then send one request and read its five headers.

Set TTLs per answer type, so policies and order status expire in minutes. Then run the same trace on Agent Command Center. Follow your support-bot request from the first key check to the final span, and keep the cache header in view.

To start one, follow the self-hosted Agent Command Center guide and point the OpenAI SDK’s base_url at it.