Best 5 AI Gateways for Embedding API Routing in 2026

Originally published May 17, 2026.

A platform team at a mid-stage RAG company indexed 10 million support tickets at 1,000 tokens each on text-embedding-3-small at $0.02 per million tokens: $200,000 of provider spend on a corpus that has to be re-embedded every time the team upgrades the model, paid in full on every backfill, shipped without an input-hash cache because the SDK doesn’t surface one. This guide compares the five AI gateways production teams should choose between in 2026 for embedding API routing, scored on the seven axes that decide whether a 10 M to 100 M document pipeline costs $200K or $40K per rebuild.

TL;DR

Future AGI Agent Command Center is the strongest pick for an AI gateway for embedding API routing because it bundles OpenAI-compatible /v1/embeddings drop-in, input-hash deduplication, batch-API submission against OpenAI and Cohere, dimension-aware adapters, per-tenant attribution by virtual key, and a model-migration controller, all wired into the same OpenTelemetry trace as the downstream retrieval and eval. The other four picks below win on specific edges.

1. Future AGI Agent Command Center — Best overall. Input-hash cache, batch-API submission, dimension adapters, per-VK attribution, re-embed controller, in one Apache-2.0 Go binary.
2. Portkey — Best for managed embedding observability with a 4-tier budget hierarchy across OpenAI, Cohere, Voyage, Mistral (verify the Palo Alto Networks acquisition timeline before signing multi-year).
3. LiteLLM — Best for Python-first indexing jobs wanting broad provider coverage. Pin commits after the March 24, 2026 PyPI compromise (1.82.7 and 1.82.8 affected; upgrade past 1.83.7).
4. OpenRouter — Best for quick chat-provider A/B. Embedding catalogue is weaker and there’s no gateway-layer cache or batch; cloud only with per-token markup.
5. Cloudflare AI Gateway — Best for edge-cached embedding traffic at internet scale with indexing on Workers. Cloud-only; cache is response-side.

Why Embedding API Routing Needs a Gateway, Not an SDK

Embedding workloads have a different cost shape from chat. A chat request is one shot, latency dominated. An embedding request is one of millions, throughput dominated, produced by a script nobody is watching. The failure modes that bite embeddings are mostly invisible to a chat-shaped SDK.

The third rebuild problem. First rebuild eats the full bill. Second you wish you had cached. Third you wish you had cached, batch-submitted, and routed multilingual to Cohere embed-multilingual-v4. At 10 M docs × 1,000 tokens on text-embedding-3-small at $0.02/M, baseline is $200K. A 30 percent cache hit cuts to $140K. The 50 percent OpenAI batch discount cuts the new portion to ~$70K. A Cohere embed-v4 route at $0.10/M on the multilingual slice cuts another quarter. Combined: $40-50K per rebuild, none in the SDK.

The dimension-mismatch problem. OpenAI 3-small is 1,536 native (truncatable to 512 or 256). 3-large is 3,072. Voyage voyage-3 is 1,024. Cohere embed-v4 is 1,536. Mistral mistral-embed is 1,024. Two providers on one vector store means either two indexes (doubles vector-store cost) or a gateway that pads, truncates, and re-norms.

The batch-API problem. OpenAI’s batch endpoint returns within 24h at 50 percent of synchronous. Cohere ships the same shape. Most pipelines never use them because the SDK doesn’t expose batch as first-class; the team pays full price on tokens that didn’t need to return in seconds.

The model-migration problem. Upgrading 3-small to 3-large isn’t a config flip; the new model emits a different vector space, the index is stale. You need a backfill that re-embeds, dual-writes, validates retrieval quality on a held-out set, and flips reads only when the new index passes. Most teams write a one-off script and pray.

The per-tenant attribution problem. The customer re-indexing its 5 M corpus three times in a week is a noisy neighbor and a billing question. Without per-tenant attribution you can’t tell. One VK per tenant + tenant_id turns “why did our bill jump 40 percent” into “tenant acme-corp re-indexed three times; charge or rate-limit.”

A gateway without input-hash cache, batch-API, dimension adapters, a re-embed controller, and per-tenant attribution is a /v1/embeddings forwarder, not an embedding gateway.

How We Picked: 7 Axes for Embedding API Routing

The Embedding Cost-At-Scale Stack, tuned for indexing at 10 M to 100 M documents and online retrieval at 1 k+ QPS.

A gateway winning on three of these and ignoring four is fine for a prototype and bad for a production index.

7-Axis Capability Matrix

Across the five below, Future AGI Agent Command Center leads on combined batch-API, cache, dimension adapters, and re-embed controller. Portkey leads on managed dashboard. LiteLLM leads on Python ergonomics. OpenRouter is the lowest-friction directory. Cloudflare is the cleanest edge-cache for retrieval reads.

The four columns that decide the pick: dimension consistency, batch-API, input-hash cache, re-embed controller. Online retrieval prioritizes latency, cache backend, and OpenTelemetry attribute shape.

Future AGI Agent Command Center: Best Overall for Embedding API Routing

Future AGI Agent Command Center tops the list because it ships every axis of the cost-at-scale stack at one network hop, and pipes the embedding span into the same OpenTelemetry trace as the retrieval, reranker, and generation. Other gateways treat /v1/embeddings as a chat-shaped passthrough and leave index, retrieval, and eval on three separate trees. Future AGI ties them by span_id so the retrieval-quality eval that flags a slice produces the labelled dataset agent-opt uses to fire the re-embed.

Best for. ML and platform teams running 10 M to 100 M document indexing and 1 k+ QPS online retrieval who want OpenAI-compatible drop-in, batch-API as default, input-hash dedup, dimension consistency, per-VK attribution, and an eval-driven re-embed controller.

Key strengths.

• OpenAI-compatible /v1/embeddings drop-in. Swap base_url to https://gateway.futureagi.com/v1. Routes to OpenAI text-embedding-3-small ($0.02 per million), 3-large ($0.13), ada-002 ($0.10), Cohere embed-v4 ($0.10), multilingual-v4 ($0.10), Voyage voyage-3 ($0.06), voyage-3-large ($0.18), voyage-code-3 ($0.18), voyage-finance-2 ($0.12), Mistral mistral-embed ($0.10), codestral-embed ($0.15), Jina, Nomic, and self-hosted BGE / E5 / Instructor.
• Deterministic input-hash cache. Hash of (provider, model, dimension, normalized text) on Redis or in-memory; survives restarts; per-tenant isolation. Second-rebuild hit rates 30 to 70 percent; online queries 5 to 20 percent.
• Batch-API as a first-class concept. OpenAI batch embeddings and Cohere batch exposed via POST /v1/batches with webhook callback and partial-failure retries. Unlocks the 50 percent batch discount as the default indexing path.
• Dimension adapters. Pad, truncate, or re-norm across providers. A/B Cohere embed-v4 (1,536), Voyage voyage-3 (1,024), and text-embedding-3-large (3,072) on one vector store without two indexes.
• Model-migration controller. Backfill re-embeds on upstream model upgrades, dual-writes, tracks coverage, flips reads when retrieval-quality eval passes the quality floor on a held-out set.
• Per-tenant attribution. VKs per tenant; tenant_id on every embedding span; per-VK budgets; cost-by-tenant exported as OpenTelemetry traces and Prometheus metrics on /-/metrics.
• Trace correlation. Embedding span shares trace ID with vector-store retrieval, reranker, generation, and Ragas eval (faithfulness, context-precision, context-recall). traceAI instruments 35+ frameworks OpenInference-natively, and Error Feed (FAGI’s “Sentry for AI agents”) sits alongside as the zero-config error monitor: auto-clusters related embedding-quality and retrieval-grounding failures (50 traces → 1 issue), auto-writes the root cause plus a quick fix plus a long-term recommendation per issue, and tracks rising/steady/falling trend per issue so embedding-drift regressions surface like exceptions rather than buried in trace search. One link closes the optimize loop.
• The Future AGI Protect model family on the path when you want it. Protect is FAGI’s own fine-tuned model family built on Google’s Gemma 3n with specialized adapters across four safety dimensions (content moderation, bias detection, security/prompt-injection, data privacy/PII), natively multi-modal across text, image, and audio, a model family, not a plugin chain. ~67 ms p50 text and ~109 ms p50 image (arXiv 2510.13351), opt-in per route, so embedding calls without PII or injection scanning skip it; the same dimensions are reusable as offline eval metrics so the prod policy and the eval rubric stay in sync.
• Apache 2.0; single Go binary. Docker, Kubernetes, AWS, GCP, Azure, on-prem, air-gapped, or cloud at gateway.futureagi.com/v1. traceAI, ai-evaluation, agent-opt also Apache 2.0.

Where it falls short.

• Cost dashboard is OpenTelemetry-first and Prometheus-on-/-/metrics; teams wanting a fully managed dashboard without a Grafana board prefer Portkey.
• Re-embed controller assumes a vector store with dual-write support (Qdrant, Pinecone, Weaviate, pgvector). Flat-file or homegrown indexes need their own backfill.
• Full agent execution tracing is “In Progress” on the public roadmap (Future AGI GitHub), rolling out alongside the existing gateway-side OpenTelemetry trace export.

from openai import OpenAI

client = OpenAI(
    api_key="$FAGI_API_KEY",
    base_url="https://gateway.futureagi.com/v1",
)

# OpenAI-compatible /v1/embeddings drop-in. Cache, batch-API, dimension
# adapters, per-VK attribution, and trace correlation at the same hop.
response = client.embeddings.create(
    model="voyage/voyage-3",
    input=batch_of_chunks,
    dimensions=1024,
)

Verdict. Strongest pick when the brief is OpenAI-compatible /v1/embeddings drop-in, input-hash cache, batch-API as the default indexing path, dimension consistency, per-VK attribution, and a re-embed controller tied to retrieval-quality eval, all under Apache 2.0.

Portkey: Best for Managed Embedding Observability Dashboard

Portkey is the strongest pick when embedding is a sub-question inside a broader multi-tenant chat workload and the team wants a managed dashboard for both. Four-tier hierarchy (org / team / user / VK) maps cleanly onto multi-tenant SaaS.

Best for. Multi-tenant SaaS already on Portkey for chat wanting the same dashboard for embedding spend without a Grafana board.

Key strengths.

• Four-tier budget hierarchy on /v1/embeddings; per-tenant cost dashboards out of the box.
• Adapter library: OpenAI text-embedding-3, Cohere embed-v4 + multilingual-v4, Voyage voyage-3, Mistral mistral-embed, Jina, self-hosted.
• Native cost dashboard by tenant, model, and route.
• Open-source gateway core (github.com/Portkey-AI/gateway); control plane in Portkey cloud.

Where it falls short.

• Batch-API submission is partial; embedding batch is less first-class than Future AGI’s, so the 50 percent discount becomes batch glue you write.
• Dimension consistency is pass-through; A/B on one vector store means pad/truncate outside the gateway.
• No model-migration controller; 3-small to 3-large is a script.
• Observability is dashboard-first; OTel export less load-bearing for embedding spans, harder to tie eval to embedding by span_id.
• Palo Alto Networks announced intent to acquire Portkey on April 30, 2026; close expected PANW fiscal Q4 2026. Verify standalone-product continuity before signing multi-year.

Verdict. Most mature managed-dashboard surface for embedding cost in 2026, four-tier hierarchy hard to match if the constraint is finance-facing reporting. PANW timeline is the live procurement risk.

LiteLLM: Best for Python-First Indexing Pipelines (Post-CVE)

LiteLLM is the Python-first proxy with the broadest provider adapter set on this list. For embeddings the appeal mirrors chat: pip install, custom adapters, broad coverage, FastAPI surface.

Best for. Python-first ML platform teams on FastAPI / uvicorn wanting broad provider coverage and willing to pin commits (or upgrade past 1.83.7).

Key strengths.

• Broadest provider coverage via 100+ adapters: OpenAI, Cohere, Voyage, Mistral, Bedrock Titan / Cohere, Vertex AI text-embedding, Jina, Nomic, self-hosted via custom adapters.
• MIT (enterprise dir separate); trivial to fork or audit.
• Virtual keys with per-key budgets on embedding traffic.
• Native fit with Python observability (Prometheus exporter, OTel middleware).

Where it falls short.

• March 24, 2026 PyPI supply-chain compromise. Versions 1.82.7 and 1.82.8 exfiltrated SSH keys, cloud credentials, and Kubernetes configs, per the Datadog Security Labs writeup. Pin commits, scan deps, rotate credentials, upgrade past 1.83.7.
• Python runtime; materially slower throughput than Go binaries at high concurrency.
• Batch-API is adapter-level not first-class; teams write polling and retry per provider.
• Dimension adapters are adapter-level.
• No model-migration controller; the re-embed pipeline is a script you write.

Verdict. Broadest embedding-provider coverage, but March 2026 shifts it from “default” to “pin commits or upgrade past 1.83.7 and audit deps.”

OpenRouter: Best for Quick Provider A/B, Weakest Fit on Embeddings

OpenRouter is the lowest-friction provider directory on this list. One API key, one base URL, 200+ models, transparent per-token markup. For embeddings it’s the weakest entry: catalogue is shallower than chat and the gateway-layer features (cache, batch, dimension adapters) don’t exist.

Best for. Small teams or experiments running provider A/B who want OpenAI 3-series and Cohere embed-v4 behind one key without infrastructure.

Key strengths.

• One key, one base URL; minimal setup overhead.
• Drop-in client.embeddings.create via OpenAI-compatible surface.
• Transparent comparison on the OpenRouter models directory.

Where it falls short.

• /v1/embeddings catalogue is narrower than chat. OpenAI 3-small/3-large and Cohere embed-v4 present; Voyage and Mistral lag.
• No input-hash cache at the gateway. Every 10 M rerun pays full price.
• No batch-API submission.
• No dimension adapters; A/B on one store means two indexes.
• No per-tenant attribution beyond the OpenRouter API token.
• Per-token markup; at 10 M+ docs crosses self-host break-even at modest scale.
• Closed source.

Verdict. Lowest-friction way to get OpenAI 3-series and Cohere embed-v4 behind one key; wrong gateway when embedding-cost optimization at scale is the brief.

Cloudflare AI Gateway: Best for Edge-Cached Online Retrieval Reads

Cloudflare AI Gateway earns a spot because it solves a different shape of the problem. Where the other four optimize indexing (write-heavy, throughput-dominated), Cloudflare optimizes the online retrieval read path (read-heavy, latency-dominated). For RAG products whose query embeddings repeat (“pricing”, “how do I reset my password”), the edge cache is the cheapest hop.

Best for. Teams with internet-scale retrieval read patterns, indexing on Workers, response cache at the edge instead of the app tier.

Key strengths.

• Edge response cache; recurring query embeddings return without a provider call.
• Workers + Durable Objects as a self-hostable cache layer.
• Workers AI for local inference of small embedding models (@cf/baai/bge-base-en-v1.5, @cf/baai/bge-large-en-v1.5).
• Native logs with per-request cost.

Where it falls short.

• Embedding catalogue smaller than the other four. Frontier models reachable as upstream calls but not first-class.
• No batch-API submission; pipelines route around the gateway for the 50 percent discount.
• No input-hash cache as first-class; edge cache is response-side, not indexing-rerun dedup.
• No dimension adapters; cross-provider A/B is on you.
• No model-migration controller; backfill is a Worker you write.
• Per-tenant attribution at API token level.
• Cloud-only; air-gapped not a path.

Verdict. Right cache surface for online retrieval reads when the stack lives on Cloudflare; wrong gateway when the brief is a 10 M+ indexing pipeline with batch-API, dimension consistency, and eval-driven re-embed. Use next to one of the other four.

A Cost Walk-Through: 10 M Documents, Four Strategies

Assume 1,000 tokens per chunk on text-embedding-3-small at $0.02 per million. Total: 10 B tokens = $200K per rebuild.

• Strategy 1: SDK loop. #1: $200K. #2: $200K. #3 (upgrade to 3-large at $0.13/M): $1.3M. Cumulative: $1.7M.
• Strategy 2: Cache only. #1: $200K. #2 with 60 percent cache hit: $80K. #3 (model change invalidates cache): $1.3M. Cumulative: $1.58M.
• Strategy 3: Cache + batch-API. #1 via batch at 50 percent: $100K. #2: $40K. #3 with model upgrade + batch: $650K. Cumulative: $790K.
• Strategy 4: Cache + batch + Voyage voyage-3 ($0.06/M) for 70 percent English, Cohere multilingual-v4 ($0.10/M) for 30 percent multilingual, plus re-embed-only-on-eval-drift. #1: $36K. #2 with cache hit: $14.4K. #3 re-embeds only the 15 percent eval flagged: $5.4K. Cumulative: $55.8K.

The gateway is a 30x cost difference over three rebuilds. Every column (cache, batch, routing, retrieval-quality-driven re-embed) is a gateway responsibility, and they only compound at one network hop.

Common Implementation Mistakes

One: caching on chunk ID instead of input hash. A chunking-strategy change (overlap, max size, separator) leaves chunk IDs identical and content different; the cache returns stale embeddings. Cache on the hash of (provider, model, dimension, normalized text).

Two: ignoring text normalization before hashing. Trailing whitespace, smart quotes, Unicode NFC vs NFD, case, all produce different hash keys for the same input. Hit rates drop from 60 percent to 20 percent. Apply Unicode NFC, strip whitespace, lowercase if tolerable.

Three: not batching indexing jobs through OpenAI batch. The biggest unforced error. Synchronous endpoints cost 2x batch on indexing; a tqdm loop leaves 50 percent of the budget on the table.

Four: A/B-ing providers without dimension adapters. Comparing Voyage voyage-3 (1,024) against text-embedding-3-large (3,072) on one vector store without adapters means two indexes and two retrieval calls per read.

Five: re-embedding the whole corpus on every model upgrade. Retrieval-quality drift is concentrated. Tie embedding to eval by span_id and fire backfill only on the slice failing the quality floor. At 10 M documents the cost difference is $130,000 vs $20,000.

Six: no per-tenant attribution. Without it, “why did our bill jump 40 percent” is a forensic exercise. One VK per tenant, tenant_id on every call, export cost as Prometheus metric and OpenTelemetry attribute.

Decision Framework

Choose Future AGI Agent Command Center if:
  - You run a 10 M to 100 M document indexing pipeline and the bill matters
  - You want input-hash deduplication that survives restarts
  - You want batch-API submission as the default indexing path
  - You want dimension consistency across providers on one vector store
  - You want per-tenant attribution by VK and a re-embed controller tied
    to retrieval-quality eval
  - OSS instrumentation is hard requirement (Apache 2.0 across traceAI,
    ai-evaluation, agent-opt)

Choose Portkey if:
  - You already use Portkey for chat and want the same managed dashboard
    for embedding spend
  - You can absorb the PANW acquisition risk on a multi-year contract

Choose LiteLLM (post-incident pinned) if:
  - You are Python-first and broad provider adapter coverage is binding
  - You can pin commit hashes (or upgrade past 1.83.7) and audit deps

Choose OpenRouter if:
  - You want quick provider A/B without standing up infrastructure
  - Your corpus is under 100 K documents

Choose Cloudflare AI Gateway if:
  - Your retrieval read path is internet-scale and indexing lives on Workers
  - You compose batch and re-embed logic in Workers yourself

Future AGI Implementation Walk-Through: Closing the Loop

The wedge: embedding, retrieval, reranker, generation, and retrieval-quality eval spans share the same trace ID, and the optimizer reads that trace and fires actions back to the gateway.

1. Indexing call. client.embeddings.create(model="voyage/voyage-3", input=batch, dimensions=1024). Span emits provider, model, dimension, tokens, cache_hit, batch_id, tenant_id, cost_usd. Gateway submits via Voyage batch; cost is half of synchronous.
2. Index write + query. Vector store (Qdrant / Pinecone / Weaviate / pgvector) writes on same trace ID. Online query hits the gateway; input-hash cache returns on repeats (5 to 20 percent); miss path embeds via the cheapest provider passing the quality floor.
3. Retrieval, reranker, generation. Vector retrieval, Cohere rerank-v3 or Voyage rerank-2, and final LLM call land on the same trace.
4. Retrieval-quality eval. Held-out suite runs Ragas faithfulness, context-precision, context-recall. Eval span ties back by span_id to embedding, retrieval, reranker, and generation spans.
5. Optimizer reads the trace. When eval flags a slice where context-recall drops below 0.8, agent-opt fires a backfill that re-embeds only that slice with a different model or dimension.

The embedding pick stops being a static config and becomes an artifact of the trace, eval, optimize loop. Choosing between text-embedding-3-small at $0.02 per million and Voyage voyage-3 at $0.06 per million isn’t a Slack debate; it’s a slice-by-slice decision the optimizer makes from observed retrieval quality. No other gateway in this cohort closes the loop in one product.

The 2026 Trust Cohort

Every embedding-gateway listicle on the SERP treats procurement as if Q1 to Q2 2026 didn’t happen.

• Helicone joining Mintlify (March 3, 2026). Roadmap shifts toward documentation-platform-first. Teams running Helicone observability over LiteLLM should plan migration.
• LiteLLM PyPI supply-chain compromise (March 24, 2026). Versions 1.82.7 and 1.82.8 exfiltrated SSH keys, cloud credentials, and Kubernetes configs. Pin commits, scan dependencies, rotate credentials, upgrade past 1.83.7. Source: Datadog Security Labs.
• Anthropic MCP STDIO RCE class (April 2026). OX Security disclosed an STDIO transport flaw affecting 7,000+ MCP servers and 150M+ downstream downloads. Indexing pipelines pulling sources via MCP should pin to Streamable HTTP with OAuth 2.1 and least-privilege scopes.
• Portkey acquired by Palo Alto Networks (April 30, 2026). Becomes the AI Gateway for Prisma AIRS; close expected PANW fiscal Q4 2026. Source: PANW press release.

License clarity and acquisition independence are part of the embedding-routing decision. A cheap gateway you have to migrate off in six months isn’t cheap when the migration cost is re-embedding 10 M documents.

Which Embedding-Routing Gateway Is Right for You in 2026?

Embedding API routing in 2026 is a seven-axis stack: provider breadth, dimension consistency, batch-API, input-hash cache, model-migration, per-tenant attribution, and online p95 latency at one network hop, under a license not about to be re-platformed.

Future AGI Agent Command Center is the strongest single pick when the constraint is one Apache-2.0 binary shipping every axis self-hostable and tying embedding to retrieval, reranker, and eval by span_id.

Deeper reads: Agent Command Center docs, GitHub, observability docs, Protect docs, Evaluation docs, OpenTelemetry GenAI conventions.

Try Future AGI Agent Command Center free: drop-in OpenAI-compatible /v1/embeddings, input-hash cache, batch-API submission, dimension adapters, per-tenant attribution, and an eval-driven re-embed controller, all under Apache 2.0.

Related reading

• Best 7 AI Gateways for Multi-Model Routing in 2026, how cost-quality routing decisions get made at the gateway hop
• Best 5 AI Gateways for LLM Cost Optimization in 2026, the five-layer cost stack and the 2026 trust cohort
• Best 5 AI Gateways for LLM Failover and Fallback in 2026, fallback and failover gateway picks
• Best 5 AI Gateways for Prompt Management in 2026, the prompt-management gateway picks