Sovereign AI Infrastructure and Where the Model Gateway Fits
Sovereign AI infrastructure turns on where requests may go. See how data sovereignty, residency and localization differ and how a gateway keeps calls local.
Table of Contents
A platform lead at a European health insurer gets a clear brief: the new claims assistant runs on sovereign AI infrastructure, and member data stays in the country. The GPUs sit in a national data center, and the model is open weights served on them. The review looks finished.
Then the load test runs. When the local model is busy, a fallback route sends overflow to a hosted model API in another region. Every conversation’s trace also exports to a hosted observability backend abroad. Neither problem shows up in the hardware plan, and both move member data across a border.
Sovereign AI infrastructure is decided as much by where each request is allowed to go as by where the servers sit. The request path, with its fallbacks and exporters, is where this review failed, and it is what the rest of this guide designs.
TL;DR
- Sovereign AI infrastructure means compute, models, data and request controls that all answer to one jurisdiction, with operators that jurisdiction can reach.
- Hardware location is one check. Fallback routes, default endpoints, trace exporters and personal keys each open a path abroad.
- The model gateway is the control layer: one network exit for model calls, allow lists of in-region destinations, fallbacks inside the boundary and logs that stay local.
- A self-hosted gateway only governs traffic that reaches it, so pair it with egress rules that block every other route to model APIs.
Why in-country GPUs do not make an AI system sovereign
In-country GPUs do not make an AI system sovereign, because sovereignty usually leaks through ordinary engineering defaults. A fallback provider in another region, an SDK that calls a global endpoint, a trace exporter pointed at a hosted backend, or a personal key will each send a prompt out of the country.
Fallbacks and retries that cross a border
Failover lists are written for uptime. Someone adds a hosted model in another region as the backup, and retries do the rest. The claims assistant’s overflow route is this leak, and it fires only under load, so a quiet test never sees it.
Default endpoints and regions
A provider SDK uses its built-in default endpoint whenever the code sets no base URL or region. Cloud model services work the same way: the region in the client config decides where inference runs. One service that never pinned its endpoint sends every prompt wherever that default resolves.
Telemetry, traces and logs
An LLM trace usually carries the full prompt and output. A tracing SDK pointed at a hosted backend in another region copies every member conversation there. That was the insurer’s second finding: the model stayed local, and the record of every conversation left.
Keys that skip the front door
Personal or team keys called straight from code bypass the platform team’s routing. They show up as direct calls to provider hosts in egress logs. Our guide to how shadow AI shows up in engineering teams covers finding them through egress logs, expenses and secret scanning.
Prompt injection and tool misuse are separate risks, handled by agent security controls rather than routing rules. This guide stays on one question: where data and requests may go, and which layer of the stack decides that for every call.
What is sovereign AI infrastructure?
Sovereign AI infrastructure is the compute, models, data stores and request controls that keep an AI system subject to one jurisdiction’s laws, with the organization or country holding operational control itself. Canada’s sovereign compute program sets the same test for compute: located in the country and governed there.
The program guide for Canada’s Sovereign AI Compute Infrastructure Program calls it “a Canadian-located, Canadian-governed system that ensures data residency, operational control, and decision-making authority and agency remain in Canada”.
Sovereign AI vs AI sovereignty
Sovereign AI is the system you build. AI sovereignty is the policy goal behind it. The World Economic Forum’s 2026 report on AI infrastructure and sovereignty describes it as the ability of economies to govern AI “in accordance with their own values”.
The report’s full wording adds “strategic and operational control, flexibility and, ultimately, resilience”, reached through localized investment and trusted international collaboration. A country pursues AI sovereignty. An engineering team ships sovereign AI that meets it, and that is the half this guide is about.
Sovereign AI vs private AI
Private AI keeps models and data inside one organization’s environment. Sovereign AI adds two checks: which jurisdiction that environment sits under, and who operates it. A private deployment in a foreign cloud region is still private and still fails the jurisdiction check.
The four layers, and what to verify in each
Every sovereign AI stack has four layers: compute, models, data, and the control layer that decides where each request goes. Most plans cover the first three and leave the fourth to SDK defaults. The definition holds only when all four meet it.
| Layer | Question it answers | Typical in-country options | Verify before launch |
|---|---|---|---|
| Compute | Where do training and inference run, and who operates the hardware? | National data centers, a sovereign cloud region, on-premises GPUs | Operator, its jurisdiction, support access paths |
| Models | Which models may serve a request, and can you host them yourself? | Open-weight models on your own servers, provider models pinned to an in-country region | Model licence, region per endpoint, behaviour when busy |
| Data | Where do prompts, outputs, embeddings, traces, request logs and eval datasets live? | In-region databases, object stores and observability backends | Every export target, including tracing and analytics |
| Control | What decides which provider, region and model each request reaches? | A self-hosted model gateway at the only network exit for model calls | Allowed providers per key, fallback targets, egress rules |
The self hosted LLM option for the models layer
The strictest option is a self hosted LLM, with the weights on in-country GPUs and no outside API for inference. vLLM and Ollama are two open-source servers built for this. Hosted models still fit when every endpoint is pinned to an in-country region.
Swapping a hosted model for an open-weights one can change output quality, so run the same test set on both before you cut over. Our walkthrough on evaluating a self-hosted LLM on vLLM shows how to score the replacement against the original.
Observability data belongs to the data layer
Traces, request logs and eval datasets hold the same prompts and outputs as the application database. Treat every observability backend as a data store with its own location, operator and retention. If the trace store sits abroad, the data layer is abroad.
Our guide to self-hosting LLM observability covers keeping traces inside your own perimeter.
Sovereign AI deals that landed in September and October 2026
Four sovereign AI announcements landed between September 29 and October 28, 2026, from Bell and Cisco, F5 and Telenor, HPE with Mzansi Mind, and SAP with NTT. Three of the four center on compute and cloud capacity rather than the control layer.
On September 29, Bell and Cisco announced a memorandum of understanding to collaborate on a sovereign AI infrastructure offering for Canada, pairing Bell’s data centres and network with Cisco’s technology. That is the compute layer. The details are in Bell and Cisco’s memorandum of understanding.
On October 22, HPE announced that Mzansi Mind chose HPE servers and switches to power its Siya language model. Mzansi Mind is a South African sovereign enterprise AI platform from Solcon Capital, built for “local business, regulatory and data-residency requirements”. Compute again.
On October 28, SAP and NTT announced their intention to cooperate in the areas of SAP Sovereign Cloud and SAP Business AI. SAP’s newsroom carries SAP and NTT’s planned cooperation; it is one joint statement of intention, and it covers the cloud layer.
The fourth is about control. On October 20, F5 announced that Telenor AI Factory selected F5 BIG-IP Next for Kubernetes as an “intelligent AI service plane” for its NVIDIA-powered sovereign AI infrastructure, which is designed to serve “Norwegian and European organizations”.
The same release says sovereign AI needs a service plane that can “govern where data flows”. That phrase is the control layer in plain words, the layer the insurer skipped when it planned for GPUs and left routing to SDK defaults.
Where does the model gateway fit in sovereign AI infrastructure?
The model gateway is the control layer of sovereign AI infrastructure. Every model call passes through it, so it is the one place to enforce which providers, regions and self-hosted models a team may use, to log each request inside the boundary, and to refuse anything that would leave it.
If the pattern is new, start with what an AI gateway does.
Follow one claims request from the app in the figure below: the gateway checks the team key’s allow list, an in-region provider answers, the request is logged in an in-region store, and the response comes back. The outbound route abroad stays closed to the fallback.
One exit for model traffic
Write network egress rules so that only the gateway can reach model endpoints. Without that rule, the gateway sees only the traffic that chooses to use it, and a hard-coded personal key walks past. With it, the direct call fails at the network.
Destinations as an allow list per team
Each team key lists only the in-region providers and self-hosted models it may call. Watch the empty list: where an empty list means every destination, a key created in a hurry reaches everything. In a sovereign setup every key names its destinations.
Name providers by region
Register one provider entry per region, for example a hosted model service pinned to its EU region next to the self-hosted vLLM server. The allow list on each key then reads like a residency map, one a reviewer can check line by line.
Fallbacks that stay inside the boundary
Draw failover targets only from the same allow list. When the local model is busy, the next hop is a second in-region server or a hosted endpoint pinned to the country. That closes the overflow route without giving up uptime, as long as two in-region targets exist.
Logs that stay where the data stays
The gateway records every request. Point its request logs and traces at an in-region database you operate, and check that no exporter forwards a copy elsewhere. This fixes the insurer’s second finding: the record of each conversation stays with the conversation.
Matching gateway rules to data sovereignty, data residency and data localization
Data residency names a location, data sovereignty names a jurisdiction, and data localization sets domestic storage or processing requirements. Each term asks something different of the gateway: endpoint and log locations, who operates each provider, and where fallbacks and exports may go.
The table maps each term to the gateway rule it drives, so every config line traces back to a requirement. It is an engineering reading of the terms, not legal advice; counsel decides which of them a contract or regulator imposes.
| Term | What it means | Who sets it | What it asks of the request path | Source |
|---|---|---|---|---|
| Data residency | The physical location where data is stored or processed | The organization, often in a contract | Every allowed endpoint and log store sits in the named region | IBM |
| Data sovereignty | Data is subject to the laws and authorities of a nation | Law and regulators | Allowed providers are operated under the chosen jurisdiction, not only hosted there | IBM, WEF |
| Data localization | A legal requirement that data be stored or processed inside the domestic territory | Governments | Meet local storage or processing requirements; block foreign routes when the applicable rule prohibits them | OECD |
Data sovereignty vs data residency for an AI request
Say the insurer adds a second hosted endpoint in the country, run by a provider that answers to a foreign court. The prompts never leave the country, so residency passes. Sovereignty asks whose law reaches the operator, so the same endpoint can still fail the review.
IBM’s breakdown of sovereignty and residency puts it in one line: “data sovereignty is a legal concept and data residency is a geographical category.” In config terms, the provider’s operator belongs on the allow-list review next to its region, and both get checked.
Where localization fits
Localization sets a domestic storage or processing requirement. The OECD’s study of data localisation measures distinguishes local-copy requirements that allow foreign processing, conditional transfers, and transfer bans. Block foreign fallbacks and exports when the applicable rule prohibits them. The insurer’s brief here requires member data to stay in-country, so its boundary remains strict.
Sovereignty covers more ground, and the WEF’s Data Sovereignty in Practice report puts it this way: “Localization is one instrument within it.” A localization rule is one input to the allow list. Jurisdiction and operator checks are the others, and they still apply when every server is local.
Sovereign cloud vs private cloud
A private cloud is defined by tenancy, a sovereign cloud by jurisdiction, operations and supply chain. NIST’s cloud definition calls private cloud infrastructure “provisioned for exclusive use by a single organization comprising multiple consumers”. A foreign-run private cloud can be non-sovereign, and a sovereign cloud can be multi-tenant.
The European Commission’s Cloud Sovereignty Framework measures eight objectives and grades providers on assurance levels from SEAL-0 to SEAL-4. SEAL-2, “a Data Sovereignty level”, was the bar in the Commission’s sovereign cloud tender, launched in October 2025 and awarded in April 2026. Tenancy is not one of the eight.
Running the control layer on Future AGI Agent Command Center
Teams can run Agent Command Center as this control layer. Its gateway code is Apache-2.0, it self-hosts with Docker Compose or inside the Future AGI platform Helm chart, and its virtual keys carry allow lists of models and providers, so a key can be limited to in-region or self-hosted endpoints.
A self-hosted Future AGI install already runs the gateway, inside the Docker Compose stack or as its own Deployment in the Helm chart. You can also self-host the gateway alone as a Docker image or Go binary; that page says “All requests stay within your infrastructure.”
The Future AGI Helm chart installs the platform, gateway included, from oci://ghcr.io/future-agi/charts/futureagi. Allow lists live on virtual keys: models and providers set what a key can call, and an empty list means all, so every team key names its destinations.
A key for the claims assistant lists the in-country vLLM server and one region-pinned hosted provider. A call to any model outside that list is refused at the gateway. Self-hosted providers such as Ollama and vLLM connect with a base_url and api_format: "openai".
Hosted providers that take a region can be registered as one provider entry per region: Bedrock requires an AWS region, and Vertex AI uses GCP project and location headers. The providers page also lists cross-region failover for Bedrock, so keep every failover target in the country.
A model server on a private network needs one switch first, because provider URLs on private networks are refused until you allow them, for the gateway and the API alike. The repo’s installation guide shows the switch in its Local models section:
echo "AGENTCC_ALLOW_PRIVATE_PROVIDER_URLS=true" >> .env
docker compose up -dA workspace then adds the server under Gateway, then Providers, with its OpenAI-compatible base URL, such as http://ollama:11434/v1 for a Compose service. On Helm the same switch is agentccGateway.allowPrivateProviderURLs. The docs say to turn it on only where everyone who can add a provider is trusted.
In the Docker Compose install, the gateway posts its request logs to the platform API in the same deployment, so they stay inside your perimeter. Deployment telemetry never sends traces, prompts, completions or datasets. To close it down completely, set FUTURE_AGI_TELEMETRY_DISABLED=true and block api.futureagi.com at the firewall, as the docs recommend.
For an offline install, set LITELLM_LOCAL_MODEL_COST_MAP=True so each API and worker uses its bundled price list, and pre-seed the model server’s cache with HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 set so models load from local storage. In the browser, the UI loads fonts, icons and its code editor from public hosts, so allow those hosts through your proxy.
On Helm, global.airgap=true applies the same checklist: telemetry off except one registration attempt that the api.futureagi.com block stops, the bundled price list, no model downloads, and a paused licence heartbeat on Enterprise-licensed installs unless license.heartbeat is true. The gateway source is in the Future AGI repo on GitHub.
Keeping the claims assistant in the country, one request at a time
The claims assistant stays in the country when every model request has one exit and that exit only knows in-country destinations. In practice that means a self-hosted gateway at the single egress point, with an allow list that names only in-country providers.
Back at the insurer, the overflow route now points at a second in-region model server. The trace export lands in the insurer’s own observability store. The team key for the claims assistant lists two providers, both in the country, and the next load test sends nothing across the border.
Take the four-layer table into your next launch review, fill in every row, then write the egress rule that leaves the gateway as the only way out. To build it on Agent Command Center, set up the first team key from the virtual keys docs.
Frequently Asked Questions
What is sovereign AI infrastructure?
What is the difference between data sovereignty and data residency?
What is the difference between sovereign cloud and private cloud?
What is the difference between sovereign AI and private AI?
Why is sovereign AI important?
Compare Meta Muse and OpenAI Dots on setup, approvals, privacy, price and regions, from each company's own pages, to see which always-on agent fits your budget.
Shadow AI in engineering starts with personal API keys, unapproved coding agents and unreviewed MCP servers. Find it in your logs, then build a sanctioned path.
Owning your intelligence is a per-layer choice, not a slogan. Here is what AI ownership means across model, data, and evaluation, and where lock-in bites.
