Simulate your agents in an Environment: up to 1,000 realistic scenarios from one chat, then debug failures goal by goal

Simulate your agents in an Environment

Build a simulation environment for your voice or chat agent from its GitHub repository, from uploaded code, or from an agent hosted on Vapi or Retell AI, then run the whole simulation from a single chat. Future AGI generates up to 1,000 scenarios with simulated users who sound and act like real people, and scores every conversation against its sub-goals and as many system or custom evals as you need. When conversations fail, click Debug failures: Future AGI reads every call in the run and shows which goals your agent broke, the different ways each goal broke, and the exact calls behind each one, so you know what to fix next.

What you get

• Start from what you have. Connect your agent from its GitHub repository (public or private), upload its code, or connect an agent hosted on Vapi or Retell AI, or any voice agent with a phone number. A new entry page lists your environments, and if a pre-flight check fails, it tells you what is missing and how to fix it.
• Run it all from one chat. Describe what you want to test, and the chat connects your agent, builds the environment, generates scenarios, runs them and scores the results, with progress updates as it goes. Actions with real consequences always ask for your confirmation first.
• Scale to 1,000 scenarios. Generate up to 1,000 scenarios for one agent, and see how well they cover each axis (persona, accent, background noise and adversarial intensity), reported separately instead of as one combined number. You can choose an under-18 caller age group, and ask the chat to add more scenarios to a suite that is already finished.
• Simulated users who sound like real people. They use natural, varied phrasing, push back and grow more urgent, and bring a different personality to each conversation, so repeated runs of the same scenario don’t play out the same way. On voice calls, they also interrupt mid-sentence, with background noise that fits where they are.
• Catch flaky behavior. Run each scenario 3 or 5 times, and the run’s analytics show which scenarios flipped between passing and failing across those runs.
• Score the whole trajectory. Every conversation is scored against its sub-goals, not just its final reply. Add as many evals as your use case needs, system or custom, including ten new built-in evals for voice and chat agents: identity verification, consent before taking an action, AI-disclosure compliance, unclear audio, persona consistency, system-prompt leakage, jailbreak resistance, call openings, questions the agent can’t answer, and natural conversation. You can add an eval to a finished run without changing other results, and a removed eval keeps its history.
• Debug failures, goal by goal. On a finished run, click Debug failures. Future AGI reads every call and sums up the run, for example: “Your agent broke 2 goals on 14 of 40 calls.” For each broken goal, you see what a pass looks like and the different ways the goal broke, each with a count, and View opens exactly those calls, so you don’t have to dig through call recordings and transcripts to find the pattern. Issues seen on only one call are listed separately.
• See what happened in each conversation. Tool calls appear in the transcript at the point where they happened. Group a run’s calls by use case, sub-goal, accent, age, attack, task or status, filter them by use case, sub-goal or status, and download call analytics as a PDF or call data as a CSV. Infrastructure problems are tracked separately, so they never count against your agent’s score.
• Fix it and check again. Update your agent, click Run again to rerun the same scenarios, and confirm the fix holds.

How to turn it on

In Simulate, start a new environment from the entry page by choosing a GitHub repository, uploaded code or a hosted agent. Then open the environment’s chat, describe what you want to test, and follow along as the run is built, executed and scored. When the run finishes, click Debug failures at the top of the run page.

Why this matters

A failing score only helps if you can tell what to change. Debug failures turns a run’s failed calls into a short list of broken goals and the ways they broke, each linked to the calls that show it. You go from a red number to a specific fix, then rerun the same scenarios to confirm it worked.

Who this is for

Teams building voice and chat agents who want thorough, realistic testing before release, without building the testing setup themselves.

A deeper Error Feed that finds the real defect

The Error Feed, which surfaces recurring problems in your deployed agents, now investigates each problem in depth instead of scanning each trace only once. It compares what the user asked for with what the agent actually did, backs every finding with evidence from the trace, and double-checks its conclusions before showing them. Related failures are grouped by their underlying defect, and each issue shows how often it happens, what it affects, how severe it is, and where to look for a fix.

Improvements

Simulate

Interruption counts and stop latency on every call. Call lists and run summaries now show how many times each call was interrupted and its stop latency, so you can compare conversation quality across calls without opening each one.

A more useful run page. You can stop a simulation run while it’s in progress, from the runs table or the run page, and builds and runs show accurate in-between states such as Finalizing and Cancelling. A run’s calls table can show each call’s Situation, Sub-goals and Ideal outcome in their own columns (turn them on from Columns), and you can step to the previous or next call straight from the call drawer, for both voice and chat. The Runs tab lists every run of an environment, 10 per page, and the graph can plot your latest 1 to 5 runs.

An environment chat that keeps going. The environment chat now recovers on its own from brief platform interruptions, and stops with a clear error if a run’s saved files are missing. Scenarios, contracts and sub-goals you add or edit in chat show up in the Scenarios tab and are used in the next calls.

Scenario edits carry into the next run. Editing or deleting a scenario in the Scenarios tab leaves every other scenario untouched, and your changes are used in the next run.

Call recording and transcript improvements. Transcript highlighting now stays in sync with the audio during playback, the agent’s last words are kept when a call ends, outbound calls wait for someone to answer, and the simulated caller waits for a reply after asking a question.

Observe

Observe keeps up with large projects. On large projects, filtered lists of spans, traces, sessions and users, as well as dashboard graphs, now show the results found so far with an option to load more. Observe also recovers smoothly from brief connection drops, and the Voice calls list loads reliably in voice projects.

Consistent Users filters and latency charts. Filtering the Users list by fields such as status or model now returns the right users, latency charts show the same statistic with or without a filter, and the Users grid stays visible while it refreshes.

Error Feed stays responsive on large error clusters. The Error Feed stays responsive when you investigate very large clusters of related errors.

Agents

MCP server tools that always match the API. The Future AGI MCP server’s tools for datasets, evaluations and observability are now generated directly from the platform’s API, so their inputs and behavior always match what the API supports.

Falcon AI saves the dashboards you build in chat. Ask Falcon AI to build a dashboard in the general chat, and it now saves the dashboard for you.

Gateway and API

Route Claude-format requests to Gemini on Google Vertex AI. The Gateway can now serve Claude-format requests, including those from the Claude Agent SDK, using Google’s Gemini models on Vertex AI. Keep your Anthropic-style request code and run it against Gemini on Vertex AI through the Gateway.

Google Vertex AI as a Gateway provider. You can now add Google Vertex AI as a provider on the Gateway, with a deployment guide to help you set it up.

Gateway logs and analytics by application and service. Filter Request Logs, and group usage, cost and error analytics, by application and service as well as by provider and model. These values come from the metadata you attach to each request, for example an x-agentcc-metadata header with application and service fields.

Private model servers for self-hosted Gateways. On self-hosted installs, you can point the Gateway at local or private model servers such as Ollama or vLLM by turning on an explicit opt-in setting.

Provider keys stay with your organization’s endpoint. On every Gateway, cloud or self-hosted, an organization’s provider keys and requests are sent only to that organization’s own configured endpoint. Gateway API keys created in the UI now also survive Gateway restarts.

OpenAI-compatible providers without a /v1 path. Providers whose endpoints don’t use a /v1 path, such as Perplexity, now work through the Gateway, because provider endpoint paths are now configurable and resolved consistently.

Gateway analytics and request handling refinements. Requests from an application labeled “Other” now keep their own group in grouped analytics, request schema properties are preserved, and resolved model aliases are passed on to the provider.

Platform

A lighter self-hosted Future AGI, plus a signed Helm chart. Self-hosting is now lightweight by default. A Standalone setup runs in 3 containers and uses about 1 GB of RAM when idle, a Distributed setup scales out for larger teams, and a signed Helm chart published to an OCI registry is available for Kubernetes, with production and enterprise options such as autoscaling, external secrets and air-gapped installs. Third-party integrations stay off until you add their keys.

Smoother self-hosted install and first-run setup. A fresh self-hosted install now takes you straight to creating the first account. If a first-run check fails, such as object storage or TLS, the setup screen explains why and how to fix it, and lets you continue in a test-flight mode or contact support. The backend also finishes in-progress requests before it shuts down during restarts and deploys.

A stable object storage build for self-hosting. Self-hosted deployments now use a fixed build of the last community release of MinIO, so object storage stays on a known, stable version.