Home / Changelog

Changelog

Weekly digests of everything we ship. New features, improvements, and fixes to the Future AGI platform.

Follow
W42 Oct 13 – Oct 26, 2026

Simulate your LiveKit voice agents in an Environment, a voice-native Error Feed, and more

Connect a LiveKit voice agent you already run to an Environment, alongside Vapi and Retell. The Error Feed is now voice-native, run results tables load up to 98% faster, run analytics add nine call and voice metrics, you can edit and re-grade evaluations on a finished run, your AI assistant can do more through the MCP server, and more.

Simulate Agents New

Simulate your LiveKit voice agents in an Environment

Connect a LiveKit voice agent you already run to an Environment, the same way you connect a Vapi or Retell agent. Enter the agent's registered name, your LiveKit URL, API key and secret, and its system prompt. Preflight checks run before the environment builds, and in every simulation a simulated caller talks to your agent and puts it to the test.

6 platform releases shipped this fortnight
9 new call and voice metrics in run analytics
11 new MCP server tools
Monitor Agents New

A voice-native Error Feed

The Error Feed is now voice-native. It investigates a voice call from the audio of the call, along with the voice provider's own warnings and errors, not just the transcript, so it can tell a silence or dropped connection caused by the provider apart from a fault in your agent, without burning a hole in your pocket. Grouping also scales to projects with tens of thousands of findings, and personal information in stored conversations is redacted.
Simulate Monitor Improved

Nine more call and voice metrics in run analytics

A simulation run's analytics header now shows nine more call and voice metrics, including connected rate, agent latency, words per minute, talk ratio, and average duration and turns. The Detailed Analytics toggle controls the reliability section, and analytics stay visible even when a run has no reliability data.
Simulate Improved

Run results tables load up to 98% faster

On a 500-call run, the run results table now loads in about 0.3 seconds once it has loaded before, down from as long as 25 seconds, and a first load is about 3.5 times faster. The filter panel uses the same "Use case" and "Sub-goal" labels as the rest of the run, lets you exclude a use case, sub-goal or status with an "Is not" filter, and its status chips reflect your other active filters.
Simulate Improved

Filter a run's calls by status

Click a status chip (Queued, In progress, Passed, Failed, Inconclusive or Error) to filter a run's calls, with matching counts in every group, and see each sub-goal's verdict in run details.
Simulate Fixed

A cleaner Calls tab

On a run's Calls tab, the toolbar wraps cleanly at narrow widths, calls you open from a Debug failures result appear right away with their groups expanded, and deleting a call updates the run's totals immediately.
Simulate Improved

Simulated callers that behave more like real people

Simulated callers confirm a recap briefly instead of reading it back in full, ask for a repeat when the audio is unclear, and follow up on questions your agent left unanswered. Generated scenario suites also include a minimum share of adversarial cases and give every caller a unique name.
Simulate Fixed

Simulated callers read codes out clearly

Simulated callers now read booking codes, order numbers and other letter-and-digit references one character at a time, the way people do on the phone, so your agent is tested on hearing the exact code.
Simulate Improved

More reliable simulated calls

Simulated calls run more reliably, so fewer calls end early because of a model provider issue.
Simulate Fixed

Chat runs follow your scenario settings

When you change a scenario's maximum number of turns, chat runs now stop at that number. The environment chat clears a whole scenario suite only when you explicitly ask it to, and hosted chat runs for agents that reply through a callback start reliably.
Simulate Fixed

A resizable, collapsible builder chat

The environment builder's chat panel can be collapsed and resized, remembers the width you choose, and wraps a long unbroken string such as a URL instead of scrolling sideways.
Evaluate Simulate Improved

Edit and re-grade evaluations on a finished run

Edit a single evaluation from a run, pre-filled, and re-run just that evaluation on the run's calls, or re-grade the evaluations you pick. The same edit is available from the environment's Evaluations tab and from an evaluation column's header on the run table, you can show or hide each evaluation's column, and an evaluation added after the run finishes appears without a reload.
Evaluate Fixed

Feedback keeps few-shot examples in sync

Updating a dataset, experiment or eval-playground feedback submission replaces its full set of few-shot examples, and removing a submission removes its examples from retrieval immediately.
Evaluate Guard Fixed

Stronger security for evaluation prompts

Security of prompt-template rendering in the JSON Validation evaluation has been strengthened.
Monitor Evaluate Improved

Set alerts for score evals

You can now set alerts for score evals. An alert tracks the eval's average score against your threshold, so it fires as configured. The preview chart works before you pick a label, and threshold changes refresh it immediately.
Monitor Fixed

Sharper dashboard charts

Widgets with no axis bounds fit the data tightly, the Out of Bounds toggle clips or widens as labeled, and a right-hand axis scales its series together. Stacked line charts show continuous lines and accurate tooltips, and minute-granularity widgets stay responsive.
Monitor Fixed

Observe polish

A filter field or value that is cut off shows its full name in a tooltip. On a saved tracing view, editing a custom date range or a compare-mode filter shows the Save view control so you can update the view.
API AgentCC Monitor Improved

Agent Command Center: more control over logs, configuration and alerts

Add a Request Logs column for any custom property your organization declares. Configurations are validated when you save them, so one that can't apply is rejected with the exact field and reason while the previous version stays active. Set a model timeout per organization, override it per request with `x-agentcc-timeout` or `x-agentcc-request-timeout`, and get latency alerts on a window's average latency, sent to the channels you choose.
API AgentCC Improved

Agent Command Center: broader route and provider coverage

Model overrides and conditional routes work on the Anthropic-native, Gemini-native and audio routes, and streamed speech is budget-checked and logged like every other request. `tool_choice: "none"` withholds tools from Anthropic models, transcription honors `response_format` (text, srt, vtt, verbose_json), Gemini text-to-speech on Vertex returns the format you request, models that need the Responses API are tagged in Edit Provider, and custom OpenAI-compatible or Anthropic-format providers use the correct `/v1` path.
API Simulate Fixed

Clearer statuses in the simulation API

The run and call-execution API responses return `queued` or `in_progress` while a call is active and `error` when a call failed or was cancelled, and add a `sub_goal_results` field with each sub-goal's verdict. The existing `sub_goals` field stays for compatibility.
Agents SDK Improved

Do more from your AI assistant through the MCP server

Connect your AI assistant to the Future AGI MCP server and it can now work through annotation queues, edit and clean up datasets, check fired alerts, review alert and guardrail rules, and inspect Agent Command Center sessions, right from the assistant you already use.
Agents Fixed

Smoother prompt editing

Select All inside a prompt message selects only that message's text and treats an attached image, audio or PDF as one block. Copy and cut give plain text, undo and redo restore attachments, and attachments keep their filename and size after you save.
Platform Guard Improved

Stronger SAML single sign-on

Security for SAML single sign-on has been strengthened. Sign-in now starts from the login page's "Sign in with SSO" option, and admins who want an identity-provider app tile can point it at the Future AGI SSO login page. A SAML session can't change two-factor, passkey, recovery-code or API-key settings, approve MCP OAuth apps or reset a password.
Read full digest
W40 Sep 29 – Oct 12, 2026

Simulate your agents in an Environment: up to 1,000 realistic scenarios from one chat, then debug failures goal by goal

Build a simulation environment for your voice or chat agent from its GitHub repository, from uploaded code, or from a Vapi or Retell AI agent, then run it all from one chat: up to 1,000 scenarios with simulated users who sound like real people, scoring against each conversation's sub-goals and as many system or custom evals as you need, and a Debug failures view that shows which goals your agent broke and how. Also this fortnight: a deeper Error Feed that groups failures by their real defect, Claude-format requests routed to Gemini on Google Vertex AI, a lighter self-hosted setup with a signed Helm chart, and more.

Simulate Agents Evaluate New

Simulate your agents in an Environment

Build a simulation environment for your voice or chat agent from its GitHub repository, from uploaded code, or from an agent hosted on Vapi or Retell AI, then run the whole simulation from a single chat. Future AGI generates up to 1,000 scenarios with simulated users who sound and act like real people, and scores every conversation against its sub-goals and as many system or custom evals as you need. When conversations fail, click Debug failures: Future AGI reads every call in the run and shows which goals your agent broke, the different ways each goal broke, and the exact calls behind each one, so you know what to fix next.

15 platform releases shipped this fortnight
1,000 scenarios you can now generate for one agent
10 new built-in evals for voice and chat agents
Monitor Agents New

A deeper Error Feed that finds the real defect

The Error Feed, which surfaces recurring problems in your deployed agents, now investigates each problem in depth instead of scanning each trace only once. It compares what the user asked for with what the agent actually did, backs every finding with evidence from the trace, and double-checks its conclusions before showing them. Related failures are grouped by their underlying defect, and each issue shows how often it happens, what it affects, how severe it is, and where to look for a fix.
Simulate Monitor Improved

Interruption counts and stop latency on every call

Call lists and run summaries now show how many times each call was interrupted and its stop latency, so you can compare conversation quality across calls without opening each one.
Simulate Improved

A more useful run page

You can stop a simulation run while it's in progress, from the runs table or the run page, and builds and runs show accurate in-between states such as Finalizing and Cancelling. A run's calls table can show each call's Situation, Sub-goals and Ideal outcome in their own columns (turn them on from Columns), and you can step to the previous or next call straight from the call drawer, for both voice and chat. The Runs tab lists every run of an environment, 10 per page, and the graph can plot your latest 1 to 5 runs.
Simulate Fixed

An environment chat that keeps going

The environment chat now recovers on its own from brief platform interruptions, and stops with a clear error if a run's saved files are missing. Scenarios, contracts and sub-goals you add or edit in chat show up in the Scenarios tab and are used in the next calls.
Simulate Fixed

Scenario edits carry into the next run

Editing or deleting a scenario in the Scenarios tab leaves every other scenario untouched, and your changes are used in the next run.
Simulate Fixed

Call recording and transcript improvements

Transcript highlighting now stays in sync with the audio during playback, the agent's last words are kept when a call ends, outbound calls wait for someone to answer, and the simulated caller waits for a reply after asking a question.
API AgentCC SDK Improved

Route Claude-format requests to Gemini on Google Vertex AI

The Gateway can now serve Claude-format requests, including those from the Claude Agent SDK, using Google's Gemini models on Vertex AI. Keep your Anthropic-style request code and run it against Gemini on Vertex AI through the Gateway.
API AgentCC Improved

Google Vertex AI as a Gateway provider

You can now add Google Vertex AI as a provider on the Gateway, with a deployment guide to help you set it up.
API AgentCC Monitor Improved

Gateway logs and analytics by application and service

Filter Request Logs, and group usage, cost and error analytics, by application and service as well as by provider and model. These values come from the metadata you attach to each request, for example an `x-agentcc-metadata` header with `application` and `service` fields.
API AgentCC Platform Improved

Private model servers for self-hosted Gateways

On self-hosted installs, you can point the Gateway at local or private model servers such as Ollama or vLLM by turning on an explicit opt-in setting.
API AgentCC Guard Improved

Provider keys stay with your organization's endpoint

On every Gateway, cloud or self-hosted, an organization's provider keys and requests are sent only to that organization's own configured endpoint. Gateway API keys created in the UI now also survive Gateway restarts.
API AgentCC Fixed

OpenAI-compatible providers without a /v1 path

Providers whose endpoints don't use a `/v1` path, such as Perplexity, now work through the Gateway, because provider endpoint paths are now configurable and resolved consistently.
API AgentCC Fixed

Gateway analytics and request handling refinements

Requests from an application labeled "Other" now keep their own group in grouped analytics, request schema properties are preserved, and resolved model aliases are passed on to the provider.
Agents SDK Improved

MCP server tools that always match the API

The Future AGI MCP server's tools for datasets, evaluations and observability are now generated directly from the platform's API, so their inputs and behavior always match what the API supports.
Agents Monitor Fixed

Falcon AI saves the dashboards you build in chat

Ask Falcon AI to build a dashboard in the general chat, and it now saves the dashboard for you.
Monitor Improved

Observe keeps up with large projects

On large projects, filtered lists of spans, traces, sessions and users, as well as dashboard graphs, now show the results found so far with an option to load more. Observe also recovers smoothly from brief connection drops, and the Voice calls list loads reliably in voice projects.
Monitor Fixed

Consistent Users filters and latency charts

Filtering the Users list by fields such as status or model now returns the right users, latency charts show the same statistic with or without a filter, and the Users grid stays visible while it refreshes.
Monitor Fixed

Error Feed stays responsive on large error clusters

The Error Feed stays responsive when you investigate very large clusters of related errors.
Platform Improved

A lighter self-hosted Future AGI, plus a signed Helm chart

Self-hosting is now lightweight by default. A Standalone setup runs in 3 containers and uses about 1 GB of RAM when idle, a Distributed setup scales out for larger teams, and a signed Helm chart published to an OCI registry is available for Kubernetes, with production and enterprise options such as autoscaling, external secrets and air-gapped installs. Third-party integrations stay off until you add their keys.
Platform Improved

Smoother self-hosted install and first-run setup

A fresh self-hosted install now takes you straight to creating the first account. If a first-run check fails, such as object storage or TLS, the setup screen explains why and how to fix it, and lets you continue in a test-flight mode or contact support. The backend also finishes in-progress requests before it shuts down during restarts and deploys.
Platform Fixed

A stable object storage build for self-hosting

Self-hosted deployments now use a fixed build of the last community release of MinIO, so object storage stays on a known, stable version.
Read full digest
W38 Sep 1 – Sep 14, 2026

Simulate voice and chat agents in a rich environment, buy Future AGI on Google Cloud Marketplace, and outbound Retell voice simulation

Point Future AGI at your agent's code and it builds a full simulation environment with real tool calls, deterministic evals, and an agent-as-judge. Future AGI is now on Google Cloud Marketplace, so you can buy it through your Google Cloud account and have the spend count toward your commitment where your agreement allows. Outbound Retell voice agents can now be simulated and observed through the Agent Learning Kit, Retell calls are captured into Observe by polling, a fired alert opens its own traces, and more.

Simulate Agents Evaluate New

Simulate voice and chat agents in a rich environment

You can now build a full simulation environment for your hosted agents, or any voice or chat agent, by pointing Future AGI at the agent's code, either by sharing a GitHub repository or uploading the codebase. Future AGI generates varied scenarios that exercise the agent with real tool calls rather than stubbed responses, adds deterministic evals automatically, and applies an agent-as-judge.

Google Cloud Marketplace listing, billed through your Google Cloud account
2 new ways to simulate: from your code, or over a real Retell call
Platform New

Buy Future AGI on Google Cloud Marketplace

Future AGI is now available on Google Cloud Marketplace. Subscribe from the Google Cloud console and Google bills you directly, so the spend runs through your Google Cloud billing account and can count toward your Google Cloud commitment. Sign up straight from the listing, including with Continue with Google, and your usage is metered and billed through Google. Plan changes and cancellations you make in the Cloud console carry through to your Future AGI organization automatically. Available on the Pay-as-you-go, Scale, and Enterprise plans.
Simulate Agents New

Simulate outbound Retell voice agents and enable observability for them

You can now simulate an outbound Retell voice agent through the Agent Learning Kit and enable observability for it in the same flow. Your agent places real outbound calls, one run covers every scenario, and transcripts, recordings, and evals land in the platform exactly like a native simulation, while the agent's calls show up in Observe without a separate setup.
Monitor Simulate Improved

Retell voice calls captured by polling

Retell voice calls are now captured by polling Retell directly instead of relying on inbound webhooks, so calls are ingested reliably even when a webhook is missed, and high call volumes are covered page by page without gaps.
Monitor Improved

Open a trace from a fired alert with its context

Following View Trace from a fired alert now carries the alert's own time window and filters into the trace view, so you land on the traces the alert was about instead of an unscoped list.
Monitor Evaluate Improved

Attributes search, project sharing, and eval mapping in Observe

The Attributes tab in the trace and voice drawers now filters as you type, project sharing works as expected, and the eval-task variable mapping screen is smoother to work through.
Monitor Evaluate Improved

Eval charts honor Dataset and Eval Source filters

Dashboard widgets built on an eval metric now apply the Dataset and Eval Source filters to the query, so you can build a per-environment or per-source eval trend from a single eval template.
Monitor Evaluate Fixed

Eval alert thresholds show their unit

An eval alert threshold is now labeled as a 0-to-1 fraction, so the value you enter lines up with the metric you are alerting on.
Evaluate Monitor Improved

More reliable continuous eval tasks

Eval tasks that run continuously or over large historical windows now pick up every candidate row within the run's budget, even on dense windows, and continuous tasks stop re-running failures that have already settled.
Evaluate Fixed

System eval bindings keep their configuration

Editing the dataset bindings on a system evaluator keeps the rest of its configuration intact across saves.
Monitor Fixed

Dashboard metric selection persists on save

Unselecting and reordering the metrics on a dashboard widget and saving now sticks, so reopening the editor shows the selection you made.
API Fixed

Gateway provider dialog hardened

The Gateway provider dialog saves cleanly when a timeout value is present, surfaces validation errors clearly instead of silently blocking Save, and shows full session IDs.
Agents Evaluate Fixed

Falcon defaults to an agent eval

Creating an evaluation through Falcon without specifying a type now creates an agent eval by default.
Evaluate Monitor Fixed

Annotation tracing filter supports contains

Filtering annotations on tracing with the contains operator now returns the matching rows.
Platform Fixed

Open-source UI fixes

A round of open-source-build polish, including dark-theme eval-score colors and a working Help button.
Read full digest
W36 Aug 25 – Sep 7, 2026

Richer AI Gateway traces, full gateway coverage for the Anthropic and Google GenAI SDKs, seven insurance-agent evals, and a wave of Observe polish

AI Gateway traces now carry your caller metadata, request-body fields, and allowlisted headers as span attributes, and calls made with the Anthropic SDK or Google GenAI SDK get the same traces, cost attribution, budgets, and rate limits as OpenAI-format calls. Anthropic server tools work through the OpenAI-format endpoint, seven insurance-agent system evals join the catalog, the Error Feed labels each cluster by source, and more.

API AgentCC Monitor New

Gateway traces now carry your own metadata, request fields, and headers

The AI Gateway now exports caller metadata, selected request-body fields, and allowlisted headers as span attributes, so gateway traces carry more of the context you need to filter and analyze them. Your own dimensions become filterable attributes on the trace instead of an opaque blob.

7 new insurance-agent system evals in the catalog
2 more SDKs with full gateway coverage
API AgentCC Monitor Guard New

Anthropic and Google GenAI SDK calls get full gateway coverage

Calls made through the Gateway with the Anthropic SDK (`/v1/messages`) or the Google GenAI SDK now run through the same pipeline as the rest of the Gateway: full traces, cost attribution, and your budget and rate-limit controls. These two endpoints are now in line with the OpenAI-format endpoint.
API AgentCC Agents Improved

Anthropic server tools on the OpenAI-format endpoint

Anthropic server-side tools, such as web search, now work when called through the Gateway's OpenAI-format endpoint, so Claude searches when you ask it to.
Evaluate Simulate Agents Improved

Seven insurance-agent system evals

Seven built-in system evaluators for insurance agents are now in the catalog, ready to add to eval and simulation runs without writing custom criteria. They cover misselling, advice boundaries, lead qualification, claim intake, intake field accuracy, and voicemail detection and handling on the call audio.
Monitor Evaluate Improved

Every Error Feed cluster shows its source

Every cluster card in the Error Feed now shows where it came from, the built-in error taxonomy or a specific eval task you configured, so it is clear at a glance which findings are surfaced by Future AGI and which come from your own evals.
Monitor Improved

Custom attribute filters: every key, faster matches

The attribute discovery behind filter dropdowns now reads keys exhaustively rather than sampling rows, so projects with many distinct span attributes see the full list to filter on, and custom attributes stay in the picker. Exact-value filters on custom attributes in Observe and on dashboard graphs now resolve faster and more consistently, including on unicode values.
Evaluate Improved

Re-run only the errored evals, or the whole task

From an eval task you can now re-run just the evals that errored, or re-run the entire task, from the task detail.
Monitor Improved

Total Tokens on the Sessions grid

Total Tokens is available as a filter and column on the Sessions grid, and the column toggle now persists.
Evaluate Fixed

Eval task detail fixes

Opening an eval task whose saved mapping value is a JSON object now loads cleanly, with the value handled as invalid input instead of taking down the dashboard. The eval-task time window shows the preset you picked and labels custom ranges correctly.
Evaluate Fixed

Dataset reasoning toggle keeps its state

In a dataset, the Show/Hide Reasoning option on an eval column's menu now remembers your choice, so reasoning you hide stays hidden when the column expands.
Monitor Fixed

Dashboard polish

Dashboard pies now render one chart per metric and only show a pie when a breakdown is set, so the chart matches the data. Chart line colors are now consistent for a series across a widget, and a widget's description is shown on the individual widget.
Evaluate Fixed

Dataset column reorder and hide stick

Reordering or hiding dataset columns now takes effect and stays that way.
Monitor Improved

Alerts: faithful editing, working email links, clearer charts

Reopening a saved alert now shows exactly what was stored, including the metric, threshold, and frequency, so saving from the editor keeps your real configuration. The Open monitor button in alert emails now opens the alert that fired through a region-correct link, and alert config charts default to a 7-day window with readable tooltips in dark mode.
Evaluate Monitor Fixed

Ground Truth visibility on Observe evals

When Ground Truth is enabled on a template but cannot be applied to a given Observe eval run, that is now surfaced to you rather than the run proceeding silently without calibration.
Monitor Fixed

Clearer sampling rate control

The Error Feed sampling-rate control now states what it controls, so it is distinct from a similarly labeled eval setting.
Agents Fixed

Falcon shows a connector's discovered tools

The Falcon Customize panel now lists the tools discovered from a connected connector, so you can see exactly what an agent can call.
Monitor Evaluate Fixed

Observe graph metric picker scoped to the project

The metric picker on the Observe graph, across the Traces, Spans, Sessions, Users, and Compare views, now lists only the evals and annotation labels in the current project rather than every one in the organization.
Monitor Simulate Fixed

Voice Observe: quick filters and distinct saved views

Quick filters are now available in voice Observe, so you can narrow to the calls you care about without building a full filter expression. Voice observability now rejects a saved view whose name is already in use, so your saved views stay distinct.
Platform Fixed

Open-source build fixes

A large batch of fixes for the open-source build: gateway API-key field styling, the agent-scenario dark-theme popup, action-button styling, default-tag visibility, Hugging Face import, datapoint row selection, and more.
Read full digest
W34 Aug 11 – Aug 24, 2026

Send AI Gateway traces to any OpenTelemetry backend, voice agents over WebRTC, and simulations from your own harness

The AI Gateway can now export its traces over OTLP/HTTP to any backend you configure, with the request and response attached when you opt in. Voice agents run as web calls over WebRTC, simulations from the Simulate SDK and Agent Learning Kit post back into Future AGI, enterprise code joins the open-source repository, Error Feed clustering gets sharper, and more.

API Monitor New

Send AI Gateway traces to any OpenTelemetry backend

The AI Gateway can now export a span for each call it handles, or a sampled share if you set a sample rate, to any OTLP/HTTP endpoint you configure, authenticated with headers you set. Turn on include_bodies and each span also carries the request messages and the model response, streamed or not, redacted with your organization's privacy rules before it leaves the gateway. Chat, image, audio, embedding, and retrieval calls are all covered, and large payloads like base64 images and embedding vectors are recorded by size, not copied.

5 gateway endpoint types exported over OTLP/HTTP
2 SDKs that post simulations back into Future AGI
Simulate Agents New

Voice agents now run as web calls over WebRTC

Voice agents can run as web calls over WebRTC, not just telephony. An explicit WebRTC versus Telephony toggle replaces the old behavior that inferred the transport from whether a phone number was present, and setting up an outbound WebRTC agent no longer requires a phone number or a telephony API key.
Simulate SDK Monitor Improved

Post simulations from the Simulate SDK and Agent Learning Kit into Future AGI

Voice simulations run through the Simulate SDK or the Agent Learning Kit can now be posted back to Future AGI and render exactly like a native voice simulation: transcript, recording, turn-by-turn detail, words per minute, CSAT, tokens, and cost. Results from your own harness are fully observable in the platform.
Platform Guard Agents Improved

Enterprise code joins the open-source repository

Future AGI's enterprise code now sits in the same open-source repository as the core, behind a license. Capabilities like the Cluster RCA agent, guardrails, and agent optimization build from one place alongside the Apache 2.0 core, so self-hosted and licensed deployments come from a single source.
Monitor Improved

Sharper error clustering in the Error Feed

The Error Feed's clustering engine reads each turn in full context, groups related failures more tightly, and titles every cluster from the shared pattern across its traces. Each cluster gives you a cleaner, more accurate picture of what is going wrong and how widespread it is.
API Evaluate Platform Improved

Renamed and retired models are handled for you

Future AGI now tracks when a model is renamed or retired from the catalog. Historical runs that reference a retired model continue to load, the model is flagged with a deprecated badge, and starting a new run on an unavailable model returns a clear message pointing you to a supported one.
Evaluate Improved

Re-run a task from its detail header

The task detail header now has a Re-run button, so you can re-run a task without going back to the list.
Evaluate Monitor Fixed

Eval scores populate from structured outputs

Evaluations that return a structured output now materialize the eval score, so scores show up on the dashboard where you expect them.
API Agents Fixed

Gateway and MCP reliability

A round of Gateway and MCP hardening: tool-schema validation runs on empty arguments, large MCP messages are supported, rate-limited trace exports are retried, and span bodies are truncated on a character boundary so multi-byte text is never cut mid-character.
Evaluate Fixed

Annotation queue actions and label restore

The Archive action in the queue actions menu is clearly visible in light theme, and restoring an archived annotation label works for users whose current organization differs from their original one.
Evaluate Fixed

Editable dataset filter chips

Filter chips on datasets are now editable, so you can adjust a filter in place instead of removing and re-adding it.
Evaluate Fixed

Numeric annotation labels accept clean values

Creating a numeric annotation label keeps the min and max fields exactly as you typed them, with no trailing zero trapped in either field.
Platform Fixed

Checkboxes render clearly in every state

Checkboxes now render clearly in every state, including in dark mode and the selected-evaluation checkbox in the Add Evaluation dialog.
Platform Fixed

A fresh self-hosted install comes up cleanly

The root environment example now includes the integration encryption key and the Docker Compose default is corrected, so a fresh self-hosted install starts without manual edits.
Read full digest
W32 Jul 21 – Aug 3, 2026

A Root-Cause Analysis Agent for Error Feed, Observe Loads 50 to 70% Faster, and Claude 5 and the Gemini 3 Family

Error Feed gets a root-cause analysis agent that investigates a failing cluster on its own and lays out the likely cause. Observe load times drop by 50 to 70% on high-volume projects, the newest Claude 5 and Gemini 3 models are live across evaluations and the AI gateway, and editing a custom eval's prompt now saves and runs a new version. Plus custom date ranges, a Users CSV export, read-only access for the Viewer role, Bland.ai voice support across Simulate and Observe, and a larger dataset upload limit.

2 new model families in evals and the gateway: Claude 5 and Gemini 3
25 MB dataset upload limit, up from 10 MB
Bland.ai added as a voice provider across Simulate and Observe
Monitor New

Root-cause analysis agent in Error Feed

Error Feed's new root-cause analysis agent investigates a failing cluster on its own. It works through the traces in the cluster, streams its reasoning live in the Fix tab, and lands on the likely root cause, caching the result so reopening the cluster brings it back instantly. Feed pages also load a lot faster.
Evaluate New

Claude 5 and the Gemini 3 family are live

Add your Anthropic or Google API key and the newest Claude 5 and Gemini 3 models appear in the model picker across evaluations and the AI gateway. Gemini 3.6 Flash, 3.5 Flash, and 3.1 Flash Lite are priced, so requests to them return an accurate per-call cost and land in your usage with the right credit math from the first call.
Monitor Improved

Faster Observe and dashboards on high-volume projects

Observe load times drop by 50 to 70% on high-volume projects. Trace lists, the Users and Sessions tabs, and dashboard breakdown charts stay fast even across long date ranges, because queries read only the data inside your selected window, so a wide breakdown returns in seconds. Results are unchanged, just faster to return.
Simulate New

Bland.ai as an inbound and outbound voice provider

Connect a Bland.ai voice agent as a provider and its production calls, inbound and outbound, come into both Simulate and Observe. You can run simulations against your Bland agents and trace their real calls alongside your other voice providers.
Evaluate Improved

Custom eval prompt edits are versioned

Editing a custom eval's prompt on the dataset page now saves it as a new eval version and runs that version, so the prompt you edited is the one that gets used. Each edit is captured as a version you can pick from the version dropdown.
Platform Improved

Read-only access for the Viewer role

The Viewer role now gets a consistent read-only view across tasks, evaluations, dashboards, and the agent playground. Write actions like Create, Save, Test, Duplicate, and Delete render as disabled, matching the permissions the role already has, so viewers see exactly what they can act on.
Monitor Improved

Custom date ranges on dashboards

The dashboard date bar now opens a Custom option where you pick any start and end date, and every widget updates to that range. The widget editor keeps the same range on reopen with its dates intact, instead of falling back to a preset.
Evaluate Improved

Eval usage tab

A new Eval Usage tab shows how often each eval has run, with the eval version recorded on every execution, so you can see usage over any time window and tie it back to the version that produced it.
Evaluate Improved

Eval runs show live status

Eval tasks now report their lifecycle status across the product, so a run's state is clear while it works. Long-running and continuous eval tasks keep going and reconcile on their own instead of needing a manual nudge.
Evaluate Improved

A clearer way to map variables into evals

You can now click any row to map it into a field instead of typing the path, across tracing, simulation, and dataset variable mapping. The dataset variable-mapping picker also handles nested JSON and array fields, with search that reaches deep fields directly and a tree that stays legible in dark mode.
Monitor Improved

Export the Users tab to CSV

The Observe Users tab now exports to CSV. The full list streams to a file straight from the view you are already looking at, so you can pull every user out for analysis without paging through the grid.
Evaluate Improved

Export experiment results to CSV

The aggregate experiment Summary view now exports to CSV, with download progress and clear success or error feedback, so you can pull a full experiment out for analysis in one click.
Platform Fixed

Larger dataset uploads

The dataset upload limit moves from 10 MB to 25 MB, so realistic prompt and eval datasets upload without hitting the cap.
API Improved

The AI gateway works with more clients and models

The gateway now accepts max_completion_tokens for the latest OpenAI reasoning models, so your existing OpenAI client reaches them without extra configuration. Chat completions always return a content field, so clients written to the OpenAI shape keep working even when a model returns no text. Sub-cent request costs show at full precision in the gateway logs.
Platform Improved

Smoother self-hosted setup

Self-hosting is easier to stand up. A guided first-run setup walks an operator through launching a self-hosted stack, with live infrastructure checks, browser-based signup, and invite links for the team. One Docker setup now serves both self-hosted and internal deployments, sign-in has a guided flow for creating an account and resetting a password, and Optimization and Knowledge Base are available in the open-source build.
Monitor Fixed

More users per page

The Observe Users grid now shows 50 rows by default and adds a rows-per-page selector. A filter reads against the full set rather than a short first page, so counts match what you expect.
Monitor Improved

View a session from the annotation queue

You can now open a session directly from the annotation queue, without leaving it. When a queued trace or span belongs to a session, open the full session inline and jump back to the trace when you are done.
Monitor Fixed

Dashboard polish

Deleting a dashboard or widget now asks you to confirm first. A chart whose data falls outside its configured axis range shows a clear warning instead of a blank canvas, and an empty dashboard shows an empty state rather than a blank page. Filter chips display the selected value names instead of a count, and the global time filter adds 30-minute and 6-hour options.
Platform Fixed

Delete your own prompt templates

Prompt templates you created can now be deleted from the Prompt Templates drawer, with a confirm step and a success message, so your template list stays tidy.
Evaluate Fixed

LLM-as-a-judge editor matches the agent layout

The LLM-as-a-judge editor now uses the same layout as the agent editor, with correct model logos, so editing a judge feels the same as editing an agent.
Read full digest
W24 May 26 – Jun 8, 2026

Perplexity ships Sonar and gpt-5.1 to self-hosted Future AGI, plus polish across Evals, Observe, and Platform

Straight from Perplexity. The team sent a PR into the open-source Future AGI repo with first-party support for every current Perplexity model on self-host. Plus a new customer-agent task-completion eval, faster audio eval runs when there's no feedback to retrieve, filter and column fixes across Observe, and a pricing and usage page that renders correctly.

API New

Perplexity contributes Sonar and gpt-5.1 to self-hosted Future AGI

Perplexity contributes Sonar and gpt-5.1 to self-hosted Future AGI

Five Sonar variants are now in the eval model picker and the AI gateway: `sonar` (fast, search-augmented chat), `sonar-pro` (the new default with 200K context), `sonar-reasoning` and `sonar-reasoning-pro` (chain-of-thought over live web results), and `sonar-deep-research` (long-form, multi-source investigations). Perplexity's Agent API adds `gpt-5.1` for agent workflows with built-in web search and tool use. Add your Perplexity API key as `PERPLEXITY_API_KEY` to enable them on self-host.

6 new Perplexity models on self-host
50 improvements across evals, observe, and platform
1 new built-in customer-agent eval
Evaluate Improved

New built-in eval: customer-agent task completion

Score whether your agent actually finished the job. A new built-in eval, `customer_agent_task_completion`, reads the full conversation and returns Pass or Fail with a reason on whether a customer-facing agent completed the task the user asked for. Useful as a top-line outcome metric on voice and chat support conversations, alongside the evals you run on each individual turn.
Evaluate Fixed

Regex PII detection: default to all five PII types

If you leave `detect_types` empty on the Regex PII Detection eval, it now scans for all five built-in types (emails, phone numbers, SSNs, credit cards, IP addresses) instead of returning nothing. You can still narrow the list per eval.
Evaluate Fixed

Eval type persists correctly when creating LLM and Code evals

Creating a new LLM-as-Judge or Code eval now writes the right eval type to the row, instead of defaulting to a placeholder. Reopening the eval picks up the correct type. Templates and saved evals behave the same.
Evaluate Fixed

Eval config auto-populates from the baseline column

When you add an eval inside an experiment and pick a baseline column, the eval settings (model, criteria, output type) now auto-fill from that baseline. Faster setup when you're rerunning an existing eval on a new run.
Evaluate Fixed

Eval workflow: a batch of refinements

Variables in your prompt highlight when you connect them to data in the test panel. Turning data injection on or off is now a clean toggle. Once you pick how the eval scores (pass/fail, numeric, choices), that choice is locked, but the specific labels, score numbers, and thresholds stay editable. When you upload expected answers as ground-truth data, you can watch the processing progress in real time. `Pass Rate` is now called `Task Completion Rate` across all stats and charts.
Evaluate Fixed

Eval picker: stale data prevented, mapping disabled while columns load

The eval picker no longer flashes stale data when you reopen it after switching projects. Variable mapping rows are disabled while the source columns are still loading, so you can't map to a column that hasn't arrived yet.
Evaluate Fixed

Custom eval URLs: open on default version, ?v= survives, task errors expandable

Opening a custom eval URL without a version parameter now lands on the default version. A `?v=` parameter in the URL is preserved across navigation inside the eval. When an eval task fails, the error panel has a Show more toggle that expands the full message.
Evaluate Improved

Audio eval runs are ~3.5x faster when there's no feedback to retrieve

Evals that use feedback-based retrieval now skip the lookup when there are no feedback rows yet. On audio-heavy projects with no feedback corpus, a single eval run is roughly 3.5x faster (about a 70% drop in time).
Evaluate Fixed

Clear error when an eval can't reach a media URL

When an eval fails because it can't fetch an image or audio file from the URL you mapped, the error message now says so and points at the URL, instead of returning a generic backend error. Easier to fix mapping mistakes and broken links.
Evaluate Fixed

Dataset evals with template variables in instructions no longer crash

Dataset evals whose instructions contain `{{variable}}` placeholders no longer crash trying to resolve those placeholders as IDs. The instructions stay untouched and the eval runs cleanly.
Evaluate Fixed

semantic_list_contains handles numeric expected values

The semantic_list_contains comparator no longer crashes when the expected value is a number (for example, an ID or a quantity). Numbers are coerced to strings before comparison.
Evaluate Fixed

Static few-shot examples reach the LLM eval

Few-shot example rows stored on a dataset now expand into the prompt for the LLM eval, instead of being passed as metadata only. Few-shot prompting on dataset-backed evals works as documented.
Evaluate Improved

Voice call mapping reaches deeper into the raw log

When voice eval mapping can't resolve a field in the structured trace data, it now falls back to the raw call log. More voice rows map cleanly without manual intervention.
Evaluate Fixed

PDF filter chip in the eval picker is populated

Evals that accept PDF input are now tagged so the PDF filter chip in the eval picker actually lists them. The chip previously returned no results; it now lists every PDF-capable eval.
Evaluate Fixed

Confirmation dialog before deleting an eval template

Deleting an eval template now asks for confirmation. Prevents accidental clicks on the row action.
Evaluate Fixed

Pass and Fail chips on trace eval drawer

When an eval returns a Pass or Fail result, the trace eval drawer shows a Pass or Fail chip instead of 100% or 0%. Makes the outcome obvious at a glance.
Evaluate Fixed

Multi-choice eval output type flows through the playground and picker

If an eval is configured as multi-choice, that flag is now propagated through the playground run and the eval picker, so the run is scored as multi-choice end to end.
Evaluate Fixed

Error localiser: Show more works and composite evals are hidden

The Show more toggle on the error localiser now expands the full per-highlight reason for the highlighted span. The error localiser panel is hidden for composite evals, where it doesn't apply.
Evaluate Fixed

Fix-with-Falcon hides on label-field eval rows that already pass

On evals where the result is stored in the label field, the Fix-with-Falcon button no longer shows up on rows that already pass. Cleans up the action menu on passing rows.
Evaluate Improved

Eval usage rollups: date range and session-target row exclusion

The `/eval-task/get_usage/` endpoint accepts a date range so you can scope usage to any window. Rows targeting sessions are now excluded from the per-trace usage rollup, so the count matches what you see in the UI.
Evaluate Fixed

Data injection on system evals checks your variable mappings

When you turn on data injection for a system eval, every required variable (like the question and expected answer) now needs to be mapped before you can save. The eval is ready to run as soon as you create it.
Evaluate Fixed

Long eval and label names are readable on hover

Truncated names in the Evals list and truncated labels in the Eval Usage detail panel now show the full text in a hover tooltip. The Evals list grid also fills its container properly instead of clipping.
Monitor Fixed

Trace Name and Span Name filter dropdowns populate suggestions

The Trace Name and Span Name filter pickers on Observe now show the list of names from the current project as suggestions, instead of an empty dropdown.
Monitor Fixed

Span-attached annotations appear in the Annotations filter category

Annotations applied at the span level (not just the trace level) are now included in the metrics endpoint that powers the filter picker. The Annotations category in the filter dropdown is populated for span-only score rows.
Monitor Improved

Text filters are case-insensitive across Observe

Filters on text fields (Trace Name, Span Name, model, provider, span attribute strings) now match regardless of case. Searching for `gpt-4o` finds rows tagged `GPT-4O`.
Monitor Fixed

Trace ID and Span ID filters render as single-select

The Trace ID and Span ID filters now render as a single-select control (a radio, no chips, no +N count) since a span belongs to one trace at a time. Reloading and then editing a saved filter with an ID also returns the right rows instead of zero.
Monitor Fixed

Cleared filters stay cleared after a page refresh

Clearing a filter on Observe now sticks. The filter no longer reappears when you navigate away and come back to the page.
Monitor Fixed

Annotation filters: operator dropdown, chip label, and task filter

Categorical and thumbs-up annotation filters now show the right operator dropdown after the page hydrates. The chip label matches the selected operator. The Tasks filter returns rows on thumbs annotations instead of returning empty.
Monitor Fixed

Add and edit filter chips on Sessions, Users, and User Traces

The plus icon to add a filter chip, and click-to-edit on existing chips, now work on the Sessions, Users, and User Traces tabs. Previously these actions only worked on the Traces and Spans tabs.
Monitor Fixed

Column order persists across auto-refresh on every Observe grid

When you reorder columns by dragging on Traces, Spans, Sessions, or Call Logs, the new order survives the next auto-refresh and stays in sync with the display panel.
Monitor Fixed

Consistent grid theme across Traces, Spans, Sessions, and Users

Header text colour, row hover state, and cell padding are now identical across all four Observe grids (Traces, Spans, Sessions, Users), instead of each grid having a slightly different look.
Monitor Fixed

Call Logs grid: autosize works and Call ID column respects the width you set

On the Call Logs grid (used by voice projects), the autosize columns action now fits column widths to content instead of being a no-op. Widening the Call ID column also shows more of the UUID, instead of clamping at 130 pixels and truncating in the middle of the value.
Monitor Fixed

Hover tooltip on long span-attribute column keys in Live Preview

Long span-attribute paths in the Task Live Preview now show the full key as a tooltip on hover, so you can read the path without resizing the column.
Monitor Fixed

Single loading spinner on the Users page

The Users page no longer shows two stacked loading spinners on initial mount. One spinner appears while data loads, then the rows render.
Monitor Fixed

Save View buttons visible, full tab name preserved

The Save View button is now visible against the dark theme on every Observe tab. Long view names truncate visually with an ellipsis instead of being cut down to a shorter stored string, so the full name stays in state.
Monitor Fixed

Agent Graph and Agent Path: real fullscreen, disabled on voice projects

The fullscreen button on the Agent Graph and Agent Path views now opens a real fullscreen view of the graph, and dead zoom controls on Agent Path are removed. On voice projects, where there's no agent graph to render, both tabs are now disabled with a tooltip explaining why, instead of opening to a black screen.
Monitor Fixed

Tasks list chip hover and popover stay usable

Cell chips on the Tasks list now keep their hover colour while the cursor moves onto the popover, so you can read the popover without it closing the moment you move toward it.
Monitor Fixed

Eval tasks list refreshes while rows are pending or running

The eval tasks list now polls while any row is pending or running, so progress updates appear without a manual refresh. Relative-time cells (`2 minutes ago`) also refresh on each poll instead of going stale.
Monitor Improved

Voice projects: call recording on the Error Feed cluster overview

On voice projects, the Error Feed cluster Overview tab now shows the call recording in place of the agent flow diagram, so you can listen to the call that produced the error.
Monitor Fixed

agent_talk_percentage is filterable on voice traces

`agent_talk_percentage` is now a first-class system metric for voice projects. The filter dropdown matches what you see in the grid (`40%` as a single metric), instead of showing `0.4` as a custom attribute.
Monitor Improved

Faster Users tab and session detail loads

The Users tab now loads instead of timing out on workspaces with high session volume, and session detail no longer returns a 400 on slow queries. Both pages reach the data through a faster path on the backend.
Monitor Fixed

Reliable trace ingestion: custom user IDs and span PK retries

Two ingestion fixes. Traces with custom user IDs from the SDK no longer fail to ingest (the CUSTOM enum value was restored). And bulk span writes tolerate primary-key collisions during retry races, instead of dropping the whole batch.
Monitor Fixed

Voice eval fan-out: longer scheduling window

The scheduling window for voice eval fan-out is now twelve hours, so large voice runs don't lose rows when individual evals queue for a long time before starting.
Platform Improved

Pricing and usage page: correct rounding, units, and free-tier savings

The pricing and usage page now rounds small numbers correctly, shows the right units on per-call costs, and surfaces free-tier savings as a separate line. Sub-1 usage and sub-cent costs render legibly. The legend is visible and the period caption matches the selected window.
Platform Fixed

Faster, more reliable signup with reCAPTCHA

The signup form now waits for reCAPTCHA to be ready before allowing submit, so fast typists no longer hit a `reCAPTCHA not ready` error. The page also preconnects to the reCAPTCHA host, so the script loads sooner.
Platform Fixed

Commit action surfaced on the prompt workbench

The Commit button on the prompt workbench is now on the main toolbar, not buried in the More menu. Saved queries are invalidated right after a commit, so the new version shows up immediately in the version list.
Platform Fixed

Better text contrast across Agent, Persona, Scenarios, Test Detail, Eval Picker, and Error Feed

Secondary text colour has been adjusted across the Agent builder, Persona, Scenarios, Test Detail, Eval Picker, and Error Feed pages so labels and descriptions read clearly against the dark theme.
Platform Fixed

Dataset media fetches no longer hang

Image and audio fetches on dataset rows now have a request timeout, so a slow or unreachable URL fails fast instead of hanging the eval indefinitely.
Platform Fixed

ffmpeg calls time out on malformed audio

ffmpeg subprocess calls inside voice and storage paths now have a timeout, so a malformed audio file can't hang a worker indefinitely.
Platform Fixed

Agentic eval Azure callback handles missing invocation params

The agentic eval Azure callback no longer crashes when `invocation_params` comes back as `None`. Useful when Azure returns a short-form response that omits the params block.
Read full digest
W22 May 12 – May 25, 2026

Evals on Traces and Sessions, Configurable Eval Context, and Polish Across Evals, Observe, and Simulate

Every eval type (LLM-as-Judge, Code, and Agent) can now score at every level: spans, traces, and sessions. Eval setup also gets simpler: turn on context injection and skip variable mapping entirely. The eval reads the context on its own. Plus two new conversation evals: Dead Air Detection (a Code eval at zero LLM cost) and Conversation Hallucination. Plus eval inputs up to 200K characters, partial inputs as warnings, custom dotted paths in mapping, span-level fields and API columns, and polish across Observe and Simulate.

Evaluate New

Score traces and sessions with any eval

You can now run any eval (LLM-as-Judge, Code, or Agent) at three levels: on one step the agent took (a span), on a full run (a trace), or on a whole conversation (a session). Composite evals work at all three too. A composite eval is one score that combines several evals. Inside a composite, each eval can have its own settings, like a minimum word count or a regex pattern. In Tasks, you can also type custom paths to map deeper fields, not just pick from the dropdown. And if some mapped variables are missing, the eval runs with a warning instead of failing. It still errors only if all of them are missing. Built-in system evals stay strict.

Evaluate New

Skip variable mapping with context injection

You can now skip variable mapping. Turn on context injection and the eval reads the context on its own. Just pick what you want the eval to see: a dataset row, a span, a trace, a session, or a voice call. This works for every eval type (LLM-as-Judge, Code, and Agent). Inside a Task, the right context is picked for you based on the row type. For large traces or sessions, the eval looks at just the spans it needs, not the whole thing.

Trace + Session added to evals
5 context types in eval setup
2 new built-in conversation evals
~70 improvements across evals, observe, and simulate
Evaluate Improved

Two new conversation evals: Dead Air Detection and Conversation Hallucination

Two new built-in evals for voice and conversation projects. Dead Air Detection flags conversations with too much silence. It's a Code eval, so it runs at zero LLM cost (signal analysis on the audio, no model call). You can tune the share of silent audio, the longest single silent gap, and a silence threshold. Conversation Hallucination scores the whole conversation, not one turn at a time. It catches invented facts and unsupported claims that build up across the full exchange, which a single-turn hallucination eval would miss. Add either eval to any voice or conversation project.
Evaluate Improved

Eval inputs accept up to 200K characters

Long inputs no longer get clipped. LLM-as-Judge and Agent evals now accept up to 200K characters of input (was 15K). Fits long agent transcripts, large retrieved contexts, and multi-document grading.
Evaluate Improved

Consistent scoring across built-in numeric evals

Built-in scoring evals (accuracy, balanced accuracy, Cohen's kappa, F-beta, Matthews correlation, precision, and more) now normalise inputs the same way every time: case is ignored, whitespace is handled consistently, and comparing two empty values returns a perfect-match score. Several system eval rule prompts are also refined.
Evaluate API Improved

Eval mapping reaches span-level fields and API columns

You can now map eval variables to span-level fields like latency, cost, and tokens. Use paths like `spans.0.latency_ms`. The API column also accepts the same dotted-path syntax in URL, body, params, and headers. You can send a JSON column as the request body directly (no wrapping needed). And numeric array positions like `items.0.role` work everywhere, including LLM-as-Judge evals.
Evaluate Improved

Code eval parameters: numbers in, required fields enforced

Numeric parameters on code evals (latency limits, percentage thresholds, scoring weights) accept the text values that form fields produce (`200`, `0.75`) and validate them against the min and max bounds. Required parameters keep the Save button disabled until they're filled. Any parameters you add to your custom `evaluate` function appear in the Variable Mapping panel as soon as you type them.
Evaluate Improved

Deleting an eval task removes all related results

Deleting an eval task now removes its results from eval columns, charts, monitors, and filters in one step. No leftover rows or stale entries.
Evaluate Improved

Eval versioning: V1 on publish, full-field snapshots, draft separation

V1 is created the first time you publish an eval, not when you save a draft. Restoring an earlier version brings back every field on the eval (output type, pass threshold, choice scores, error localisation, and tags). When you run an eval, you can change a few settings just for that one run (for example, whether the eval has internet access, or what context to inject) without saving the change to the eval. And draft evals stay in the editor; they join the eval picker only after you publish them.
Evaluate Improved

Eval picker: configuration persists, source-aware naming, local preview filter

Every option you set in the eval picker (model, criteria, run settings, mapping) is saved and restored when you reopen the eval. Auto-generated names include the source (dataset, experiment, workbench, etc.) and a precise timestamp. The right pane has a local preview filter that shows the level you're scoring at (span, trace, session, or voice call). The Save & Add Evaluation button becomes clickable for code evals as soon as mapping is complete.
Evaluate Improved

Built-in evals work out of the box on open-source installs

Open-source installs ship every built-in eval as LLM-as-Judge, ready to run on first boot (no extra setup). On system evals, you can change the model used to judge, but the instructions stay read-only.
Evaluate Fixed

Cleaner eval cells across the platform

When an eval errors, the cell now shows a red 'Error' chip instead of looking blank. Voice and call-log grids hide Reason columns until an eval has filled them in. And eval result values display consistently across formats (number, category, choice list, or score-with-reason).
Evaluate Fixed

Eval tasks: detail-page edits reflect in the list automatically

Updating, pausing, resuming, or renaming an eval task from the detail page refreshes the task list automatically, so changes show up without a hard refresh.
Evaluate Fixed

Evals UI: faster reruns, accurate version banner, cleaner sliders and animations

Editing an eval inside a test execution triggers a rerun. The `Viewing version X` banner reflects the current version state in real time. The Save Version button activates when you've made a change. Required-field asterisks use a consistent style across every field. Pass-threshold sliders display clean whole-number percentages. The eval picker search shows a spinner while a search runs. The in-progress step in optimisation timelines animates with progress. The Evals login carousel uses up-to-date screenshots.
Monitor Improved

Trace ID and Span ID filters on the Tasks page; consistent operators across Observe

Trace ID and Span ID filters work on the Tasks page the same way they do on the Tracing page. Filter operators on span attributes are consistent across every Observe page. Saved views with ID filters reopen with the right operator selected.
Monitor Improved

More discoverable column resize, persistent custom columns across Observe

Observe tables show a clear column resize handle at rest, brighter on hover, with a wider grab area. Custom columns added on any Observe tab (Traces, Spans, Sessions, Users, Voice Calls) persist across refresh, tab switch, and switching saved views.
Monitor Fixed

Trace list: complete user data, filter chips, tag input, and overlay timing

The User column on the trace list shows the trace's end user. Span-attribute filter chips on the Tasks page display the complete operator and value. The enter-arrow icon on the tag input adds the tag on click. Categorical eval scores display consistently across new and historical runs. The `No traces found` overlay is suppressed while data is loading or filters are changing. The property picker handles duplicate property names across categories cleanly.
Simulate Improved

Voice projects: usage visibility, trace-level evals, surfaced errors

Voice provider traffic shows up in usage summaries and billing reports. Trace-level evals (evals that score a trace rather than one step) show up on the trace detail page. Errored evals on voice traces show a red Error chip, kept out of score averages. Customer call logs from the voice provider appear inside the simulation call detail page. Eval tasks run end-to-end on voice calls.
Simulate Improved

Simulate: safer runs and better failure handling

Starting a simulation against a scenario that hadn't finished generating used to give you zero connected calls and a permanent `Pending`. Run New Simulation now waits until every selected scenario is fully ready before it activates. The Edit Agent form used to leave you guessing whether the agent picked up provider changes; it now auto-syncs with the provider the same way Create Agent does, with a clear syncing indicator and an error notification on failure. And when every call in a run errored out, the eval bar used to disappear entirely; it now shows zero so the run still reports a meaningful result.
API Improved

AI gateway: Bedrock structured output and streaming failover

If you're calling Bedrock through the gateway, you can now use the same structured-output format you use for OpenAI and Gemini. The gateway translates it for Bedrock automatically, so you don't need provider-specific request shaping. And streaming chat with provider failover handles a common silent failure: when the upstream stream returns nothing, the gateway switches to the fallback model before the client sees a broken response.
Read full digest
W20 Apr 28 – May 11, 2026

Self-Host in One Command, Jinja2 Prompts, and Polish Across Evals and Observability

Self-host Future AGI in one command with pre-built images and a Windows installer. Prompts add Jinja2 alongside Mustache. Plus Request Explorer and polish.

Platform New

Self-Host Future AGI in One Command

Self-hosting Future AGI is one script: clone the repo, run `bin/install`, log in. The script pulls pre-built service images from Docker Hub, brings the stack up in about 30 seconds, and works on macOS, Linux, and Windows (through Git Bash, WSL, or a new native PowerShell installer). The first account is created from the command line, with no email server required.

Platform New

Prompts now support Jinja2 templates alongside Mustache

Use `{% if %}` for conditionals, `{% for %}` for loops, and Jinja2 filters to transform values inline. The existing Mustache `{{ variable }}` syntax keeps working, and a Template Format dropdown picks between the two per prompt.

~30s self-host pull-to-up
5 pre-built service images
Mustache + Jinja2 prompt templating
14 improvements across evals, observe, and voice
Monitor Improved

Reliable voice evals, no dependency on provider URLs

Future AGI now stores its own copy of every recording the moment the call arrives. Evals on voice traces run against audio you control, with no dependency on the original provider URL, so traces, replays, and eval reruns stay valid as long as you keep the trace. Storing your own copy is on by default for every voice provider and can be switched off per provider.
Evaluate Improved

Export annotations and eval scores alongside traces to datasets

Pick which annotation scores and eval results to export alongside traces when sending them to a dataset. The selected metrics arrive as dedicated columns in the dataset, so it's immediately ready for experiments with baseline scores, training data, or downstream exports — no re-running the evals.
API Improved

Filter and export AI gateway request logs

A new Request Explorer in the AI gateway logs filters every request by model, status, or metadata and exports the result. Useful for cost audits, error investigations, and compliance pulls.
Evaluate Improved

Reach into nested JSON fields from eval and prompt variables

Eval and run-prompt variables accept dotted paths into nested JSON. A single column like `payload` can feed multiple variables (`payload.user`, `payload.request.id`) without flattening the data upstream first.
Monitor Improved

Filter Observe by trace ID or span ID, plus Select all view columns

Paste a trace ID or span ID into the Observe filter bar to jump straight to that trace or span. The view-columns picker has a new Select all toggle.
Monitor Improved

Faster session list on high-volume traces

The session list now loads in seconds on workspaces ingesting millions of traces. No more waiting at a spinner to pick up a session for review.
Evaluate Improved

API columns: smoother edits, visible progress

Edits to an API column's URL, method, headers, params, or body now sync across devices and browsers. Requests run in batches, so rows fill in as each batch completes and the column's progress is visible while it runs. Param and header keys with underscores (like `api_key`) display exactly as you typed them.
Evaluate Improved

Cleaner eval columns

Inside eval column groups, Result now appears before Reason. Each eval column displays the template's current default version instead of defaulting to V1. Eval scores of exactly 0 render distinctly, and column averages refresh after you rearrange columns.
Evaluate Improved

Add traces to annotation queues from any source

Traces sent via our SDK or OpenTelemetry (OTLP) can be queued for annotation from the trace list. Dataset rows added to queues no longer get stuck loading, and CSV exports include annotator notes.
Monitor Improved

Trace attributes: long values expand, rows are easier to scan

Long string values in the trace attributes panel are click-to-expand instead of clipped. Dividers between rows make it clearer where one attribute ends and the next begins.
Monitor Fixed

Error Feed comes to voice and simulation projects

Error clusters from evals are now available on voice and simulation projects, visible across the summary, trend charts, and the trace panel. Clicking a voice trace opens the voice call panel.
Monitor Improved

Voice analytics: consistent units and a tighter default view

Latency, silence, and Time to First Word display in milliseconds across the call analytics panel and the call-logs table for at-a-glance comparison. Per-role talk-time percentages and Talk Ratio sit in the call detail panel for every transcript format, with Talk Ratio hidden by default to keep the view tight.
Platform Fixed

Workspace invites work for existing org members

Inviting an existing org member to a new workspace now sends the email and adds the workspace to their list straight away.
Platform Fixed

j and k row-navigation shortcuts no longer hijack text input

The j and k keys (used to move between rows in eval and task detail panels) now yield to focused text inputs, so the letters land in your comment instead of moving the row selection.
Read full digest
W18 Apr 14 – Apr 27, 2026

Evals Revamp, Experiment V2, Observe Revamp, and Error Feed

130+ ready-made evals that pull live web data and call your tools for scoring. Agent-level experiments. Plain-English Observe filters. Clustered errors.

Evaluate New

Evals Revamp: Score Against the Tools and Data Your Agent Actually Uses

Evals now reach outside the prompt: pull live web data, query your databases, and call the same tools your agent runs against in production. 130+ ready-made evals, composite scoring in one pass, full version history, and a unified picker that opens everywhere.

Evaluate New

Experiment V2: From Prompts to Agents

Experiments now run end-to-end on agents, not just prompts. Compare prompts, agents, or stacked model variants in one run, view per-node output for each agent, score against any dataset column, and edit, stop, or rerun mid-flight.

Monitor New

Observe Revamp: Plain-English Filters, Custom Views, and Imagine-with-Falcon Charts

Observe is rebuilt around how you actually debug. Filter traces in plain English, save and switch between custom views in a click, search inline across logs and previews, turn any trace into an interactive chart with Falcon, and run evals or push to datasets without leaving the trace list.

Monitor New

Error Feed: Cluster, Triage, and Push to Linear in One Click

The Error Feed clusters failing traces by root cause, updates about 5 seconds after a trace completes, and runs at roughly 10x lower cost than its previous version. Push a cluster to Linear with one click, run on-demand Deep Analysis when you need root cause, and read failing-vs-working trace evidence side by side.

130+ ready-made evals
~5s Error Feed update lag
10x lower-cost Error Feed
1-click push to Linear
Platform New

Custom pricing plans with billing dashboard v2

Sales can configure enterprise deals with custom pricing terms, and finance can generate invoices from the admin surface. The billing dashboard is rebuilt around the new model so cost breakdowns, usage trends, and per-key spend match the actual contract.
Platform Improved

Standardised search bar across list pages

Every list page in the platform now uses the same search bar, so the keyboard shortcut, the search syntax, and the result behaviour stay the same whether you're searching traces, datasets, evals, or experiments. No re-learning per surface.
Evaluate Improved

Annotator picker: search and add at scale

Search and add annotators directly from the picker, even on workspaces with a large user list. Infinite scroll loads results as you go, and server-side search returns matches instantly.
Platform Improved

Custom dashboards polish

Round of polish on Custom Dashboards: time range syncs to the widget editor, widgets resize cleanly, description fields render instantly, empty state on metric removal, filter Add button visibility, and dashboard description position.
API Improved

Agent definition API revamp

Agent definition endpoints now share one consistent payload shape across create, read, update, and delete (instead of variations per operation), so SDK and integration code can drop the per-endpoint marshalling logic and use a single type.
API Improved

Simulate analytics API

New analytics API for simulation data: pull aggregate metrics, breakdowns, and trends for any simulation run.
Monitor Improved

All Attributes JSON viewer search filter

The All Attributes JSON viewer (in the trace detail drawer) gets a search box that filters the JSON tree as you type. Useful when a trace has hundreds of attributes and you only want to see the ones matching a specific key or value.
Read full digest
W16 Mar 31 – Apr 13, 2026

Voice AI Production-to-Simulation, Annotation Queue Assignment, and API Docs Improvements

Turn any live voice call into a simulation test case, manually assign annotation queue items, and navigate API docs with full context on a single page.

Simulate New

Voice AI: Production to Simulation in One Click

Take any production voice call and turn it directly into a simulation test case. Rerun it against different prompt versions, prompt chains, or agent definitions and compare behaviour on real inputs.

API New

API Reference: Everything You Need on One Page

Each endpoint now renders cURL example, expected response, and full parameter details side by side. No tab switching, no scrolling between sections when you are writing integration code.

Evaluate New

Annotation Queue: Faster Review, Stronger Agreement

Multi-assignment, prefetching, and manual annotator assignment for annotation queues. Reviewer approval is now optional and can be toggled per workflow.

1-click production call to simulation
days → seconds regression to reproducible test
Simulate Improved

Voice Metrics in Call Lists

New voice metrics are available as columns in the call list view, so you can filter and triage calls by latency, interruptions, or turn duration without opening each one.
Read full digest
W14 Mar 17 – Mar 30, 2026

Falcon AI and 4x Faster Frontend

Falcon AI: context-aware assistant that debugs traces, scaffolds simulations, drafts evals. Plus 4x faster frontend, ClickHouse Replicated, LiveKit voice.

Platform New

Falcon AI

Context-aware AI assistant embedded in the platform for trace debugging, simulation creation, evaluation building, and dataset construction.

4x faster page loads
page-aware Falcon AI
high-availability ClickHouse trace storage
Platform New

Frontend speed improvements

Four targeted optimizations delivering 4x faster page loads across the platform.
Simulate Improved

Voice simulation moves to LiveKit

Voice simulation now runs on LiveKit instead of Vapi, with lower latency and more natural conversational dynamics during simulated calls.
Monitor Improved

ClickHouse Replicated MergeTree migration

Trace storage upgraded to Replicated MergeTree for high availability and automatic failover.
Platform Improved

Prompt generation and improvement

AI-assisted prompt generation from task descriptions and one-click prompt improvement suggestions.
Read full digest
W12 Mar 3 – Mar 16, 2026

Custom Dashboards, MCP Server, 2FA with Passkeys, and Annotation Queues

Drag-and-drop dashboard builder, an MCP server that puts Future AGI in your IDE, 2FA with passkeys, full Annotation Queue workflows, and a rebuilt ACC.

Platform New

Custom Dashboards

Drag-and-drop dashboard builder for tracking agent performance across evaluation scores, system metrics, cost, and experiment progress.

SDK New

MCP Server

Connect Future AGI to Cursor, Claude Code, VS Code, Claude Desktop, and Windsurf. Your AI coding assistant can run evaluations, query datasets, and pull traces without leaving the editor.

5 IDE integrations (MCP)
2,373+ LLM models priced out of the box
8 external platform integrations
Platform New

Two-factor authentication with passkeys

Full 2FA rollout: TOTP (authenticator app codes), WebAuthn passkeys (biometric or hardware-key sign-in), and recovery codes for account recovery.
Platform New

Role-based access control

RBAC at both organization and workspace levels. Four roles: Owner (billing), Admin (workspaces and integrations), Member (create and modify), and Viewer (read-only).
Evaluate New

Annotation Queues

Full annotation queue workflow with per-user progress tracking, inline scoring, and unified scores across traces, sessions, datasets, and simulation outputs.
Platform New

Agent Command Center: rebuilt architecture

The Agent Command Center gateway ships with a rebuilt architecture: consistent state across replicas, comprehensive security hardening, and tighter correctness guarantees.
Platform New

Integrations hub

Connect Langfuse, Datadog, PostHog, PagerDuty, Mixpanel, S3, Azure Blob, and Google Cloud Storage from a single configuration page in under a minute per integration.
Platform Improved

Pricing for 2,373+ models

Supported-model pricing grows from roughly 200 to over 2,373, so every major provider model now has accurate cost attribution out of the box.
Platform Improved

Falcon AI: skills UI and settings redesign

Falcon AI gets a redesigned skills and settings surface, plus a working stop-generation control for cancelling long responses cleanly.
Platform Improved

Chat history popover and feedback affordances

Chat history popover with thumbs up/down feedback and tool-call persistence across sessions.
Evaluate Improved

Task description field for optimizers

Every optimizer type now accepts a task description field (optimization objective) as input.
Platform Improved

Multi-breakdown analytics

Analytics now support breakdown by multiple dimensions at once. Slice the same widget by model, agent version, and time period together.
Monitor Improved

Central evaluation metrics rollups

Evaluation metrics now feed into one analytics rollup pipeline shared across surfaces, so a metric's value matches everywhere it appears (dashboards, the eval explorer, API). Aggregations on large eval datasets stay fast because the rollup is precomputed.
SDK Improved

Unified tracing across every traceAI language

traceAI now emits a single standardized span shape and attribute set across all four supported languages (Python, TypeScript, Java, C#), so traces from any part of your stack share one consistent shape end to end. A bundled migration utility upgrades historical traces in place.
Platform New

Enterprise multi-org architecture

Production-ready multi-org deployments with per-organization data boundaries, workspace-aware enforcement at every platform endpoint, and end-to-end test coverage across the multi-org lifecycle.
Simulate Improved

Outbound calls support

Backend coverage for outbound voice calls in simulation extended: more provider-side call states tracked, more failure modes surfaced cleanly. Builds on the outbound calling flow already in Simulate.
SDK Improved

futureagi-mcp-server tools

futureagi-mcp-server exposes the platform's tool surface to your IDE assistant: evaluations, datasets, prompts, experiments, simulations, tracing, agents, annotations, optimization, usage, and more. Your AI editor calls Future AGI directly without context-switching to the dashboard.
Read full digest
W10 Feb 17 – Mar 2, 2026

Agent Command Center, Agent Playground, and ClickHouse Migration

Agent Command Center: routing, guardrails, fallbacks, per-key cost controls. Agent Playground: visual multi-step graph builder. Plus ClickHouse migration.

Platform Guard New

Agent Command Center

Multi-provider routing, API key management, inline guardrails, automatic fallbacks, per-key cost tracking, and real-time analytics for every LLM call.

Agents New

Agent Playground

Visual graph builder for multi-step agents: two node types (LLM prompt + Agent), typed node connections, global variables, draft/publish workflow, version management, workflow execution control, and a programmatic graph API.

15+ LLM providers
6 load balancing strategies
2 data regions
Monitor New

ClickHouse migration for trace storage

Trace queries that used to take seconds now return in milliseconds, even on workspaces with millions of spans. Aggregations and filters across long time ranges feel interactive instead of batch.
Evaluate New

Annotation queue

Create annotation queues for traces, sessions, datasets, and simulation outputs to organize human review workflows.
Platform New

Multi-region support

Deploy and store data in US or EU regions for compliance with data residency requirements.
Evaluate Improved

Dataset and simulation analytics

One API surface returns both dataset quality metrics and simulation result trends, so external dashboards or BI tools can pull both without stitching together separate endpoints.
Guard Improved

Real-time updates for key revocation

Instant API key revocation across all replicas via real-time updates, closing the window between revocation and enforcement.
API Improved

Per-key authentication for Agent Command Center

Per-key auth and attribution for Agent Command Center: each team or workload runs through its own API key, with request validation, usage attribution, and downstream tracking all flowing from a single per-key identity. Drop-in for any existing OpenAI-compatible client.
Read full digest
W8 Feb 3 – Feb 16, 2026

ai-evaluation 1.0, Deep Space Theme, Multi-Language SDKs, and Multimodal Workbench

ai-evaluation SDK 1.0: unified evaluate API, multimodal judge, 72+ metrics. Deep Space dark mode. C#/Java SDKs + 31 TS instrumentors. Multimodal Workbench.

Evaluate New

ai-evaluation v1.0.0

ai-evaluation v1.0.0

The evaluation SDK's stable release: unified evaluate API, multimodal LLM judge, auto-generated grading criteria, vector-store feedback loops, 72+ local metrics, OpenTelemetry integration, streaming evaluation.

72+ ai-evaluation metrics
4 SDK languages (Py, TS, C#, Java)
31 new TypeScript instrumentors
Platform New

Deep Space dark mode migration

Comprehensive monochrome theme across the entire platform, with every component, every surface, and every interaction state rethought for visual consistency and reduced eye strain.
SDK New

traceAI C# SDK

Full traceAI instrumentation for C# applications and .NET environments. ASP.NET services, Azure Functions, and standalone apps get automatic instrumentation for LLM calls, tool invocations, and agent workflows.
SDK New

traceAI Java SDK

Java SDK with 25 instrumentation modules covering Spring Boot, Micronaut, Quarkus, Apache HttpClient, OkHttp, JDBC drivers, and major Java-based LLM client libraries.
SDK New

31 new TypeScript instrumentor packages

Large expansion of TypeScript instrumentation covering frameworks, databases, HTTP clients, and more, the long tail that TypeScript agent developers rely on.
Platform New

Multimodal Prompt Workbench with WebSocket streaming

The Prompt Workbench goes multimodal end-to-end, with WebSocket-streamed responses that render in real time.
Simulate New

Graph-aware dataset scenario generation

Dataset-based scenarios now generate with branch-context. The LLM call that generates each scenario understands which branch of the graph it's in, producing scenarios that stay consistent with surrounding branches.
Simulate Improved

WebSocket simulation grid

Simulation grid updates stream to the UI via WebSocket instead of polling.
Monitor Improved

Langfuse integration

Configure Langfuse directly from the platform UI, with Langfuse-compatible endpoints for Vapi-routed traffic.
Monitor Improved

Voice observability annotations

Annotation workflow now extends into voice observability. Add annotations directly on voice traces with dedicated filters.
Simulate Improved

Provider-agnostic voice simulation runs

Voice simulation runs are now provider-agnostic, with LiveKit signal monitoring giving precise call-state tracking across providers.
SDK Improved

traceAI Python SDK update

New framework support and end-to-end tests for the Python traceAI SDK.
Platform Improved

Per-tab workspace context

Each browser tab keeps its own workspace context: production traces in one tab, staging simulations in another, no cross-tab interference.
Evaluate Improved

Function parameters in evaluations

Pass function parameters directly to evaluation metrics for dynamic scoring configurations.
Evaluate Improved

Reasoning parameters support in prompts

Configure reasoning-specific parameters (reasoning effort, max thinking tokens, whether to surface the reasoning trace) when working with chain-of-thought models in the Prompt Workbench. The same prompt can be tuned for fast and cheap or slow and thorough without rewriting it.
Monitor Improved

has_eval filter for traces and spans

Filter traces and spans in Observe by whether they have evaluations attached.
Monitor Improved

Workspace-scoped error analysis (Feed display)

Workspace-aware error analysis: the Error Feed surfaces failures from the active workspace, keeping triage focused on the project you're debugging. Cleaner empty states for users still being onboarded to projects.
Monitor Improved

Optimized trace session queries

Filtering and sorting on trace sessions now happen in the database layer instead of in app code, so session views load quickly even on workspaces with millions of spans. Most visible when filtering long lists by date or status.
Read full digest
W6 Jan 20 – Feb 2, 2026

Simulate from Prompt Workbench, Voice Annotations, and Agent Health for Voice Agents

Launch simulations without leaving the Prompt Workbench, annotate voice calls with structured human feedback, and extend Agent Compass health to voice.

Simulate New

Simulate from Prompt Workbench

Simulate from Prompt Workbench

Add and configure simulations directly from the Prompt Workbench. Full simulation engine, accessible from where you're already working.

5 label types for voice annotations
2x faster file processing
Evaluate New

Human annotations for voice calls

Structured feedback for voice transcripts, with five label types and support for multiple reviewers on the same call.
Monitor New

Agent health monitoring for voice agents

Agent Compass now extends to voice agents, with call duration distributions, response latency percentiles, interruption rates, and conversation completion metrics in real time.
Simulate Improved

Voice simulation revamp

Voice simulation runs against a rebuilt runtime that cuts end-to-end run time on multi-scenario suites. The API surface was consolidated to fewer endpoints with consistent payload shapes, so a test harness wiring against it doesn't have to handle one-off response formats per call type.
Evaluate Improved

Multi-image support in evaluations and datasets

Evaluations accept and score multi-image inputs; datasets accept multiple images per row.
Platform Improved

Reasoning model support

First-class support for reasoning models. Chain-of-thought steps appear as distinct spans (the individual steps inside a trace) in the trace view.
Simulate Improved

WebSocket simulation grid updates

Simulation results stream to the grid in real time, with no manual refreshes.
Platform Improved

Image and audio output rendering in Prompt Workbench

The Prompt Workbench renders image and audio outputs inline for multimodal prompt iteration.
Platform Improved

Azure endpoint type selector

Select Azure-specific endpoint types when configuring custom models, with proper API format handling for Azure-hosted deployments.
Platform Improved

Read traffic and write traffic separated

Dashboard loads and search no longer slow down during heavy evaluation runs. Query traffic and write operations run on independent paths.
Platform Improved

2x faster dataset imports

CSV, Excel, and JSON imports now run roughly 2x faster with significantly lower memory consumption.
Simulate Improved

Faster simulation results and evaluations dashboard

Simulation results and the evaluations dashboard load noticeably faster: queries pushed down to the database layer, page-by-page rendering replacing full result-set loads. Most visible when scrolling through hundreds of test runs.
Evaluate Improved

Function evaluations in test evaluations

Function-type evaluations (deterministic Python or JavaScript checks you author yourself, not LLM-judged) now run inside test evaluation workflows. Useful when pass/fail is logic, not opinion: schema validation, exact-match comparisons, custom string parsing.
API Improved

Simulate API changes for run tables and optimization

Simulate run tables and optimisation endpoints have cleaner request shapes and more consistent error responses, so existing API consumers can drop one-off conditionals around handling individual endpoints.
Read full digest
W4 Jan 6 – Jan 19, 2026

Baseline Chat Comparison, Fix My Agent Polish, and OpenTelemetry Instrumentation

Baseline chat comparison wires production conversations into simulation as the fastest path to reproducible tests. Plus Fix My Agent, OTel, image outputs.

1-click production trace to simulation baseline
4 Agent framework wrappers
Simulate New

Baseline chat comparison from Observe to Simulation

Compare real production conversations against simulated outputs to identify drift and regressions. Fastest path from "something went wrong in production" to "here's a reproducible test case."
Agents Improved

Fix My Agent: final polish

The drawer is restructured so you reach the suggested fix faster, call-selection bugs that affected long simulation runs are fixed, and a restore-with-conflicts flow keeps your local edits intact when Fix My Agent's suggestion lands on top of changes you've already made.
Platform Improved

OpenTelemetry instrumentation in the platform

Future AGI now emits OpenTelemetry (OTEL) traces for its own operations, with Sentry integration across long-running workflows.
Agents Improved

Agent Prompt Optimiser: resume support

Long-running optimiser jobs now survive restarts and pick up from where they left off.
Evaluate Improved

Dataset optimisation with direct evaluation

Run evaluations directly from datasets and manage trial items without leaving the dataset view.
Evaluate Improved

Image output support in datasets and Prompt Workbench

Datasets and the Prompt Workbench render image outputs natively, useful for document analysis, chart generation, and visual QA workflows.
Evaluate Improved

Multiple image upload in datasets

Upload many images at once when building datasets. No more one-by-one file selection.
Simulate Improved

Bulk delete and bulk rerun test executions

Select dozens of test executions and delete or rerun them in a single action. What used to require API scripting is now two clicks.
Agents Improved

Output type selection in Playground results

Set the expected response format on a Playground run so multimodal output renders correctly instead of defaulting to a raw string preview.
Simulate Improved

Chat inputs for simulation analysis agent

The simulation analysis agent now accepts chat-formatted inputs, so chat-based simulation runs can be triaged by the same diagnostic agent that already handles voice runs. Useful when reviewing mixed simulation suites.
SDK Improved

simulate-sdk v0.1.2

Cloud mode for offloading simulation execution, agent wrappers for OpenAI, LangChain, Gemini, and Anthropic, and tool-call capture built into every simulation run.
Read full digest
W2 Dec 23 – Jan 5, 2026

Chat Simulation via Observe, Pre-Built Evaluation Groups, and Fix My Agent for Chat

Launch chat simulations directly from real production conversations, 10 ready-to-use evaluation groups with no configuration, and Fix My Agent for chat.

Simulate New

Chat Simulation via Observe

Chat Simulation via Observe

Browse real customer conversations in Observe (the view of your live production traces), find the one you want to test against, launch a chat simulation from it with one click.

10 Pre-built evaluation groups
1-click Simulate from Observe
Evaluate New

Pre-built evaluation groups for simulations

10 ready-to-use evaluation groups covering common quality dimensions: attach one to a simulation and run, no configuration required.
Agents New

Fix My Agent support for chat agents

The Fix My Agent diagnostic engine now works on chat-based agents with text-specific failure-mode detection.
Simulate New

Chat Simulation V1 launch polish

The remaining launch pieces for Chat Simulation V1 land together: the persona section, chat logs with inline traces and attributes, the evaluation mapping flow, and the status/UI polish that ties them into one product. Everything needed to run Chat Sim end to end is now in place.
Agents Improved

Agent Prompt Optimiser launch on platform

The Agent Prompt Optimiser is now accessible directly from the platform UI. Pick a strategy, choose target calls, run optimisation without writing API code.
Evaluate Improved

Domain-level metrics, insights, and human comparison summary

Simulation runs now roll up metrics by domain (support, sales, onboarding, whatever you've tagged) and surface a side-by-side summary that compares the agent's behaviour against human-handled conversations. Useful for spotting where automation is on par and where it lags.
Evaluate Improved

Agent prompt conformance evaluation

New evaluation metric that measures how closely an agent follows its prompt instructions.
Platform Improved

Additional LLM models

Latest model releases from the major providers (OpenAI, Anthropic, Google, and others) are wired into evaluation and prompt surfaces, so you can run scoring or generation against new models without waiting for a separate Future AGI release.
Platform Improved

Unified chat message roles

Message roles (user, assistant, system, tool) and the way the dashboard labels them now match across every chat surface. A conversation captured in Observe reads identically inside Simulate or the Workbench, so you can copy a transcript across tools without manual remapping.
Simulate Improved

Call analytics drawer with transcripts and manual graph creation

Call analytics drawer gains transcripts view and a manual graph-creation flow.
Platform Improved

Dynamic model parameter updates from API

Model parameters automatically refresh based on provider API capabilities, so new model variants and parameter changes from providers flow through without manual updates.
Platform Improved

Audio content validation for audio models

Automatic validation of audio content format and quality before submission to audio models. Catches bad inputs before they consume compute.
API Improved

Manage replay sessions through the API

Programmatic management of replay sessions, useful for wiring replay regression checks into CI/CD pipelines.
Simulate Improved

Streamlined persona management in scenarios

Assign and swap personas inside scenarios with a simplified inline interface; edit persona info directly from scenario columns.
Simulate Improved

Complete simulation status visibility

Stage-level progress indicators on every simulation run (scenario generation, conversation execution, evaluation scoring), visible in real time.
Platform Improved

API key management: delete from dashboard

Delete API keys directly from the dashboard to revoke access.
Evaluate Improved

Edit Run Experiment

Experiment configurations can now be edited mid-run (swap a model variant, adjust a scoring threshold) without restarting. Configuration history is tracked per data point so the audit trail stays clean.
Read full digest
W52 Dec 9 – Dec 22, 2025

Chat Simulation V1, Agent Prompt Optimiser, and Reliability Upgrades

Simulation for text chat agents, a six-strategy automated prompt optimiser, selective optimisation against specific calls, and resilience on restarts.

Simulate New

Chat Simulation V1

Chat Simulation V1

Full simulation engine for chat-based agents with persona-driven conversations, scenario generation, and in-drawer analytics.

6 Optimisation strategies (agent-opt)
200+ Conversation turns per simulation
Agents New

Agent Prompt Optimiser

Automated prompt optimisation runs against your evaluation data: six strategies wired into the platform with a results UI.
Agents New

Optimize My Agent V3: targeted optimisation

Select specific calls to optimise against, instead of optimising across the whole dataset. Direct the optimiser at the failure modes you actually care about.
Simulate New

Create scenario from Observe

Convert any real production session in Observe (the view of your live production traces) into a reusable simulation scenario with one click.
Simulate Improved

Replay sessions from real traces

Re-run historical production conversations through your current agent configuration: regression tests built from production traffic.
Platform Improved

Dot notation for JSON column variables

Reference nested JSON fields with dot notation (e.g., `user.profile.language`) in prompt templates and experiment configurations.
Platform Improved

Document format preview in Dataset and Experiment

Inline preview for documents referenced in datasets and experiment results. No download needed.
Simulate Improved

Instruction input in scenario creation

Describe what you want to test in natural language and the system generates scenarios accordingly.
Evaluate Improved

Evals filtering in dataset summary

Filter evaluation results inside the dataset summary view for faster drill-down.
Monitor Improved

Agent-centric metrics and call log improvements

Per-agent metrics now appear on the dashboard, so agent quality is comparable directly across agents instead of aggregated. Call log capture broadened to record more of what the agent does inside a call.
Evaluate Improved

Edit synthetic data configuration

Modify synthetic-data generation configuration after a run has started.
Read full digest
W50 Nov 25 – Dec 8, 2025

Fix My Agent, Persona Management Suite, and JSON Input/Output in Sessions

Context-aware debugging that explains why a simulation failed and how to fix it, full lifecycle for personas, structured JSON sessions, Optimiser backend.

Agents New

Fix My Agent

Fix My Agent

Context-aware debugging that reads the full execution context of a failed simulation and generates ranked, actionable suggestions across two classes: agent-level (prompt, context, tool usage) and infrastructure-level (provider timeouts, rate limits, integration misconfiguration).

ranked Fix My Agent suggestions
3x Faster issue resolution (reported)
Simulate New

Persona management suite

Full lifecycle management for simulation personas: view across workspace, duplicate as starting points, edit inline, and delete.
Platform New

JSON input/output in session view

Session views render structured JSON input and output natively: collapsible trees, syntax highlighting, Markdown rendering in table cells.
Agents Improved

Agent Prompt Optimiser groundwork

Backend infrastructure for the Agent Prompt Optimiser is now in place: evaluation hookups, the optimisation job runner, and the result schema. The user-facing optimiser builds on top of this foundation.
Evaluate Improved

Edit experiment configuration after starting

Modify experiment parameters mid-run without restarting. The system tracks which configuration was active for each data point so result integrity is preserved.
Platform Improved

JSON dot notation in Run Prompts and Experiments

Reference nested JSON fields using dot notation (e.g., `user.profile.preferences.language`) in prompt templates and experiment configs.
Simulate Improved

Custom voices for ElevenLabs and Cartesia

Custom voice support for ElevenLabs and Cartesia in Run Prompt and Experiments.
Platform Improved

Rate limits update for custom subscription tier

Rate limits raised on the CUSTOM subscription tier so high-volume customers no longer hit the previous ceiling on bursty workloads.
Platform Improved

Enhanced audio player with lazy loading

Rebuilt audio player with lazy loading. Faster page loads on views with many session recordings.
Agents Improved

Fetch agent definition from providers

Import agent configurations directly from Vapi and Retell with one click. No manual recreation.
Monitor Improved

Polling and loading state in error localization

Error-localization analysis runs asynchronously. The view now polls for the result and shows a loading state in the meantime, so the page no longer appears to hang while the analysis completes in the background.
Simulate Improved

Real-time loading states for calls

Live progress indicators and estimated time remaining for ongoing calls.
Platform Improved

Workspace issues view

Dedicated view that surfaces issues affecting the whole workspace (not just a single project), so admins can spot platform-level problems without drilling into each project individually.
Read full digest
W48 Nov 11 – Nov 24, 2025

Multi-Branch Scenarios, Custom Background Noises, and Critical-Issue Feed in Simulate

Scenarios that branch into multiple conversation paths, ambient noise profiles that push simulations closer to production, and a critical-issue feed.

Simulate New

Multi-branch scenario generation

Multi-branch scenario generation

AI-generated scenarios that branch into multiple paths: intent forks, escalation paths, and digression branches. Navigate the tree and run simulations against specific branches.

10+ Background noise profiles
3 Scenario branch types
Simulate New

Custom background noises in simulation

Add realistic ambient noise profiles to voice simulations (coffee shop chatter, traffic, typing, crowds, office HVAC) so tests match production conditions.
Agents New

Enable Others option for agent definition

Define agents against custom voice or LLM providers beyond the standard Vapi and Retell integrations. Open configuration for teams with heterogeneous stacks.
Simulate New

Critical-issue feed in Simulate

A dedicated feed surfaces the most important simulation failures across runs: the issues worth fixing first.
Evaluate Improved

Eval explanation summary for simulations

Every simulation evaluation now includes a human-readable explanation: not just a score, but why the score landed there.
Simulate Improved

Call analytics revamp

Redesigned call analytics and call detail views with clearer metrics, latency breakdowns, and per-call cost reporting.
Simulate Improved

Latency metrics in Simulate

Per-step latency tracking for simulation calls. See how long each phase took.
Simulate Improved

Persona polish: more accents and language sorting

Southern accent and additional accents added to personas. Languages and accents now sorted alphabetically in the persona selector.
Platform Improved

Prompt WebSocket streaming

Real-time prompt execution with streaming responses via WebSocket. No more wait-for-completion in the Workbench.
Evaluate Improved

Edit evaluation variable remapping

Remap evaluation variables after creation without rebuilding the entire evaluation configuration.
Monitor Improved

Observe enhancements

Sticky filters, pagination improvements, inline metadata display, and updated pricing logic in Observe (the view of your live production traces).
Monitor Improved

New columns in Observe

Additional columns in the Observe table surface agent and call metadata directly without drilling in.
Evaluate Improved

Scenario column support in evals and run test

Scenario columns are now available inside evaluation runs and run-test results, providing richer test data without leaving the evaluation view.
Simulate Fixed

Simulated assistant call ending fixes

Simulated assistant calls now end reliably when the agent finishes its final turn or hits a configured timeout, instead of leaving the session hanging.
Read full digest
W46 Oct 28 – Nov 10, 2025

Simulation Call Observability, Retell and Outbound Calls in Simulate, Tool Evaluation

Logs, latency, cost on every simulation call. Retell agents, outbound calling, tool-level verification in Simulate. Plus editable personas and Run Prompt.

Simulate New

Logs, latency, and cost breakdown on simulation calls

Logs, latency, and cost breakdown on simulation calls

Every simulation call now shows structured logs, per-call latency metrics, and cost attribution in one view.

3 simulation observability layers
2 voice providers in Simulate
Simulate New

Retell agents in Simulate

Native Retell support inside the simulation loop. Retell joins Vapi as a supported voice provider for end-to-end test runs.
Simulate New

Outbound calling in Simulate

Simulate outbound voice flows where your agent places the call: appointment reminders, sales outreach, proactive notifications, payment collection.
Evaluate New

Tool evaluation in Simulate

Verify every tool and function call your voice agent makes during a simulation. Did it call the right tool? Were the parameters right? Did it handle the response?
Simulate New

Run Prompt and Experiment revamp

Contextual provider selection and streamlined configuration for prompt runs and experiments. Audio output supported from both workflows.
Simulate Improved

Personas: full CRUD and edit-after-creation

Create personas, edit them after creation, and manage the full persona library. Pairs with the persona system.
Simulate Improved

Reasoning column in Simulate

View the reasoning trace behind each simulation decision directly in the results table.
Simulate Improved

Custom voices in Run Prompt and Experiments

Use custom voices from ElevenLabs and Cartesia in prompt runs and experiment workflows.
Monitor Improved

Expanded evaluation attributes in voice observability

New evaluation dimensions for voice quality, latency consistency, and naturalness in voice agent monitoring.
Simulate Improved

Error localization in Simulate

Errors in simulation runs are now pinpointed to the exact step and provider that caused them.
Evaluate Improved

Edit evaluations within experiment page

Modify evaluation configurations inline without leaving the experiment view.
API Improved

Configure and re-run evaluations via API

Programmatically configure evaluation parameters and trigger re-runs through the API.
Platform Improved

Session history enhancements

Session history now renders full transcripts instead of truncating long ones, and supports more locales. A session shows everything your users actually said.
Monitor Improved

Observe homepage revamp

Observe landing page now leads with recent traces and active alerts, so the most common starting points (jump to a trace, check an alert) take fewer clicks. Initial load is noticeably faster on high-volume workspaces.
Read full digest
W44 Oct 14 – Oct 27, 2025

Credit Usage Revamp, Multi-Language Agents, and New TTS Providers

Workspace credit attribution, a 3-step guided agent builder with multi-language, rebuilt Prompt Workbench with commit history, and 4 new TTS providers.

Platform New

Credit usage summary redesign

Credit usage summary redesign

Workspace-level credit attribution with per-feature breakdowns and historical trend lines. Forecast AI spend with real granularity.

4 New TTS providers
15+ Languages supported
Agents New

New agent definition UX

A 3-step guided flow for building agents: identity and behavior, then tools and integrations, then a preview sandbox before deploying.
Platform New

Prompt Workbench revamp

Commit-based version history comes to the Prompt Workbench. Think git, but for prompts.
Agents New

Multi-language support in agent definition

Agents can be defined to operate natively in 15+ languages with locale-aware behavior that goes beyond simple translation.
Simulate Improved

Add columns to scenarios via AI and manual input

Enrich simulation scenarios with custom data columns. Generate them with AI, or enter them manually.
Simulate Improved

Enhanced language and accent support in simulation

Broader dialect and accent coverage for realistic multi-language voice simulations.
Simulate Improved

Simulate metrics revamp

Redesigned metrics dashboard with real-time pass/fail rates and drill-down into individual test cases.
SDK New

ai-evaluation v0.2.2

First-class LLM-as-a-Judge and new heuristic metrics: JSON schema validation, similarity, string matching, aggregation.
Platform Improved

Call analytics integration

Unified analytics for voice agent calls with cost, duration, and quality breakdowns in one view.
Simulate Improved

Detailed voice provider logs

Full request and response logs for every voice provider interaction during simulation.
Simulate New

New TTS model integrations

Four new text-to-speech (TTS) providers (Cartesia, Hume, Neuphonics, and LMNT) now available in simulation.
Read full digest
W42 Sep 30 – Oct 13, 2025

ai-evaluation SDK v0.1.5, Personas, and Run-Prompt Enhancements

ai-evaluation SDK launches with 50+ templates. Pre-built and custom personas come to simulation, with dataset-derived personas from real call transcripts.

SDK New

ai-evaluation v0.1.5 launch

ai-evaluation v0.1.5 launch

Initial release of the ai-evaluation SDK with 50+ evaluation templates covering faithfulness, relevance, safety, and domain-specific metrics.

50+ Eval templates in ai-evaluation
3 Persona sources
Simulate New

Pre-built and custom personas

Pre-built caller personas, custom persona definitions, and dataset-derived personas generated from your real call transcripts.
Evaluate Improved

Provider transcript as evaluation attribute

The voice provider's native transcript is now available as an evaluation input, useful for comparing automatic speech recognition (ASR) accuracy and response quality side by side.
Platform Improved

Enhanced onboarding flow

First-run setup branches by user role, so each new user lands on the steps that match how they will use the platform instead of a single generic walkthrough. Time from sign-up to first evaluation drops accordingly.
Simulate Improved

Voice output in Run Prompt and Run Experiment

Generate and evaluate spoken responses directly from the prompt playground and experiment workflows.
Simulate Improved

Manual, AI-generated, and dataset-sourced scenario rows

Three ways to add scenario rows to a simulation: type them in, generate them with AI, or import from an existing dataset.
Evaluate Improved

Run evaluations on completed test runs

Apply new evaluation criteria to previously completed simulation runs, with no need to re-execute the calls.
Agents Improved

Agent definition version selection in simulation

Select a specific agent definition version when running a simulation, so regression testing and A/B comparisons become straightforward.
SDK Improved

ai-evaluation v0.2.1

Batch evaluation support and bias detection added to the ai-evaluation SDK.
SDK Improved

traceAI OpenAI Agents SDK support

Native instrumentation for OpenAI's Agents SDK that captures tool calls, agent handoffs, and multi-agent orchestration as traces.
Platform Fixed

Updated pricing calculation in Observe

Trace cost is calculated as spans land in Observe, not after a post-processing pass. The cost figures on the dashboard and on the underlying trace stay in sync, so usage reporting matches what the trace view shows.
Read full digest
W40 Sep 16 – Sep 29, 2025

Voice Observability for Vapi, Retell, and ElevenLabs; Eval Groups in Experiments; Simulate via SDK

Observability ships for three voice platforms at once. Evaluation groups integrate with experiments and optimization. Call Simulation now SDK-triggerable.

Monitor New

Voice Observability for Vapi, Retell, and ElevenLabs

Voice Observability for Vapi, Retell, and ElevenLabs

Observability ships for three voice platforms at once: Vapi, Retell, and ElevenLabs. Call metrics, transcript analysis, and utterance-level tracking on the same trace and evaluation surface you already use for text agents.

3 Voice platforms supported
60-70% Cost reduction via SDK simulation
Evaluate New

Eval groups in experiments and optimization

Evaluation groups now integrate with experiments and agent optimization. Run a whole group of evaluations together and optimize against group-level scores.
SDK New

Simulate via SDK

Trigger Call Simulation programmatically through the SDK against Vapi- or Retell-backed voice agents. Plug simulation into CI/CD pipelines, scheduled jobs, or custom orchestration.
Simulate Improved

Selective test rerun in Simulate

Rerun only the specific failed or flagged test cases in a simulation suite, without re-executing the entire suite.
Evaluate Improved

Default eval groups

Pre-built evaluation groups for common use cases: retrieval-augmented generation (RAG), computer vision, conversational AI.
Simulate Improved

Advanced simulation management

Auto-refresh on the simulation dashboard, stop-simulation control for running suites, and a visual workflow trace of the execution path for each scenario.
Platform Improved

Workbench revamp

Slide-out code drawer in the Workbench keeps prompt code visible while you iterate, so you don't tab-switch to inspect what you're running. The header is restructured to keep run controls reachable without scrolling.
SDK Improved

traceAI session support

Native `session.id` attribute support in traceAI, with automatic session grouping across all instrumented frameworks and no custom middleware.
Platform Fixed

Agent definition design changes

Agent configuration page redesigned for agents with many prompts and tools. Sections stay grouped and reachable as the configuration grows past a handful of nodes.
Read full digest
W38 Sep 2 – Sep 15, 2025

Automated Scenario Builder, Agent Definition Versioning, and Simplified Session Tracking

Upload an SOP or call transcript to auto-generate test scenarios with edge cases. Commit-style version control for agents. Session observability per span.

Simulate New

Automated scenario and workflow builder

Automated scenario and workflow builder

Upload standard operating procedures (SOPs), call transcripts, or product documentation. The system generates simulation scenarios with edge cases and branching conversation flows.

10x Faster scenario generation
3 Audio download formats
Agents New

Agent definition versioning

Commit-style version control for agent definitions. Every change gets a commit message, consolidated test reports show how each version performed, and rollback is one click.
Monitor Improved

Simplified session tracking

Add a single `session.id` attribute to spans and Future AGI groups related traces into a session view automatically.
Evaluate Improved

Advanced evaluation group management

Full create, read, update, delete (CRUD) operations for evaluation groups. Organize and manage evaluation suites programmatically.
Simulate Improved

Multi-channel audio player for simulation calls

Listen to simulation calls with separate channels for agent and caller audio — focus on one side or hear both together.
Simulate Improved

Flexible call recording downloads

Download simulation call recordings in three audio formats, for offline analysis, compliance archiving, and sharing.
Platform Improved

Prompt collaboration features

Collaborative editing and commenting on prompts. Multiple team members can work on the same prompt, leave feedback, and track revisions.
Platform Improved

Dedicated background worker pool for trace ingestion

Dedicated background worker pool for trace ingestion. Separates trace processing from other background jobs and eliminates resource contention during high-volume ingestion.
Platform Improved

Optimized trace ingestion pipeline

End-to-end optimization of the trace ingestion pipeline. Faster time from trace emission to dashboard availability.
Platform Fixed

Annotation and prompt import fixes in datasets

Resolved issues with importing annotations and prompt data into datasets for smoother data pipeline operations.
Read full digest
W36 Aug 19 – Sep 1, 2025

Agent Compass, Annotation Quality Dashboard, and Enterprise Multi-Workspace Security

Zero-config performance insights on agent traces, statistical dashboards for annotator agreement, and multi-workspace isolation with audit logging.

Agents New

Agent Compass

Agent Compass

Automatic, zero-configuration performance insights on your agent traces. Detects issues at the trace level with no evaluation setup required.

0 Config required for Agent Compass
5 Eval group templates
Evaluate New

Annotation quality dashboard

Inter-annotator agreement metrics (including Cohen's kappa) to measure how consistent your human reviewers are.
Platform New

Enterprise multi-workspace security

Multiple independent workspaces with full data isolation, per-workspace role-based access control (RBAC), and cross-workspace audit logging.
Monitor Improved

Feed insights with error clusters

Observability feed now groups related errors into clusters and surfaces trend lines showing whether issues are getting better or worse.
Platform Improved

Intelligent onboarding navigation

Guided onboarding flow that adapts to your role and use case.
Simulate Improved

Voice agent testing and analytics improvements

Dashboard metrics and scenario column views for voice simulation. Call success rate, average call duration, and scenario coverage in one place.
Platform Improved

Prompt library organization

Folder-based prompt architecture with templates for structuring large prompt libraries across teams and projects.
Evaluate Improved

Evaluation grouping API

API support for evaluation grouping — organize related evaluations programmatically, run evaluation groups as test suites in CI/CD.
Platform Fixed

Enhanced plans and pricing experience

Redesigned pricing page with clearer plan comparisons and streamlined upgrade flows.
Read full digest
W34 Aug 5 – Aug 18, 2025

Summary Dashboards, Alerts Revamp, Prompt SDK, and Workspaces RBAC

Redesigned summary dashboards with new chart types and side-by-side compare, a rebuilt alerts system, Prompt SDK upgrades, and RBAC workspace access.

Monitor New

Summary screen revamp

Summary screen revamp

Rebuilt summary dashboards. Spider, bar, and pie chart visualizations, plus side-by-side comparison between any two runs, prompt versions, or time periods.

4 Role levels
3 Chart types
Monitor New

Alerts revamp with Slack and email

Rebuilt alerts system with Slack and email notification channels, customizable thresholds, and alert grouping to reduce notification noise.
SDK New

Prompt SDK upgrades

Caching, A/B testing, and multi-environment deployment in the Prompt SDK — production-grade prompt management.
Platform New

Workspaces RBAC

Role-based access control with Owner, Admin, Member, and Viewer roles. Granular permissions on evaluations, simulations, and production traces.
Platform Improved

AWS Marketplace integration

Purchase and manage Future AGI subscriptions through AWS Marketplace with consolidated AWS billing.
SDK Improved

Error localizer via SDK

Synchronous and asynchronous standalone error localization through the SDK to pinpoint failures in agent execution chains.
Evaluate Improved

Critical issue detection on datasets

Automatic detection of critical issues in datasets with actionable advice for data quality problems.
Monitor Improved

Prompt metrics in Observe

Track trace performance per prompt version inside Observe — measure the real-world impact of a prompt change.
SDK Fixed

traceAI optional dependencies cleanup

Reduced install bloat by making framework-specific dependencies optional, cutting package size significantly.
Read full digest
W32 Jul 22 – Aug 4, 2025

Document Columns, Function Evaluations, and Async Evals via SDK

Upload documents into datasets with built-in OCR, write deterministic function evals for objective checks, and run evaluations async from the SDK.

Platform New

Document column support in datasets

Document column support in datasets

Upload TXT, DOC, DOCX, PDF, and scanned documents directly into datasets. Built-in OCR handles scans and images.

5 Document types supported
50+ Eval templates
Platform Improved

Edit synthetic data after generation

Modify AI-generated synthetic data before committing it to your datasets.
Monitor Improved

User tab in Dashboard and Observe

Per-user views in Dashboard and Observe surface aggregate metrics across sessions and traces.
Platform Improved

Configure labels per prompt version

Tag each prompt version with custom labels to track experiments, A/B tests, and rollout stages.
Monitor Improved

Video support in Observe

Capture and replay video outputs from multimodal agents directly in the Observe trace view.
Platform Improved

OCR support for document processing

Optical character recognition extracts text from scanned documents and images — legacy paper-based workflows become testable without manual transcription.
Evaluate Improved

Comparison summary

Compare evaluation results and prompt summaries across two datasets side by side to measure improvement or catch regressions.
SDK Improved

Bulk annotation and user feedback via API/SDK

Submit thousands of annotations and feedback entries in a single call. For high-throughput labeling workflows and integrations with existing annotation tools.
SDK Improved

traceAI v0.1.10 with LLM prompt template labels

traceAI automatically labels LLM spans with prompt template identifiers, so you can filter traces by prompt version.
SDK Improved

traceAI Pipecat integration

Native Pipecat instrumentation for tracing voice and multimodal AI pipelines built on Pipecat.
SDK Improved

traceAI LlamaIndex TypeScript

TypeScript instrumentation for LlamaIndex, bringing tracing to the RAG framework in Node.js environments.
Monitor Fixed

Timestamp column in trace/spans

Precise timestamp columns in trace and span views for accurate timing analysis and debugging.
Evaluate Fixed

JSON view for evals log

Inspect raw evaluation log data in structured JSON format for debugging and integration purposes.
Read full digest
W30 Jul 8 – Jul 21, 2025

Voice Simulation and the Evals Playground

AI-conducted phone calls test voice agents end-to-end. Plus an interactive sandbox for evaluations in real time, with inline scoring on any trace span.

Simulate New

Call Simulation

Call Simulation

AI-powered simulator agents place real phone calls to your voice agents. Scenario-driven conversations with real turn-taking, interruptions, and the unpredictable patterns real users bring.

sub-second Voice latency
60-70% Cost reduction vs manual QA
Evaluate New

Evals Playground

An interactive sandbox for testing and refining evaluations before you commit. Paste a prompt-response pair, pick criteria and a judge model, see the score in real time. Also available inline inside the trace view.
Simulate Improved

Simulator agent form and agent definition dropdowns

Configure simulation agents through a form interface — select target agent, define simulator persona, choose scenarios, launch. No YAML files.
Simulate Improved

Add scenarios from datasets

Import test scenarios directly from your existing datasets. Turn a collection of real customer transcripts into repeatable simulation test cases with a few clicks.
Platform Improved

Mixpanel analytics integration

Full Mixpanel integration across the platform. Track feature usage, evaluation runs, and simulation sessions to understand adoption and workflow bottlenecks.
SDK Improved

traceAI TypeScript Vercel instrumentor

Automatic observability for serverless AI deployments on Vercel. Wrap your handler, every LLM call, tool invocation, and response is captured as a span.
Evaluate Improved

CRUD on custom evaluations

Full create, read, update, and delete operations for custom evaluations — manage your evaluation library as it grows.
Evaluate Improved

Add feedback to evaluations

Attach human feedback directly to evaluation results. Build labeled datasets and improve evaluation accuracy over time.
Platform Fixed

Refresh token cycle for session management

Automatic token refresh cycle ensures uninterrupted simulation sessions without manual re-authentication.
Platform Fixed

Span name display in traces

Span names now appear directly in the trace view, making it faster to navigate complex agent execution trees.
Read full digest
W28 Jun 24 – Jul 7, 2025

System Metrics in Observe, Multimodal Bedrock Tracing, and Eval Playground Upgrades

Infrastructure metrics alongside agent traces in Observe, image tracing for AWS Bedrock, and standalone mode + feedback loops in the Eval Playground.

Monitor New

System metrics in Observe graph

System metrics in Observe graph

System metrics are now selectable directly from the Observe graph dropdown — render them in the primary graph or the compare graph alongside agent traces.

4 chart types
25+ instrumentors
Monitor New

Multimodal support for AWS Bedrock

Trace image inputs and outputs for AWS Bedrock multimodal models (Amazon Titan Image Generator, Anthropic Claude on Bedrock).
Evaluate Improved

Eval Playground improvements

Standalone evaluation mode, feedback collection on playground results, and clearer scoring visualization.
Evaluate Improved

Multi-line graphs in evaluations

Plot multiple evaluation metrics on a single chart — overlay hallucination rate against relevance score, or compare accuracy across models on the same axes.
Platform Improved

API key management revamp

Redesigned API key management with a cleaner UI, bulk operations, and clearer permission displays.
SDK Improved

Google GenAI instrumentor

Automatic tracing for Google's Generative AI SDK with support for Gemini models and function calling.
Evaluate Improved

Langfuse evaluations integration

Route Langfuse evaluation data into Future AGI for unified analysis alongside native evaluations.
SDK Improved

traceAI v0.1.11

Google GenAI instrumentor and multimodal AWS Bedrock support in the core SDK.
Monitor Fixed

Annotation notes

Add free-form text notes to annotations for richer context on human feedback.
Platform Fixed

Draft prompts

Save work-in-progress prompts as drafts without publishing them to the shared prompt library.
Read full digest
W26 Jun 10 – Jun 23, 2025

Alerts and Monitors, gRPC Trace Ingestion, and the Observe Graph

Real-time alerts with Slack and email, gRPC trace transport with 60% less latency, and a visual graph of agent execution inside Observe.

Monitor New

Alerts and monitors

Alerts and monitors

Set thresholds on any evaluation metric, trace property, or system measurement — get notified in Slack or email when the threshold is crossed.

60% less latency with gRPC
2 notification channels
Platform New

gRPC support for trace ingestion

High-performance gRPC transport for trace data, with 60% lower latency than HTTP/JSON.
Monitor New

Observe graph visualization

An interactive directed graph view of agent execution. Each node is a span, each edge is the flow of execution.
Platform Improved

Developer keys

API key management with scoped permissions — read-only, write, or admin keys, each with its own usage metrics and one-click rotation.
Platform Improved

Model serving infrastructure

New internal infrastructure for serving evaluation and guardrail models. Autoscales based on demand, optimizes for low-latency inference.
SDK Improved

traceAI gRPC transport support

Python and TypeScript SDKs now support gRPC as a trace transport alongside HTTP.
Evaluate Improved

Eval tab revamp

Redesigned evaluation results tab with better data visualization, more flexible filtering, and one-click CSV and JSON export.
Evaluate Fixed

Prompt eval updates

Improved prompt evaluation workflow with batch execution and result comparison across prompt versions.
Read full digest
W24 May 27 – Jun 9, 2025

Breaking Bad UI Redesign, Custom Model Endpoints, and Observe Enhancements

Platform UI redesign: new navigation, component library, consistent patterns. Azure OpenAI and self-hosted judges. New Observe filters and provider logos.

Platform New

Breaking Bad — platform update

Breaking Bad — platform update

A platform-wide release: model-choice in evaluations, enhanced error localization, extended alerts, and a redesigned UI across every surface.

3 custom judge endpoint types
3x faster dataset loading
Evaluate New

Custom model dropdown with Azure and custom endpoints

Use Azure OpenAI deployments, OpenAI-compatible endpoints, or self-hosted models as your evaluation judge — critical for teams with data residency, cost, or fine-tuned-judge requirements.
Monitor Improved

Attribute filters in Observe

Filter traces by custom attributes, metadata, and span properties in the Observe view.
Monitor Improved

Sentry error monitoring integration

When Sentry captures an exception in your app, the corresponding trace links automatically. Full-stack debugging context from application error to agent execution path.
Evaluate Improved

Image and audio support in evaluation log table

View image and audio outputs directly in evaluation log tables with inline rendering.
Platform Improved

Faster dataset loading

3x performance improvement for dataset loading through pagination and lazy rendering.
Evaluate Improved

Evaluation feedback in Observe

Attach feedback directly to evaluation results viewed from the Observe surface.
SDK Improved

traceAI Google ADK support

Automatic instrumentation for Google's Agent Development Kit (ADK) with full span capture and tool-call tracing.
SDK Improved

traceAI TypeScript: new evaluation support

Expanded evaluation capabilities in the TypeScript SDK, with new metric types and batch submission.
Evaluate Fixed

Eval template validation

Automatic validation of evaluation templates before execution to catch configuration errors early.
Monitor Fixed

Provider logos for tracing

Visual provider identification in trace views with logos for OpenAI, Anthropic, Google, and other LLM providers.
Read full digest
W22 May 13 – May 26, 2025

Protect Flash, TypeScript SDK v0.1.0, and Custom Evaluations in Observe

Speed-optimized guardrails path with binary harmful/not-harmful decision, the first official TypeScript SDK, and configurable evals on production traces.

Guard New

Protect Flash

Protect Flash

A speed-optimized binary content moderation path for Protect — harmful / not-harmful classification when full multi-category protection is overkill.

binary Protect Flash classification
1st TypeScript SDK
Evaluate New

Custom evaluations in Observe

Run configurable custom evaluations directly on production traces from the Observe view — no exports or separate evaluation runs.
SDK New

TypeScript @traceai/fi-core v0.1.0

The first official TypeScript SDK. Full tracing and evaluation for Node.js, Deno, and edge runtimes like Cloudflare Workers.
Platform Improved

API-based pricing for evaluations and error localizer

Pay-per-use pricing for evaluation runs and error localization, replacing fixed-tier limits. Start small, scale without hitting plan boundaries.
Platform Improved

Stop streaming for long-running prompts

Cancel in-progress LLM generations mid-stream to save time and tokens on runaway outputs. The partial output is preserved so you can still analyze what went wrong.
Evaluate Improved

Evaluations in the prompt Workbench

Run evaluations directly inside the Workbench to score prompt outputs without leaving the editor.
Platform Improved

Add to dataset from Prototype and Observe

Push prompt-response pairs from Prototype directly into any evaluation dataset, and add traces from Observe into datasets with one click. Closes the loop between iteration and systematic evaluation.
Evaluate Improved

Import saved prompts in datasets

Pull prompts from your saved prompt library directly into dataset rows for evaluation.
Evaluate Fixed

Feedback enhancement system

Structured feedback collection on evaluation results to continuously improve eval accuracy.
Read full digest
W20 Apr 29 – May 12, 2025

Workbench V2, Custom Evaluations Revamp, and SDK Updates

Rebuilt Workbench for prompt engineering, redesigned eval builder with judge-model selection, and three traceAI SDKs: audio, image, multimodal.

Platform New

Workbench V2

Workbench V2

A rebuilt prompt engineering environment. New multi-section prompt editor, resizable playground layout, prompt cards to organize your prompt library, and inline editing in the test case grid.

3 new SDK versions
12 prompt templates
Evaluate New

Custom evaluations revamp with model dropdown

Redesigned custom evaluation builder. Pick which LLM serves as the judge — and compare how different judges score the same data.
Monitor Improved

Annotations revamp with add/compare flow

Cleaner annotation workflows. The add flow is a single panel instead of a multi-step modal. Compare places two annotated traces side by side with differences highlighted.
Evaluate Improved

Sheet UI revamp for datasets

Refined spreadsheet interface with arrow-key cell navigation, keyboard shortcuts, and better scroll performance on datasets with thousands of rows.
Evaluate Improved

Import saved prompts into datasets

Pull prompts from your saved prompt library directly into dataset rows for evaluation.
SDK Improved

traceAI core v0.1.4

Audio evaluation support and prototype evaluation validation in the core SDK.
SDK Improved

traceAI OpenAI v0.1.3

The OpenAI instrumentor (library that captures OpenAI API calls automatically) now covers audio generation and image generation models.
SDK Improved

traceAI LangChain v0.1.4

The LangChain instrumentor adds image extraction from multimodal chains and support for OpenAI's Computer Use Agent.
Evaluate Fixed

Column configure dropdown in compare view

Choose which columns to display in comparison views to focus on the metrics that matter.
Evaluate Fixed

Delete dataset functionality

Clean up old or unused datasets with a delete option and confirmation safeguard.
Read full digest
W18 Apr 15 – Apr 28, 2025

Diff View in Experiments, Audio Across the Platform, and Run-Insight Views

Compare two experiment runs side by side, play audio inline in traces and datasets, and see every evaluation run at a glance with insight summaries.

Evaluate New

Diff view in experiments

Diff view in experiments

Compare two experiment runs side by side. Output differences, score deltas, and configuration changes highlighted inline.

2 experiment configurations compared
3-min KB tutorial walkthrough
Platform New

Audio support across Observe and datasets

Audio is now built in across the platform — play clips inside any trace, view waveforms in dataset tables.
Evaluate Improved

Run insight views

Every evaluation run now opens to a summary view: pass/fail distribution, score trends across runs, and outliers called out automatically.
Platform Improved

Rate-limit error UX and alerts

Clearer error messages when API rate limits are hit, plus in-app alerts when limit issues are detected.
Platform Improved

Knowledge base tutorial video

An embedded 3-minute walkthrough video inside the Knowledge Base section — pairs with the Prototype V2 knowledge base UI shipped in w16.
Evaluate Improved

Compare run eval revamp

Refined comparison view for evaluation runs with cleaner score display and easier navigation between runs.
Platform Improved

Prototype Observe filters

Apply the same filter vocabulary used in Observe to Prototype runs — consistency across the two surfaces.
Evaluate Fixed

Synthetic data UI improvements

Cleaner interface for synthetic data generation with better progress indicators and batch controls.
Read full digest
W16 Apr 1 – Apr 14, 2025

Prototype V2 and Audio Evaluations

Rebuilt prompt engineering environment with built-in knowledge base, evaluations that run on the actual audio of voice calls, plus a guided walkthrough.

Platform New

Prototype V2 with knowledge base UI

Prototype V2 with knowledge base UI

A rebuilt prompt engineering environment. Attach reference docs and few-shot examples next to your prompt, watch a 3-minute tutorial in-app, and iterate without switching tabs.

6 new eval templates
2x faster dataset loading
Evaluate New

Audio evaluations

Run evaluations on voice agent calls directly — audio content support across the platform, with three new audio evaluators for transcription, description, and quality assessment.
Monitor New

Error Localization in Prototype

Automatic root-cause detection inside Prototype runs. When a prototype run fails, the exact step where the problem originated is highlighted for you.
Evaluate Improved

Compare datasets with diff view

See exactly what changed between two dataset versions. Added rows highlight green, removed rows red, modified cells show inline diffs.
Evaluate Improved

Search in datasets

Full-text search across any column in a dataset — prompts, expected outputs, metadata. Works on datasets with thousands of rows.
Platform Improved

Gmail signup option

Sign up with Gmail. One click, no password required.
Platform Improved

First-time user walkthrough onboarding

New accounts get a guided tour of the platform — first evaluation run in under five minutes.
Monitor Improved

Quick filters for annotations

Filter annotations by label, reviewer, or status without leaving the annotation panel.
Evaluate Fixed

Audio cell renderer in datasets

Inline audio playback in dataset tables so you can listen to audio samples without leaving the dataset view.
Read full digest