Changelog

Chain prompts visually with input/output port mapping between nodes for complex multi-step agent workflows.

Multi-assignment and prefetching for annotation queues, reducing reviewer wait times and enabling parallel review workflows.

Four separate optimization PRs delivering 4x faster page loads across the platform.

Test voice agents using real call data captured through Observe, turning production conversations into simulation scenarios.

Add task descriptions to all optimizer types for better context and documentation of optimization goals.

Automatic saving with conflict resolution and manual save checkpoints in the Agent Playground.

Create, browse, compare, and restore named versions of agent playground graphs.

Improved error messages, cache invalidation, and recovery flows across all Agent Playground operations.

Trace storage upgraded to Replicated MergeTree for high availability and automatic failover.

Fine-grained APIs for node connections, ports, and edge mappings enabling programmatic agent graph management.

AI-assisted prompt generation from task descriptions and one-click prompt improvement suggestions.

Context-aware AI assistant embedded in the platform for trace debugging, simulation creation, evaluation building, and dataset construction.

Connect Future AGI to Cursor, Claude Code, VS Code, Claude Desktop, and Windsurf for access to evaluations, datasets, experiments, traces, and prompts.

Four roles at organization and workspace levels with two-factor authentication, passkeys, and recovery codes.

Connect Langfuse, Datadog, PostHog, PagerDuty, Mixpanel, S3, Azure Blob, and GCS from a single configuration page.

traceAI instrumentation now supports Java and C# in addition to Python and TypeScript.

MCP server ships with evaluate, protect, datasets, and synthetic data generation tool endpoints.

Trace storage moved to ClickHouse for massive query performance improvements on high-volume workloads.

Create annotation queues for traces, sessions, datasets, and simulation outputs to organize human review workflows.

Deploy and store data in US or EU regions for compliance with data residency requirements.

Fine-grained create, read, update, and delete operations for nodes, ports, and connections in the Agent Playground.

Unified analytics dashboard API covering dataset quality metrics and simulation result trends.

Instant API key revocation across all replicas via Redis pub/sub, closing the window between revocation and enforcement.

API key validation middleware for the Prism Gateway with rate limiting and usage tracking per key.

Comprehensive monochrome theme applied across the entire platform for reduced eye strain and visual consistency.

Full traceAI instrumentation support for C# applications and .NET environments.

Comprehensive Java SDK with 25 instrumentation modules covering major frameworks and libraries.

Massive expansion of TypeScript instrumentation covering frameworks, databases, HTTP clients, and more.

New framework support and end-to-end tests for the Python instrumentation SDK.

Each browser tab maintains independent workspace context via sessionStorage, preventing cross-tab interference.

Activate and manage specific versions of agent playground graphs with one-click version switching.

Role-based access control settings page with workspace-level usage metrics and member management.

Pass function parameters directly to evaluation metrics for dynamic scoring configurations.

Configure reasoning-specific parameters when working with chain-of-thought models in the Prompt Workbench.

Add and configure simulations directly through the Prompt Workbench without switching contexts.

Structured feedback with five label types and support for multiple reviewers on voice agent transcripts.

Agent Compass now supports voice agents, providing real-time health metrics for call-based AI systems.

Evaluations now accept and score multi-image inputs for comprehensive multimodal assessment.

First-class support for reasoning models with chain-of-thought visibility in traces and evaluations.

Simulation results stream to the grid in real time via WebSocket, eliminating manual refreshes.

The Prompt Workbench now renders image and audio outputs inline for multimodal prompt iteration.

Select Azure-specific endpoint types when configuring custom models, with proper API format handling.

Monitor, pause, resume, and cancel workflow executions directly from the Agent Playground interface.

Separate read and write database connections for improved query performance under load.

CSV, Excel, and JSON file processing rebuilt on Polars for dramatically faster dataset imports.

Optimized queries and pagination for simulation results and the evaluations dashboard.

Run evaluations directly from datasets and manage trial items without leaving the dataset view.

Datasets and the Prompt Workbench now render image outputs natively, supporting multimodal evaluation workflows.

Upload multiple images at once when building datasets, eliminating one-by-one file selection.

Compare real production conversations against simulated outputs to identify drift and regressions.

Select multiple test executions and delete or rerun them in a single action.

Core Graph, Node, and Execution models that power the Agent Playground with cycle detection and validation for graph connections.

Cloud mode, agent wrappers for OpenAI, LangChain, Gemini, and Anthropic, plus tool call support.

Simulate directly from real customer interactions with auto-generated transcripts and scenarios.

10 ready-to-use evaluation groups covering common quality dimensions for agent testing.

AI-powered debugging now works for chat-based agents with text-specific diagnostic capabilities.

Run prompt optimization strategies directly from the platform UI without API calls.

Model parameters automatically refresh based on provider API capabilities and availability.

Automatic validation of audio content format and quality before submission to audio models.

Full create, read, update, and delete API endpoints for managing replay sessions programmatically.

Streamlined optimization pipeline with progress tracking and intermediate result previews.

Assign and swap personas within scenarios using a simplified inline interface.

Real-time status tracking for all simulation runs with stage-level progress indicators.

Delete API keys from the dashboard to revoke access and maintain security hygiene.

Re-run historical production conversations through your current agent to test regressions and improvements.

Automated prompt optimization with GEPA, MetaPrompt, ProTeGI, and Bayesian strategies.

Select specific calls for targeted optimization rather than optimizing across the full dataset.

Inline PDF rendering in traces, sessions, and evaluation results.

Reference nested JSON fields with dot notation in prompt templates and configurations.

Dedicated persona configuration panel for chat simulation with role, personality, and behavior settings.

Inline analytics drawer showing cost, latency, and quality metrics for each chat simulation run.

Provide natural language instructions to guide AI-powered scenario generation.

Convert real production sessions into reusable simulation scenarios with one click.

Core simulation and optimization workloads migrated to Temporal for reliability and observability.

Full lifecycle management for simulation personas: view, duplicate, edit, and delete.

Modify experiment parameters mid-run without restarting from scratch.

Reference nested JSON fields using dot notation in prompt templates and experiment configs.

Structured data in traces now renders as formatted tables with sorting and filtering.

Preview PDFs and documents inline across the platform without downloading.

Redesigned audio player with lazy loading for faster page loads on session-heavy views.

Live progress indicators for ongoing calls with estimated time remaining.

Import agent configurations directly from Vapi and Retell with one click.

Backend infrastructure for the prompt optimiser including models, views, serializers, and Temporal migration.

Visualize and navigate branching conversation paths within generated test scenarios.

Define agents using custom providers beyond the standard Vapi and Retell integrations.

Add realistic ambient noise profiles to voice simulations for production-accurate testing.

AI-powered generation of branching conversation trees covering multiple user intent paths.

Backend migration from string to JSON fields enables structured data display in session views.

Human-readable summaries explaining why each evaluation scored the way it did.

Real-time prompt execution with streaming responses via WebSocket connections.

Remap evaluation variables after creation without rebuilding the entire evaluation configuration.

Sticky filters, pagination improvements, metadata display, and updated pricing logic in Observe.

Fixed edge cases where simulated assistant calls would not terminate cleanly.

Full observability for every simulation call with detailed logs, per-call latency, and granular cost attribution.

Contextual provider selection and streamlined configuration for prompt runs and experiments.

New evaluation dimensions for voice quality, latency, and naturalness in voice agent monitoring.

View the reasoning trace behind each simulation decision directly in the results table.

Use custom voices from Eleven Labs and Cartesia in prompt runs and experiment workflows.

Revised metrics suite in Run Test with more actionable performance indicators.

Modify evaluation configurations inline without leaving the experiment view.

Programmatically configure evaluation parameters and trigger re-runs through the API.

Errors in simulation runs are now pinpointed to the exact step and provider that caused them.

Improved session history with Indian language support and full transcript rendering.

Redesigned Observe landing page with faster load times and better navigation.

Workspace-level credit attribution with per-feature breakdowns and historical trends.

A 3-step guided flow for building agents from scratch with inline validation and previews.

Commit-based version history brings git-style prompt management to the Workbench.

Define agents that operate natively in 15+ languages with locale-aware behavior.

Enrich simulation scenarios with custom data columns using AI generation or manual entry.

Broader dialect and accent coverage for realistic multi-language voice simulations.

Redesigned metrics dashboard with real-time pass/fail rates and drill-down capabilities.

LLM-as-a-Judge, heuristic metrics for JSON, similarity, string matching, and aggregation.

Unified analytics for voice agent calls with cost, duration, and quality breakdowns.

Full request and response logs for every voice provider interaction during simulation.

Added Cartesia, Hume, Neuphonics, and LMNT as text-to-speech providers.

Native tracing support for LiveKit-powered real-time voice and video agents.

Simulate outbound voice flows where your agent places the call, testing appointment reminders, sales outreach, and proactive support scenarios.

Native Retell support for voice agent simulation, joining Vapi as the second supported voice provider.

Verify that voice agents call the correct tools and functions during simulation, catching integration errors before production.

Use the voice provider's native transcript as an evaluation input for comparing ASR accuracy and response quality.

Choose from pre-built caller personas or create custom ones with specific demographics, behaviors, and communication styles.

Streamlined onboarding with role-specific paths, interactive tutorials, and faster time-to-first-evaluation.

Cost tracking calculated at ingestion time for accurate, real-time usage reporting without post-processing delays.

Generate and evaluate voice outputs directly from the prompt playground and experiment workflows.

Add scenario rows manually, generate them with AI, or import from existing datasets for flexible test case management.

Apply new evaluation criteria to previously completed simulation test runs without re-executing the calls.

Select and test against specific agent definition versions for precise regression testing and A/B comparisons.

Initial SDK release with 50+ evaluation templates covering faithfulness, relevance, safety, and domain-specific quality metrics.

Batch evaluation support and bias detection capabilities added to the ai-evaluation SDK.

Native instrumentation for OpenAI's Agents SDK, capturing tool calls, handoffs, and multi-agent orchestration traces.

Full observability for Vapi-powered voice agents with call metrics, transcript analysis, and utterance-level performance tracking.

Run evaluation groups within experiments and use group-level scores for automated prompt and agent optimization.

Trigger ultra-low-latency customer call simulations programmatically through the SDK against LiveKit agents.

Rerun specific failed or flagged test cases without re-executing the entire simulation suite.

Pre-built evaluation groups for common use cases including RAG, Computer Vision, conversational AI, and more.

Auto-refresh dashboards, stop running simulations, and visual workflow tracing for simulation orchestration.

Redesigned Workbench with integrated code drawer and refreshed header for a streamlined development experience.

Updated agent definition interface with clearer layout and improved navigation for complex agent configurations.

Native session.id attribute support in traceAI for automatic session grouping across all instrumented frameworks.

Upload SOPs, transcripts, or documentation to auto-generate simulation scenarios with edge cases and branching conversation flows.

Version control for agent definitions with commit messages and consolidated test reports across versions.

Track user sessions with a single session.id attribute on spans, enabling session-level observability without complex setup.

Full CRUD operations for evaluation groups, enabling teams to organize and manage evaluation suites programmatically.

Listen to simulation calls with a multi-channel audio player that separates agent and caller audio streams.

Download call recordings in multiple audio formats for offline analysis, compliance archiving, and sharing.

Collaborative editing and commenting on prompts, enabling team-based prompt development workflows.

Resolved issues with importing annotations and prompt data into datasets for smoother data pipeline operations.

Dedicated background worker pool for trace ingestion, improving throughput and reducing processing latency.

End-to-end optimization of the trace ingestion pipeline for faster data availability and reduced resource consumption.

Comprehensive dashboard for annotator agreement metrics including Cohen's kappa scores and inter-rater reliability analysis.

Enterprise-grade security framework with multi-workspace isolation, RBAC policies, and comprehensive audit logging.

Advanced observability feed with automatic error clustering and trend analysis across your agent's execution history.

Guided onboarding flow that adapts to your role and use case, getting teams productive faster.

Dashboard metrics and scenario columns for voice agent simulations, providing deeper visibility into call performance.

Folder-based prompt architecture with templates for organizing large prompt libraries across teams and projects.

Redesigned pricing page with clearer plan comparisons and streamlined upgrade flows.

API support for evaluation grouping, enabling programmatic organization of related evaluations into logical sets.

Completely rebuilt summary dashboards with spider, bar, and pie chart visualizations plus side-by-side comparison views.

Rebuilt alerting system with Slack and email notification channels, customizable thresholds, and intelligent alert grouping.

Caching, A/B testing, and multi-environment deployment support in the Prompt SDK for production-grade prompt management.

Role-based access control with Owner, Admin, Member, and Viewer roles for fine-grained workspace permissions.

Purchase and manage your Future AGI subscription directly through AWS Marketplace with consolidated billing.

Synchronous and asynchronous standalone error localization through the SDK to pinpoint failures in agent execution chains.

Automatic detection of critical issues in datasets with actionable mitigation advice for data quality problems.

Track trace performance per prompt version in Observe to measure the real-world impact of prompt changes.

Reduced install bloat by making framework-specific dependencies optional, cutting package size significantly.

Define custom function-type evaluations that execute arbitrary logic against agent outputs for precise, deterministic quality checks.

Modify AI-generated synthetic data before committing it to your datasets, giving you full control over data quality.

Native support for TXT, DOC, DOCX, and PDF files as dataset columns, enabling document-centric evaluation workflows.

New user-level views in Dashboard and Observe surfaces aggregate metrics per end-user across sessions and traces.

Precise timestamp columns in trace and span views for accurate timing analysis and debugging.

Tag each prompt version with custom labels to track experiments, A/B tests, and rollout stages.

Submit evaluations asynchronously through the SDK, enabling non-blocking evaluation pipelines in production systems.

Capture and replay video outputs from multimodal agents directly within the Observe interface.

Optical character recognition for scanned documents and images, extracting text for evaluation and analysis.

Compare evaluation results and prompt summaries across two datasets side-by-side to measure improvement.

Submit annotations and feedback in bulk through the API and SDK for high-throughput labeling workflows.

Inspect raw evaluation log data in structured JSON format for debugging and integration purposes.

New traceAI release adds automatic prompt template labels to LLM spans for better trace organization.

Native Pipecat instrumentation for tracing voice and multimodal AI pipelines built with the Pipecat framework.

TypeScript instrumentation for LlamaIndex, bringing observability to the popular RAG framework in Node.js environments.

Sub-second voice latency powered by LiveKit infrastructure ensures simulation calls feel indistinguishable from real customer interactions.

Configure simulation agents through an intuitive form interface with dropdown-based agent definition selection.

Import test scenarios directly from your existing datasets to run voice simulations against real-world conversation patterns.

Automatic token refresh cycle ensures uninterrupted simulation sessions without manual re-authentication.

Full Mixpanel analytics integration across the platform for tracking usage patterns and feature adoption.

First-class Vercel instrumentation for traceAI in TypeScript, enabling seamless observability for serverless AI deployments.

Full create, read, update, and delete operations for custom evaluations, giving teams complete control over their evaluation workflows.

Span names now appear directly in the trace view, making it faster to navigate complex agent execution trees.

Attach human feedback directly to evaluation results to build labeled datasets and improve evaluation accuracy over time.

Built-in dashboards for CPU, memory, latency, and throughput metrics alongside agent traces.

Image tracing for AWS Bedrock models with full input/output capture of multimodal interactions.

Standalone evaluation mode, feedback collection, and improved scoring visualization in the playground.

Plot multiple evaluation metrics on a single chart to correlate trends and spot regressions across dimensions.

Redesigned API key management with improved UI, bulk operations, and clearer permission displays.

Automatic tracing for Google Generative AI SDK with support for Gemini models and function calling.

Backend integration enabling Langfuse evaluation data to flow into Future AGI for unified analysis.

Add free-form text notes to annotations for richer context on human feedback.

Save work-in-progress prompts as drafts without publishing them to the shared prompt library.

Google Gen AI instrumentor and multimodal AWS Bedrock support in the core SDK.

Configure metric-based alerts with Slack and email notifications for proactive production monitoring.

High-performance gRPC transport for trace data with 60% lower latency and reduced bandwidth overhead.

Interactive directed graph view of agent execution showing the flow of calls, branches, and dependencies.

Dedicated API key management with scoped permissions, rotation, and usage tracking.

Internal infrastructure for serving evaluation and guardrail models with autoscaling and low-latency inference.

Python and TypeScript SDKs now support gRPC as the trace transport protocol alongside HTTP.

Redesigned evaluation results tab with better data visualization, filtering, and export options.

Improved prompt evaluation workflow with batch execution and result comparison across prompt versions.

Evaluate individual spans directly in the trace view with on-demand scoring and result display.

Rebuilt experiment view with better comparison layouts, sortable metrics, and run history navigation.

Connect Sentry to surface application errors alongside agent traces for full-stack debugging.

Use Azure OpenAI, custom API endpoints, and self-hosted models as evaluation judges.

View image and audio outputs directly in evaluation log tables with inline rendering.

3x performance improvement for dataset loading through pagination optimization and lazy rendering.

Automatic validation of evaluation templates before execution to catch configuration errors early.

Filter traces by custom attributes, metadata, and span properties in the Observe view.

Visual provider identification in trace views with logos for OpenAI, Anthropic, Google, and other LLM providers.

Automatic instrumentation for Google Agent Development Kit with full span capture and tool call tracing.

Expanded evaluation capabilities in the TypeScript SDK with new metric types and batch submission.

Comprehensive redesign of every platform surface with new navigation, component library, and interaction patterns.

Run configurable custom evaluations directly on production traces from the Observe view.

First TypeScript SDK release enabling trace instrumentation and evaluations for Node.js and Deno applications.

Ultra-fast guardrails engine that screens LLM outputs in under 50ms for real-time protection.

Pay-per-use pricing for evaluation runs and error localization, replacing fixed tier limits.

Cancel in-progress LLM generations mid-stream to save time and tokens on runaway outputs.

Run evaluations directly within the workbench to score prompt outputs without leaving the editor.

Structured feedback collection on evaluation results to continuously improve eval accuracy.

New prompt editor, playground layout, prompt cards, and cell editing for a fundamentally better prompt engineering workflow.

Redesigned custom evaluation builder with model selection dropdown for choosing which LLM judges your outputs.

Overhauled annotation interface with streamlined add and side-by-side comparison workflows.

Refined spreadsheet interface with improved cell navigation, keyboard shortcuts, and performance for large datasets.

Pull prompts from your saved prompt library directly into dataset rows for evaluation.

Choose which columns to display in comparison views to focus on the metrics that matter.

Clean up old or unused datasets with a delete option and confirmation safeguard.

Audio evaluations support and prototype eval validation in the core SDK.

Audio and image generation model support for OpenAI instrumentor.

Image extraction and OpenAI CUA (Computer Use Agent) support for LangChain instrumentor.

Automatically identifies the root cause of failures in agent traces, highlighting the exact span where things went wrong.

Complete human feedback workflow for traces with multi-label annotations, reviewer assignment, and approval states.

Spreadsheet-style dataset editor with resizable columns, frozen headers, and bulk editing capabilities.

Compare experiment runs side by side with highlighted differences in outputs, scores, and configurations.

Native audio rendering in trace views and dataset tables for complete voice agent observability.

5x increase in API rate limits across all endpoints to support high-throughput production workloads.

Cleaner interface for synthetic data generation with better progress indicators and batch controls.

Rebuilt prototyping experience with integrated knowledge base, tutorial video, and streamlined prompt iteration workflow.

Evaluate audio agent conversations with conversational completeness metrics that measure how well agents handle spoken interactions.

Side-by-side diff comparison between dataset versions to track how your evaluation data evolves over time.

Full-text search across dataset rows to quickly find specific test cases, prompts, or expected outputs.

One-click signup with Gmail for faster onboarding, no password required.

Guided walkthrough for new users covering key platform features, eval setup, and trace inspection.

Filter annotations by label, reviewer, and status directly from the annotation panel.

Visual summaries of evaluation runs showing pass/fail distributions, score trends, and outlier detection.

Push prompt-response pairs directly from Prototype into your evaluation datasets with one click.

Inline audio playback in dataset tables so you can listen to audio samples without leaving the dataset view.