Automated Optimization for Agents in 2026: Five Axes, Six Optimizers, One Production Loop

Agent optimization is an automated search for a better agent configuration, scored against a measured eval set. It runs on five axes: the system prompt, the tool descriptions the LLM reads when picking a tool, the retrieval configuration, the few-shot bundle, and the model itself. Each axis has its own search space, optimizer family, and cost per evaluation, and a prompt-only search never touches the other four.

TL;DR

• Agent optimization has five axes: system prompt, tool descriptions, retrieval config, few-shot bundle, and model.
• Text axes take ProTeGi, GEPA, Meta-Prompt, or PromptWizard. Few-shot selection takes BayesianSearchOptimizer. Retrieval config takes a TPE search such as AgentTPEOptimizer.
• Lock four axes, optimize the fifth, and re-score the winner on a held-out slice before you ship.

Why “Optimize the Prompt” Misses the Agent

An LLM is a function from string to string, so its optimization surface is one axis. An agent is something larger. It picks tools from a registered set, decides which arguments to pass, calls a retriever, reads the result, decides whether to call another tool, and eventually produces an output. A clean score on the final response tells you nothing about the four upstream decisions that produced it.

The practical consequence: an agent that scores, say, 71 percent on your task can fail in five distinct places, and each place has a different fix. The system prompt may be ambiguous about output format. The tool descriptions may not distinguish two near-duplicate tools the agent confuses. The retriever may be returning chunks too large to fit useful context.

The few-shot bundle may show the wrong failure modes. The model may not reason at the depth the task requires. A prompt-only optimizer can only see the first. Run it long enough and you’ll find a prompt that locally beats your eval set, ship it, then watch production regress because the actual bottleneck was somewhere the optimizer couldn’t reach.

The Five Axes, Each With Its Own Search Space

Each axis is a real surface on a production agent. Each has a different shape and a different optimizer.

System prompt. The most-discussed axis and the one with the strongest research literature. Free-form text-gradient (ProTeGi) and reflective evolutionary (GEPA) methods both shine here. PromptWizard fits too, because it rewrites wording rather than picking examples.

Tool descriptions. The most underrated axis: the LLM reads the description and parameter docstring on every turn before deciding which tool to call. Two tools with overlapping descriptions create chronic tool-selection errors that no system-prompt edit fixes. The reward signal has to come from tool-call correctness at the span level, not final-answer quality.

Retrieval config. Chunk size, overlap, top-k, reranker, embedding model, query rewrites, and metadata filters make a structured space of numeric and categorical parameters, with no text for a free-form optimizer to rewrite. AgentTPEOptimizer in agent-learning-kit fits here: its TPE sampler picks candidate values for each config path, which matters because each trial is expensive (you re-index, re-run retrieval, re-eval).

Few-shot bundle. Which demonstration examples live in the prompt, how many, and in what order, a bounded combinatorial pick of k from N. BayesianSearchOptimizer searches how many examples to include and which ones. Its min_examples and max_examples settings bound how many examples one trial includes.

Model. Sometimes the model is the bottleneck. The “optimizer” is a structured comparison: run the same eval suite against each candidate model with the current prompt and tool set, score on a multi-objective rubric (quality, latency, cost), pick.

The mistake to avoid: optimizing one axis while leaving the others moving. Lock the four other axes before optimizing the fifth, then move to the next. Move two at once and a score change cannot be traced to either one.

The Reward Signal Steers Every Optimizer

Every optimizer is a search procedure against a metric. The metric bounds everything; if it’s wrong, the optimizer finds prompts that exploit the wrongness. Three rules for designing the reward signal across all five axes.

Score the right step. Final-answer quality is necessary but not sufficient. If you optimize tool descriptions using only final-answer scores, the optimizer can’t tell whether a failure came from tool selection, argument formatting, or downstream reasoning. Each axis needs an evaluator that scores the step it controls.

Tool descriptions: score tool-call correctness at the span level. Retrieval: score context-precision and context-recall on the retrieved chunks. Few-shot bundle: score consistency between the bundle’s pattern and the agent’s answer. System prompt: score instruction-adherence plus the rubric dimensions you care about (groundedness, format compliance, safety).

Use the same evaluators in CI and production. Future AGI’s agent-learning-kit ships the fi.evals engine: custom evals plus built-ins such as Groundedness, Tool Call Accuracy, Task Completion, Tone, and Toxicity. Register the evaluators once, reuse the same evaluator in the optimization loop, the CI gate, and the online monitor. If the optimizer scores high on a metric production never measures, you’ve optimized the wrong thing.

Hold a slice out. Split into training, validation, and held-out test: the optimizer sees training, validation picks the winner, and the held-out slice stays unseen until final verification. The gap between training and validation tells you how much the optimizer overfit; more than a couple of points and the eval set is too narrow or the optimizer is gaming the rubric.

One more guardrail: add a length penalty to text-axis optimizers. Most LLM judges are length-biased, so free-form optimizers drift toward verbosity. A soft cap in the score aggregator keeps prompts and tool descriptions tight. The LLM-as-judge guide covers the failure modes that bite hardest.

The Six Optimizers and Which Axis They Fit

All six live in fi.opt.optimizers inside agent-learning-kit (pip install agent-learning-kit, Apache-2.0), next to the config optimizers covered below. The companion automated prompt improvement post covers each algorithm’s internals in depth; here’s the mapping by axis.

• RandomSearchOptimizer. A teacher model generates N paraphrases of the seed text and the highest scorer wins. The baseline that pays for itself. If RandomSearch moves the score, the wording has headroom and a directed optimizer will move it further. If it doesn’t move, the wording isn’t the issue and no optimizer in this list will save you.
• BayesianSearchOptimizer. Optuna’s TPE sampler over bounded parameters. The right pick for few-shot bundle selection. Resumable across CI runs via storage and study_name, which matters when each trial is expensive.
• MetaPromptOptimizer. A teacher LLM reads the current prompt and the failures, states a hypothesis, and writes a single rewrite. The hypothesis is the auditable artifact. Use it for system prompts on shorter runs where you want one strong rewrite per round.
• ProTeGi. Text gradients with beam search. Sample failing examples, write distinct critiques, generate improved variants from each critique, keep the top beam. Use it on system prompts and tool descriptions when failures cluster into nameable critiques (classification, structured extraction, tool selection).
• GEPAOptimizer. Reflective evolutionary search that combines lessons from the Pareto frontier of its own attempts, capped by a metric-call budget. Use it when you want broad exploration across many prompt variants under a fixed evaluation budget. The GEPA paper (arXiv:2507.19457) reports up to 35x fewer rollouts than GRPO.
• PromptWizardOptimizer. Microsoft’s mutate-critique-refine loop with thinking-style mixing. Most opinionated about how a prompt should be rewritten. Use it when you want stylistic diversity in the candidate pool.

The decision rule: start with RandomSearch on whichever axis you suspect is the bottleneck. If it moves the score, switch to ProTeGi or GEPA for text axes, BayesianSearch for the few-shot bundle, AgentTPEOptimizer for retrieval config. If it doesn’t, the bottleneck is somewhere else (a different axis, the eval, the model) and another optimizer pass won’t help. The choosing an optimizer docs compare the six by signal.

Agent Optimizers for Tool Descriptions, Retrieval, and Model

The six optimizers above rewrite text or pick examples. An agent optimizer searches the rest of the configuration. In agent-learning-kit, AgentOptimizer and its siblings take an OptimizationTarget: a base config plus a search space that maps dot paths such as tools.0.description or retriever.top_k to candidate values.

• AgentTPEOptimizer: Optuna TPE sampling, for retrieval config and tool descriptions.
• AgentParetoOptimizer: Optuna NSGA-II multi-objective search, for a model axis where quality, cost, and latency trade off.
• AgentEvolutionOptimizer: evolutionary mutation, for changes that only help in combination.
• AgentBanditOptimizer: more repeat trials for promising configs when evaluation is noisy or expensive.

from fi.opt.optimizers import AgentTPEOptimizer
from fi.opt.targets import OptimizationTarget

target = OptimizationTarget(
    name="support-agent",
    base_config={"retriever": {"top_k": 3, "chunk_size": 512},
                 "tools": [{"description": "Search orders"}]},
    layers=["retriever", "tools"],
    search_space={
        "retriever.top_k": [3, 5, 8],
        "retriever.chunk_size": [256, 512, 1024],
        "tools.0.description": ["Search orders", "Search orders by id and email"],
    },
)

# my_eval_suite: your eval set, returns a 0.0 to 1.0 score for candidate.config
result = AgentTPEOptimizer(
    target, evaluate_candidate=lambda c: my_eval_suite(c.config), n_trials=24
).optimize()
print(result.final_score, result.best_candidate.config)

Managed clouds ship the same idea. Microsoft introduced an agent optimizer in Foundry Agent Service on June 3, 2026, and its docs now list instructions, skills, tool descriptions, and model selection as targets. Amazon Bedrock AgentCore added recommendations in preview on April 30, 2026, that write optimized system prompts and tool descriptions from production traces.

The Agent Optimization Loop in Production

One axis at a time. Same evaluators on both sides of the deployment line.

1. Lock the other four axes. Pin the model, the retrieval config, the few-shot bundle, and the tool set. The axis you’re optimizing is the only thing moving.
2. Build the eval set. Hand-label or synthesize 300 to 500 examples covering the failure modes you’ve seen. Split 70 / 20 / 10 into training, validation, held-out.
3. Pick the optimizer for the axis. Free-form text gets ProTeGi or GEPA. Few-shot selection gets BayesianSearch and retrieval config gets AgentTPEOptimizer. Model axis is a full eval pass per candidate, or AgentParetoOptimizer when quality, cost, and latency pull apart.
4. Bound the run. Configure EarlyStoppingConfig with patience (stop after N iterations without improvement), min_delta (the minimum improvement that counts), and max_evaluations (the hard cap). Wire max_evaluations to your dollar budget.
5. Run, validate, audit. Run against training, pick the winner by validation, re-score on the held-out slice, run a human audit on 30 to 50 production-shape outputs. If human ratings track the eval score, you have signal. If they don’t, the evaluator is the bottleneck.
6. Ship behind a flag, A / B on real traffic, promote on signal. The winning artifact (prompt, tool descriptions, retrieval config, few-shot bundle, or model) gets versioned. Every production span gets tagged so a regression is automatically attributable.
7. Mine the next round. Production traces produce a fresh failure cohort. That cohort becomes the next round’s training slice. traceAI auto-instruments these spans across Python, TypeScript, Java, and C# so the cohort builds itself.

Two cadences run in parallel. The offline optimization pass runs when you ship a new agent or swap models. The regression gate runs on every PR, scoring the candidate against the same evaluators the optimizer used. The optimizer is too slow and too expensive to run on every PR; the eval gate isn’t.

Production Gotchas

Five failure modes to keep on the dashboard.

• Eval-set overfit. The optimizer finds prompts that game the rubric without generalizing. The held-out test slice catches this when the gap between validation and test exceeds a few points. Without a held-out slice, this is silent until production finds it for you.
• LLM-judge contamination. If the same model writes the optimized prompt and judges its output, the judge is biased toward its own writing. Use a different model family for the judge, or pair it with deterministic rules (schema checks, regex, ground-truth match). The LLM-as-a-judge guide linked above covers the full failure list.
• Calibration drift on model upgrade. A prompt tuned on one model may regress when you swap to another model or a newer version of the same one. Same for tool descriptions and few-shot bundles. Re-run the optimization (or at minimum the eval) on every model swap, and tag spans with the model version. The model and prompt selection post covers the calibration side.
• Cost runaway from long prompts. Free-form optimizers drift toward verbosity. Track tokens-per-call as a guardrail metric; the winning prompt shouldn’t double your inference bill.
• Tool-selection regressions. Tool-call accuracy can rise on the eval set and fall on production traffic if production has tools the eval didn’t cover. Make sure every registered tool has at least 10 to 20 examples in the eval set, including ambiguous cases between similar tools.

Drive-Thru Voice Agent Case Study, 66 to 96 Percent

A worked example from the Future AGI team. The goal was a fast-food voice agent for “Future Burger” that survived real-customer chaos: hesitation, mid-order changes, interruptions, rushed speech. We used a brain-first architecture. STT and TTS are interchangeable peripherals.

The LLM is the brain that handles reasoning, context switching, and tool calls. If the agent can’t understand that “Actually, make that a Sprite” means removing the previous drink, no realistic voice synthesis saves it. Fix the intelligence first.

Step 1: Synthesize 500 scenarios. Future AGI’s synthetic data generator built a ground-truth set with user_transcript (what the customer says) and expected_order (what the agent should book) from a one-line prompt, a 500-row dataset covering modifications, combo meals, and “no pickles” cases. See the synthetic data docs for the pipeline.

Step 2: Baseline. Initial system prompt (“v0.1”) run against the 500 scenarios on gpt-5-nano, gemini-3-flash, and gpt-5-mini. Logic was decent (80 percent accuracy) but responses ran to multi-paragraph essays. Saved as the Control to beat.

Step 3: Simulate edge cases. We ran the agent through personas who hesitate, stutter, change their mind, lose patience, or speak in a rush, and it spoke too much (“Certainly! I have updated your order…”) and frustrated the simulated user. When the user changed their mind, the agent added both items to the cart, and the success rate dropped to 66 percent.

Step 4: Define the eval, then optimize. We defined 10 task-specific criteria including Context_Retention, Objection_Handling, and Language_Switching. Future AGI’s multi-modal evaluation suite handles audio directly, so the platform surfaced evaluation-driven fixes alongside the failing grade: reduce decision-tree depth for menu inquiries, restrict generative capabilities to the menu_items vector store to stop the agent inventing dishes. Instead of hand-editing the system prompt, we let ProTeGi run for 20 rounds, jointly optimizing Task_Completion and Customer_Interruption_Handling.

Step 5: Result. The optimized prompt scored 96 percent task completion on the held-out slice. The baseline was polite but slow and failed on complex changes, while the optimized agent handled the “Indecisive” scenario cleanly and stayed crisp: “Burger, no pickles. Got it.”

Before: wordy, error-prone baseline.

After: crisp, fast, handles mid-order changes.

The 96 percent number came from optimizing the system prompt with the other four axes locked. We had already swept retrieval config (chunk size on the menu_items index, top-k) the previous night and selected the model in a separate eval pass. The 30-point gain came from the last step of a five-axis loop.

Run All Five Axes in One Future AGI Loop

Future AGI’s agent-learning-kit ships all six prompt optimizers plus the config optimizers in fi.opt.optimizers, and its fi.evals engine supplies the reward signal. Model calls can route through the Agent Command Center gateway, where virtual keys default to your own provider accounts (BYOK). traceAI records each tool call and retrieval as its own OpenTelemetry span, so a failure maps to the right axis.

Once you apply the winning prompt, Future AGI keeps it as a named, versioned prompt you can label Production or Staging and change without shipping a deploy. Future AGI is SOC 2 Type II, offers a HIPAA BAA on paid add-ons, and lists GDPR and CCPA on its trust page.

Teams already on DSPy can keep it: traceAI’s DSPyInstrumentor sends DSPy runs into the same traces and evals. What Future AGI adds is one loop, where failing production traces become dataset rows for the next round and the same evaluator scores prompts in CI and in production.

The agent that scored 71 percent does not need a longer hunt for the perfect prompt. Lock four axes, find the one holding it back, and point the matching optimizer at it with your production evaluator. For the failure modes behind the five-axis frame, see why agents pass evals and fail in production, and for the wider set of libraries, see the top prompt optimization tools roundup.

Run your first agent optimization on Future AGI.

Sources

• PromptWizard repository (Microsoft’s mutate, critique, and refine loop with thinking-style variations)
• DSPy (Stanford NLP’s framework for programming LLM pipelines, instrumented by traceAI’s DSPyInstrumentor)
• Recursive Agent Optimization paper (Gandhi et al., submitted May 7, 2026, a reinforcement learning method for recursive agents)