What Is Bayesian Prompt Search?

What Is Bayesian Prompt Search?

Bayesian prompt search is a prompt-optimization technique that uses Bayesian optimization to find higher-scoring prompt variants with fewer eval runs than brute-force search. In production, it shows up in the prompt eval pipeline: candidate instructions, few-shot example subsets, and example orderings are scored against task quality, token cost, and latency. FutureAGI connects this loop to optimizer:BayesianSearchOptimizer, so engineers can compare candidates, trace regressions, and promote a measured prompt version.

Why It Matters in Production LLM and Agent Systems

Ignoring Bayesian prompt search usually means treating prompt selection as a manual sweep. That fails when the search space is combinatorial: ten possible examples, five slots, three orderings, two system instructions, and multiple models already create more candidates than a team can evaluate by hand. The result is overfitting to a demo prompt, stale few-shot examples, or a prompt edit that improves one cohort while breaking another.

The pain is concrete. Developers see TaskCompletion move up on internal tests but fall for production edge cases. SREs see p99 latency and token spend rise because a “better” prompt carries too many examples. Product teams see inconsistent behavior across tenants because example selection was tuned on one customer’s data. Compliance teams worry when optimized examples include old policy language or sensitive text copied from incidents.

In traces, the symptoms are visible: rising llm.token_count.prompt, flat eval-score curves after many candidate runs, high variance between prompt versions, and eval-fail-rate-by-cohort spikes after a prompt release. Agentic systems make the problem sharper. A planner prompt may choose the wrong next step, a tool prompt may call the wrong function, and a final-answer prompt may hide the mistake. Bayesian prompt search helps allocate scarce eval budget to the candidates most likely to improve the whole chain.

How FutureAGI Handles Bayesian Prompt Search

FutureAGI’s approach is to tie prompt search to versioned prompts, evaluator scores, and production traces. The anchor surface is optimizer:BayesianSearchOptimizer. In agent-opt, BayesianSearchOptimizer is the Bayesian Search optimizer for few-shot example selection: it uses Optuna TPE over example subsets and ordering, then scores each candidate against an eval dataset.

A real workflow starts with a fi.prompt.Prompt template and a candidate pool of examples. Suppose a support agent has 36 labeled examples but can only fit six before latency exceeds the target. The engineer runs BayesianSearchOptimizer over subsets and orderings, evaluates each candidate with PromptAdherence and TaskCompletion, and tracks llm.token_count.prompt through traceAI-langchain. Unlike random prompt search, the next candidate is informed by prior scores instead of sampled blindly.

The release decision is operational. If candidate B raises TaskCompletion from 0.76 to 0.84 but increases prompt tokens by 42%, the engineer can set a gate: ship only if TaskCompletion >= 0.82, schema failures stay below 1%, and prompt-token cost stays under budget. If a model upgrade changes the score curve, FutureAGI reruns the same cohort, compares prompt versions, and either rolls back, re-optimizes, or promotes the new candidate behind a regression eval.

How to Measure or Detect It

Measure Bayesian prompt search by candidate quality, search efficiency, and production side effects:

• PromptAdherence: scores whether the output follows the candidate prompt’s instructions and example-implied format.
• TaskCompletion: measures whether the optimized prompt improves the end-to-end task, not just wording style.
• llm.token_count.prompt: tracks the cost and latency pressure added by longer instructions or few-shot examples.
• Eval-score improvement per run: compares the best score after N candidates against random-prompt-search baselines.
• Eval-fail-rate-by-cohort: catches candidates that improve aggregate score while damaging a customer, language, or task slice.

Minimal Python:

from fi.evals import PromptAdherence

prompt = "Ask for the order id before approving refunds."
output = "Please share your order id so I can check eligibility."
result = PromptAdherence().evaluate(input="Refund this order", output=output, prompt=prompt)
print(result.score, result.reason)

Common Mistakes

• Searching prompts without a held-out eval set. Bayesian search will optimize the metric you give it, including a biased or tiny development set.
• Optimizing aggregate score only. Track cohort-level deltas; the winning prompt can hurt high-value accounts while improving the mean.
• Ignoring prompt-token cost. Example-rich candidates often win quality metrics while violating latency, margin, or context-window budgets.
• Treating example order as harmless. Many models overweight nearby examples, so subset selection and ordering must be evaluated together.
• Shipping the optimizer’s winner automatically. Human review still matters for policy text, privacy exposure, and business-specific tone.