What Is Few-Shot Prompting?

What Is Few-Shot Prompting?

Few-shot prompting is a prompt-engineering technique that puts a few input-output examples inside an LLM prompt so the model infers the desired pattern at inference time. In production, it appears in the prompt template, eval pipeline, and trace fields that capture prompt tokens and output quality. FutureAGI evaluates whether examples improve task success rather than only making demos look good, then optimizes example subsets and ordering with BayesianSearchOptimizer.

Why It Matters in Production LLM and Agent Systems

Bad few-shot examples create quiet regressions. A support classifier may copy the label style from a stale example, a JSON generator may learn an outdated field name, or a tool-using agent may imitate an example that chose the wrong tool for a similar-looking request. The model is not “learning” in the training sense; it is pattern-matching inside the current context window. That makes the examples powerful, but also brittle.

The pain lands on several teams. Developers see format drift after adding one “helpful” edge-case example. SREs see p99 latency rise because every request now carries 900 extra prompt tokens. Product managers see inconsistent behavior across customer cohorts because the examples overfit a narrow slice of traffic. Compliance teams struggle when examples contain sensitive customer text or policy language that no longer matches the approved workflow.

In logs and metrics, the symptoms are concrete: llm.token_count.prompt climbs, eval-fail-rate-by-cohort diverges, exact JSON-schema failures appear after a prompt edit, and thumbs-down comments mention “answered like the example, not my question.” Agentic systems amplify the issue. A bad example in the planner prompt becomes a wrong plan; a wrong plan triggers the wrong retrieval or tool call; the final answer then looks like a model failure even though the root cause was example selection. Unlike zero-shot prompting, few-shot prompting adds data-like behavior to the prompt, so it needs data-quality discipline.

How FutureAGI Handles Few-Shot Prompting

FutureAGI’s approach is to treat few-shot examples as measurable prompt assets, not decorative prompt text. A team stores the examples inside a versioned fi.prompt.Prompt template, runs the template against a held-out eval dataset, and traces each run through the relevant integration, such as traceAI-langchain, with prompt-version metadata and llm.token_count.prompt. The examples are evaluated by downstream task metrics, not by whether they look clean in a prompt review.

The anchor surface for this term is optimizer:BayesianSearchOptimizer. In agent-opt, BayesianSearchOptimizer is built for few-shot example selection: it uses Optuna TPE over example subsets and ordering. Concretely, a claims-intake agent may have 40 candidate examples but only room for six. FutureAGI can score candidate sets against PromptAdherence, TaskCompletion, and cost, then choose the subset that improves the eval cohort without pushing latency past the release threshold.

The engineer’s next action is operational. If the winning example set raises TaskCompletion from 0.74 to 0.83 but adds 38% prompt tokens, the team can set a regression gate: do not ship unless task score stays above 0.81 and prompt-token cost stays below the budget. If a new model release changes behavior, the same cohort is re-run and the prompt version is rolled back or re-optimized. Unlike a one-off DSPy or promptfoo sweep, the result is tied to traced production prompts, evaluator scores, and a repeatable release decision.

How to Measure or Detect It

Measure the examples by the behavior they cause, not by how plausible they read:

• PromptAdherence: scores whether the output follows the instructions and example-implied format.
• TaskCompletion: captures whether the full task succeeds with a given example set.
• JSONValidation or schema-failure rate: catches examples that teach stale field names or invalid structures.
• llm.token_count.prompt: tracks the token cost added by examples per trace.
• Eval-fail-rate-by-cohort: detects when examples help the happy path but hurt long-tail inputs.
• User-feedback proxy: compare thumbs-down rate and escalation rate before and after example changes.

Minimal Python:

from fi.evals import PromptAdherence

few_shot_prompt = "Classify sentiment.\nPositive: loved it\nNegative: slow refund"
result = PromptAdherence().evaluate(
    input="The setup was quick, but support never replied.",
    output="mixed",
    prompt=few_shot_prompt,
)
print(result.score, result.reason)

Common Mistakes

• Choosing examples by vibes. Pick examples from production error clusters, not from the three cases that looked good in a demo.
• Forgetting order sensitivity. Many models overweight later examples; test ordering, not just membership.
• Mixing labels and rationales inconsistently. If one example includes reasoning and another does not, outputs may alternate between terse and verbose formats.
• Adding examples without a token budget. A better score can be a bad release if p99 latency or cost jumps past the service target.
• Reusing examples across model families. A set tuned for Claude may not transfer to GPT, Gemini, or a smaller open model.