What Is Few-Shot Learning?

What Is Few-Shot Learning?

Few-shot learning is a prompt-engineering technique where an LLM learns a task pattern from a small set of examples supplied in the prompt or eval dataset. The examples show the desired input-output shape, reasoning style, tool choice, or refusal behavior before the live request arrives. In production traces, FutureAGI treats those examples as versioned prompt inputs: engineers compare output quality, token cost, and regression risk before shipping a prompt change.

Why It Matters in Production LLM and Agent Systems

Few-shot examples fail quietly. A support agent may copy the tone of an example that was written for refunds and apply it to account-security cases. A data-extraction workflow may follow the first JSON example even when later rows require nullable fields. A tool-using agent may choose the tool shown most recently, not the tool that fits the request. These are not syntax errors; they are behavioral regressions caused by example choice, example order, and example coverage.

The pain lands on several teams. Developers see green unit tests but rising eval failures on long-tail cohorts. SREs see prompt tokens grow after someone adds “just three more examples” to fix a single complaint. Product teams see inconsistent answers across segments because the few-shot set over-represents happy-path customers. Compliance reviewers need to know why a refusal example did not generalize to a regulated request.

In 2026 agent stacks, few-shot learning is rarely isolated to one chat turn. The planner may use examples for routing, the extractor may use examples for schema shape, and the summarizer may use examples for final wording. Symptoms show up as higher llm.token_count.prompt, lower PromptAdherence, more schema-validation failures, worse task completion on unseen cohorts, or a spike in user thumbs-down events after a prompt version changes.

How FutureAGI Handles Few-Shot Learning

FutureAGI’s approach is to treat few-shot examples as a tunable dataset, not prose glued onto the bottom of a prompt. The key anchor is the optimizer:BayesianSearchOptimizer surface. In the FutureAGI agent-opt inventory, BayesianSearchOptimizer is the optimizer built for few-shot example selection: it uses Optuna TPE over example subsets and ordering, then scores candidates against the team’s eval cohort.

Real example: a banking support agent has 40 labeled examples for “explain a rejected transfer” and can fit only five into the production prompt. The engineer defines a seed prompt, a candidate example pool, and a holdout eval set. BayesianSearchOptimizer searches which five examples, and in which order, maximize PromptAdherence and AnswerRelevancy while keeping llm.token_count.prompt under budget. Each candidate run is tied to a prompt version, so the team can compare eval-fail-rate-by-cohort before release.

The workflow does not stop at a winning prompt. FutureAGI records the prompt template id, selected example ids, model, cost, and output scores in traces. If the new set improves answer relevance but increases refusal errors, the engineer can add a threshold, inspect failed traces, and rerun the optimizer with a stricter evaluator mix. Unlike a plain promptfoo pass/fail run, this treats example choice and order as variables to optimize, not constants to freeze after one manual sweep.

How to Measure or Detect It

Measure few-shot learning by comparing example sets against the same holdout workload:

• PromptAdherence: checks whether the output follows the instructions implied by the prompt and examples.
• AnswerRelevancy: measures how well the response addresses the user request; useful when examples improve format but hurt usefulness.
• llm.token_count.prompt: catches example creep before a prompt becomes too expensive or too slow.
• Eval-fail-rate-by-cohort: shows whether an example set helps the average case while hurting regulated, low-frequency, or high-value requests.
• User feedback proxies: thumbs-down rate, escalation rate, and support reopen rate after a few-shot prompt rollout.

Minimal fi.evals check:

from fi.evals import PromptAdherence, AnswerRelevancy

prompt_score = PromptAdherence().evaluate(
    input=user_request,
    output=model_response,
)
relevance_score = AnswerRelevancy().evaluate(
    input=user_request,
    output=model_response,
)

Common Mistakes

Few-shot learning looks simple because the prompt is readable. The failure modes come from sampling, ordering, and measurement:

• Treating launch examples as permanent truth. Early customers are rarely a representative eval cohort; rotate examples after production feedback.
• Mixing output schemas in one prompt. The model may average incompatible examples and emit JSON that passes syntax but fails business rules.
• Adding examples without cost checks. Every example increases llm.token_count.prompt, latency, and gateway spend.
• Ignoring example order. Recency effects can make the last example dominate tool choice, tone, or refusal behavior.
• Measuring only the winner. Always keep the previous prompt as a baseline and compare against a holdout set.