What Is Prompt Tuning? A 2026 Guide With a Code Example

Prompt tuning is a parameter-efficient way to adapt a frozen language model: you train a short sequence of continuous embedding vectors, called a soft prompt, and prepend it to every input while the model weights stay untouched. It sits between prompt engineering, where a person writes the prompt, and fine-tuning, where the weights change. This guide covers how it works, a Hugging Face PEFT example, and when it beats LoRA.

Prompt Tuning vs the Alternatives at a Glance

What Prompt Tuning Actually Is

Prompt tuning, in the sense introduced by Lester, Al-Rfou, and Constant in 2021, means learning a small tensor of continuous embeddings that gets prepended to the input token embeddings at inference time. The base model weights stay frozen. The soft prompt is the only thing that gets trained.

Concretely:

• The tensor is usually 20 to 100 virtual tokens (Lester et al. defaulted to 100 and saw only marginal gains past 20), each with the model’s hidden dimension (so for a 7B model with hidden size 4096, the soft prompt is about 82,000 to 410,000 parameters).
• Training is gradient descent on a labelled dataset against the frozen base model; you backpropagate through the model but only update the soft-prompt embeddings.
• At inference, the embedding layer’s normal output gets the soft prompt concatenated in front, and the rest of the forward pass is unchanged.

The original Lester et al. paper showed that prompt tuning gets more competitive with scale: once T5 models pass billions of parameters, it “closes the gap” with full model tuning. With T5-XXL, a 5-token soft prompt is 20,480 parameters, against 11 billion for each tuned copy of the model.

Four Techniques People Call Prompt Tuning

Four techniques share the name:

• Manual prompt engineering (hard prompts). Hand-written natural-language text. Editable, interpretable, ports across closed APIs. This is what most people mean when they say “prompt tuning” in casual conversation. It is not prompt tuning in the technical sense.
• Prompt tuning (soft prompts). The Lester et al. technique above. Learned vectors, frozen base model. Specialist tool for self-hosted deployments. The hard prompt vs soft prompt breakdown covers the distinction in depth.
• Prefix tuning (Li and Liang, 2021). A close cousin. Instead of one soft prompt at the input layer, learn a small prefix at every transformer layer. More capacity, many times more parameters (the prefix repeats at every layer), similar use cases.
• P-Tuning (Liu et al., 2021). Trains continuous prompt embeddings and concatenates them with discrete prompt tokens. PEFT implements it as PromptEncoderConfig. P-Tuning v2 (Liu et al., 2021) puts deep prompts at every layer, like prefix tuning, and reports matching fine-tuning with 0.1% to 3% of parameters tuned on NLU tasks.

This article is about the second one, with brief comparisons to the others. The Hugging Face PEFT library implements prompt tuning, prefix tuning and P-tuning behind one get_peft_model() call, so you can test all three against the same frozen base model and eval set.

Why Prompt Tuning Still Matters in 2026

The frontier-API era pushed a lot of teams away from any form of weight adaptation, because gpt-6-astra plus retrieval handles most tasks. Three reasons prompt tuning remains relevant:

1. Self-hosted open-weights stacks. Llama 4, Qwen3.5 and Mistral Large 3 all publish downloadable weights. If you host the model, you can prompt-tune it.
2. Multi-tenant adapters. A single hosted base model can serve many tenants by swapping a soft prompt of a few hundred KB to under 2 MB per tenant, instead of routing to a different fine-tuned copy.
3. Cost-sensitive narrow tasks. When you have a small dataset and a large frozen base model, prompt tuning can match full fine-tuning at a fraction of the training and storage cost.

For most teams on closed APIs, the right tool is prompt engineering plus retrieval, with full fine-tuning when retrieval is not enough. Hosted tuning is shrinking too: OpenAI’s fine-tuning platform no longer accepts new users, while Google’s Gemini Enterprise Agent Platform (formerly Vertex AI) still offers supervised tuning for selected Gemini models.

When Prompt Tuning Beats the Alternatives

Use prompt tuning when all of the following are true:

• You self-host an open-weights model (Qwen3, Llama 4, Mistral Large 3).
• You have a clearly defined narrow task with a labelled dataset of at least a few hundred examples.
• Prompt engineering plus retrieval has hit a measurable ceiling on your eval set.
• You need many task adapters in the same deployment, not one big fine-tuned copy.
• Inference complexity must stay minimal (no LoRA matrix multiplies inside attention).

For most other cases LoRA wins in 2026 because it adapts attention matrices throughout the network, captures more task signal, and has mature tooling (PEFT, Hugging Face, vLLM-side LoRA hot-swapping). vLLM can also load and unload LoRA adapters at runtime through its /v1/load_lora_adapter and /v1/unload_lora_adapter endpoints.

How Prompt Tuning Works Step by Step

1. Pick a frozen base model. Open-weights options include Qwen3-8B and Qwen3-32B (Apache-2.0), Llama 3.1 8B and Llama 4 Scout.
2. Decide the soft-prompt length. Typical values are 20, 50, or 100 virtual tokens. Longer prompts add capacity but slow inference because they expand context.
3. Initialise the soft prompt. Two common strategies: random embeddings, or embed a hand-written natural-language prompt and use that as the starting point.
4. Train against your labelled dataset. Use cross-entropy loss on the target outputs. The Hugging Face PEFT guide uses AdamW at 3e-2; Lester et al. used Adafactor at a constant 0.3. Both are far higher than full fine-tuning rates because the soft prompt is small.
5. Evaluate on a held-out eval set. Compare to the baseline of the same frozen model with hand-written prompts.
6. Ship as a small checkpoint. A 50-token soft prompt for a 7B model is roughly 800KB. Store one per task, swap at request time.

Prompt Tuning Example with Hugging Face PEFT

The Hugging Face PEFT library wraps any causal language model with a trainable soft prompt. This example follows the official PEFT prompt-based methods guide, where a 560M-parameter BLOOMZ model learns to label tweets from the RAFT twitter_complaints set as complaint or no complaint.

# pip install -q peft transformers datasets
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PromptTuningConfig, PromptTuningInit, get_peft_model

base = "bigscience/bloomz-560m"
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base)

init_text = "Classify if the tweet is a complaint or no complaint.\n"
config = PromptTuningConfig(
    task_type="CAUSAL_LM",
    prompt_tuning_init=PromptTuningInit.TEXT,
    num_virtual_tokens=len(tokenizer(init_text)["input_ids"]),
    prompt_tuning_init_text=init_text,
    tokenizer_name_or_path=base,
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
# trainable params: 12,288 || all params: 559,226,880 || trainable%: 0.0022

optimizer = torch.optim.AdamW(model.parameters(), lr=3e-2)
# train over tokenized batches, then save the adapter
model.save_pretrained("tweet-complaints-soft-prompt")

Only the soft prompt trains: 12,288 parameters (12 virtual tokens, one per token of the init text, times 1,024 embedding dimensions), while about 559 million weights stay frozen. The saved adapter holds only the soft prompt, so it is a file of under 50 KB next to the base model.

Before you ship it, score the tuned model against the same base model with a hand-written prompt on one held-out set, using the same evaluators for both. The measurement section below runs that loop with Future AGI evals.

Prompt Tuning vs Prompt Engineering vs Fine-Tuning in Practice

Cost and Time

• Prompt engineering: hours of human iteration.
• Prompt tuning: minutes to hours of single-GPU training.
• LoRA fine-tuning: hours of single-GPU training.
• Full fine-tuning: hours to days of multi-GPU training.

Storage

• Prompt engineering: a text string in your prompt config.
• Prompt tuning: a few hundred KB to a few MB per task.
• LoRA: tens of MB per task.
• Full fine-tuning: a full model copy (tens to hundreds of GB).

Inference Overhead

• Prompt engineering: token cost only.
• Prompt tuning: token cost (soft prompt occupies context slots) plus negligible compute overhead.
• LoRA: small per-layer matrix multiply per forward pass; near-zero if merged.
• Full fine-tuning: zero overhead, but you must route to the specific model copy.

Interpretability

• Prompt engineering: fully readable.
• Prompt tuning: opaque embeddings; not interpretable.
• LoRA: opaque weight deltas; not interpretable.
• Full fine-tuning: opaque weight deltas; not interpretable.

Cross-Provider Portability

• Prompt engineering: works everywhere, though convention varies per provider.
• Prompt tuning / LoRA / fine-tuning: tied to a specific base model.

When to Reach for Prompt Tuning in a Multi-Tenant App

A common 2026 scenario: a self-hosted Qwen3-8B serving a multi-tenant SaaS where each customer has a slightly different output style and vocabulary. You have a few hundred labelled examples per customer. The base model stays the same for every tenant; only the adapter changes.

Options:

• Prompt engineering only. Cheap, fast to iterate, but cross-customer prompts get long and expensive.
• Few-shot per customer. Better than zero-shot, but a few hundred examples is too many to fit in context.
• LoRA per customer. Strong baseline. Each tenant’s LoRA is tens of MB; hot-swappable at request time in vLLM.
• Prompt tuning per customer. Strong baseline for the multi-tenant case. Each soft prompt is sub-MB; much cheaper to store and load than LoRA.
• Full fine-tuning per customer. Almost never the right answer here; storage and routing explode.

For multi-tenant adapter-heavy setups, prompt tuning and LoRA are both reasonable; prompt tuning wins on per-tenant storage cost. Train both on one sample tenant and score them on that tenant’s held-out examples with the same evaluators before you commit.

Limitations You Should Know Before You Start

• Closed APIs cannot be prompt-tuned. gpt-6-astra, claude-opus-5-5, and gemini-4-argon do not expose embedding-layer access. Hosted tuning, where it exists, runs server-side and never gives you a soft prompt.
• Smaller base models benefit less. Lester et al. showed that prompt tuning closes the gap with full fine-tuning at large model scale; for sub-1B models, full fine-tuning often wins.
• The soft prompt is not interpretable. Debugging regressions is harder than with hand-written prompts. Pair every prompt-tuned deployment with continuous evaluators on production traffic.
• Tasks far from the pretraining distribution. Prompt tuning only moves the model so far. For novel tasks, retrieval or full fine-tuning beats it.

How to Measure Whether Prompt Tuning Helped

Same playbook as any other adaptation method:

1. Build a golden dataset of 100 to 500 representative inputs with rubrics or expected outputs.
2. Run the frozen base model with your best hand-written prompt as the baseline.
3. Train the soft prompt against a separate training split.
4. Evaluate the tuned model on the same golden dataset with the same evaluators.
5. Compare average score, worst-decile score, and per-example diffs.
6. Promote the tuned variant only if it wins both on average and on the worst decile.

With Future AGI:

from fi.evals import evaluate

# Baseline: frozen Qwen3-8B with hand-written system prompt
baseline_score = evaluate(
    "context_adherence",
    output=baseline_response,
    context=retrieved_chunks,
    model="turing_flash",
)

# Tuned: frozen Qwen3-8B with learned soft prompt
tuned_score = evaluate(
    "context_adherence",
    output=tuned_response,
    context=retrieved_chunks,
    model="turing_flash",
)

print(baseline_score.score, tuned_score.score)

Per the cloud-evals docs, turing_flash returns in about 1 to 2 seconds, turing_small in 2 to 3 seconds, and turing_large in 3 to 5 seconds. Pass the same context to both calls so the soft prompt is the only variable.

For task-specific rubrics, write the rubric in plain English and run both variants through an LLM judge:

from fi.evals import evaluate

rubric = "Rate from 0 to 1 how closely the output matches the brand tone described in the style_guide field."

baseline_judge = evaluate(prompt=rubric, output=baseline_response, style_guide=style_guide, engine="llm", model="gpt-6-astra")
tuned_judge = evaluate(prompt=rubric, output=tuned_response, style_guide=style_guide, engine="llm", model="gpt-6-astra")

print(baseline_judge.score, tuned_judge.score)

Common Pitfalls Worth Watching

• Treating “prompt tuning” as a synonym for “writing a better prompt.” It is a specific technique with specific cost and capability trade-offs.
• Picking prompt tuning before exhausting retrieval and prompt engineering. Adaptation methods add complexity. The first lever is usually a better prompt or better retrieval.
• Skipping the eval gate. Soft prompts are opaque; without a measurable evaluator suite, you cannot tell whether the tuned model actually improved.
• Mixing training and eval data. Same trap as in any ML training. Hash and dedupe.

How Future AGI Fits into a Prompt Tuning Workflow

Future AGI is the measurement and observability layer around your prompt-tuning workflow:

• fi.evals: groundedness, context adherence, faithfulness, toxicity, summary quality, plus LLM-as-judge evals where you write the rubric in plain English. Use it to compare frozen-baseline versus tuned variants on the same dataset.
• Prompt optimisation: six algorithms, including GEPA, ProTeGi and PromptWizard, that search over hand-written prompts; the optimised prompt is the baseline a soft prompt has to beat.
• traceAI (Apache 2.0): OpenTelemetry-compatible spans that capture which model and prompt version produced each production response; tag the active soft-prompt adapter with using_metadata to filter traces by adapter.

• fi.simulate: multi-turn conversations with simulated customer personas, scored automatically, to test a tuned adapter before it ships.
• Agent Command Center at /platform/monitor/command-center: OpenAI-compatible gateway with 100+ providers, 15 routing strategies and inline guardrails, for routing traffic between model variants.

Set FI_API_KEY and FI_SECRET_KEY to authenticate the SDK. Cloud evals such as context_adherence on turing_flash run on Future AGI’s Turing models and need that pair. The LLM-judge calls also need your judge provider’s key, such as OPENAI_API_KEY for gpt-6-astra.

Choosing Between a Soft Prompt, LoRA and a Better Prompt

Prompt engineering, prompt tuning and fine-tuning solve different problems, and most of the confusion is about the name. Teams on closed frontier APIs get further with prompt engineering plus retrieval. Teams that self-host open-weights models and need cheap per-task adapters should train a soft prompt and a LoRA, then keep whichever beats the hand-written baseline on average and on the worst decile.

Run that comparison on your own golden dataset with Agent Learning Kit, installed with pip install agent-learning-kit.

Sources

• Hu et al. 2021: LoRA, Low-Rank Adaptation of Large Language Models
• vLLM LoRA multi-adapter docs
• Meta Llama 4 model card
• Qwen 3 on Hugging Face