Embeddings in LLMs Explained: A Complete 2026 Guide to Dense Vectors, Training, and Use

What Are Embeddings in LLMs?

A support bot gets the question “How long is the refund period?” and keyword search comes back empty, because the policy says “Returns must be initiated within thirty days of purchase.” Embedding search still finds the right chunk, because embeddings in LLMs capture meaning: the question and the policy line say the same thing, so their vectors sit close together.

An embedding is a numerical vector that captures the meaning of a token, sentence, or document, so that content with similar meaning sits close together in vector space. In LLM work the word covers two related things: the LLM’s own token embedding layer, which turns each token ID into a vector, and separate embedding models that turn passages into vectors for search and RAG.

Embeddings are the bridge between human language and the linear algebra a model can compute on. This guide covers what embeddings are, the embedding matrix inside an LLM, how embeddings are produced, dense versus sparse representations, training methods, dimensionality choices, and where they fit in RAG and agent stacks. For a ranked listicle of which embedding model to pick today, see our best embedding models guide.

TL;DR

What Is the Embedding Matrix in an LLM?

Before an LLM can work with text, a tokenizer splits it into tokens and maps each token to an integer ID. The model’s first layer is the input embedding matrix, a learned table with one row per vocabulary entry and one column per hidden dimension, so its shape is vocabulary size x hidden size.

Each token ID selects one row, and that row becomes the token’s starting vector before the attention layers mix in context from the rest of the sequence. The lookup itself ignores context, so the same token ID always pulls the same row; the layers after it do the contextual work.

Two real configs make the shape concrete. Qwen3-0.6B has a vocabulary of 151,936 tokens and a hidden size of 1,024, so its embedding matrix is 151,936 x 1,024 = 155,582,464 parameters (our arithmetic). Qwen3-8B keeps the same 151,936-token vocabulary with a hidden size of 4,096.

At the other end of the network, an output layer (named lm_head in the Hugging Face Qwen3 code) maps each final hidden state back to one score per vocabulary token. Some models tie the two and reuse the input embedding weights as that output layer: Qwen3-0.6B sets tie_word_embeddings to true, while Qwen3-8B sets it to false and keeps a separate output matrix.

From one-hot encoding to dense vectors

The simplest way to give a model a word is one-hot encoding: a vector with a 1 in one position and 0 everywhere else. With a vocabulary of 100,000 words, each word becomes a 100,000-dimensional sparse vector. Synonyms get completely unrelated vectors. The encoding cannot represent meaning.

Embeddings replace this with a dense vector of a few hundred or thousand dimensions where the geometry encodes meaning. The classical demo: vector(“king”) minus vector(“man”) plus vector(“woman”) lands near vector(“queen”) in some training regimes. Modern transformer embeddings also encode contextual meaning that shifts with the surrounding text.

How an embedding is produced

A modern embedding pipeline has three steps:

1. Tokenize the input into subword units (byte-pair encoding, SentencePiece, tiktoken).
2. Pass the tokens through a transformer encoder.
3. Pool the final hidden states (mean pool, CLS-token pool, or learned attention pool) into one vector of fixed dimensionality.

The same idea works for images (vision transformer plus pooling), audio (audio encoder plus pooling), and even structured data when the encoder is trained on it. Whatever the input, the result is one fixed-length vector you can compare with other vectors from the same model.

Are Embedding Models LLMs?

An embedding model runs the three steps above (tokenize, run the transformer, pool) and returns one vector instead of generated text, so it does a different job from a chat LLM. The token embeddings inside an LLM differ from both: they form an internal layer that gives each token its starting vector before attention runs, and they never stand for a whole passage.

The two families do overlap. Per its model card, the Qwen3 Embedding series builds upon the dense foundational models of the Qwen3 series and comes in 0.6B, 4B and 8B sizes. So an embedding model can start from an LLM foundation model and still serve a different purpose: turning text into vectors for search and RAG.

Embeddings also power evaluation itself. Future AGI’s Embedding Similarity eval scores how close a generated answer is in meaning to a reference answer, which helps when the right answer can be worded many ways, like the thirty-day refund policy.

Types of Embeddings in LLMs (Dense, Sparse, Semantic, Syntactic)

Dense embeddings

Most dimensions can contain continuous values that carry signal, unlike vocabulary-sparse lexical vectors. Dense vectors capture semantic meaning and are used for retrieval, classification, and similarity. Example families: OpenAI text-embedding-3, Google gemini-embedding-001 and gemini-embedding-2, Cohere Embed v4 and v5, Voyage 4, Qwen3-Embedding.

Sparse embeddings

Most dimensions are zero. Each non-zero dimension corresponds to a vocabulary term and a weight. BM25 is the classical example.

SPLADE (2021) is a neural first-stage ranker that learns highly sparse representations of queries and documents, with weights on terms, instead of a dense vector. Sparse representations can inherit the exact term matching of bag-of-words models.

Semantic vs syntactic

• Syntactic embeddings emphasize part of speech, word order, and grammatical role. Static word vectors (Word2Vec, GloVe, FastText) treat each word as one fixed vector.
• Semantic embeddings emphasize meaning. Contextual transformer embeddings shift with neighbors so “bank” near “river” and “bank” near “account” produce different vectors.

Hybrid retrieval

Hybrid search runs dense and sparse retrievers in parallel, then fuses the ranked lists with reciprocal rank fusion or a small reranker. The dense list ranks chunks by meaning and the sparse list by exact terms, so fusion decides which chunks from each list reach the prompt.

Training embeddings

The training recipe below has two stages. Self-supervised pretraining builds a general encoder, and contrastive fine-tuning then shapes it for retrieval and similarity.

Self-supervised pretraining

A base encoder (BERT, T5, Llama-like) learns general language structure with masked language modeling or contrastive language objectives over web-scale text. This gives the model a useful but generic representation.

Contrastive fine-tuning

The encoder is then trained to pull related pairs together and push unrelated pairs apart. Loss functions:

• InfoNCE / NT-Xent: contrastive losses over batches of positive and negative pairs.
• Multiple-negatives ranking loss: every other item in the batch is treated as a negative.
• Triplet loss: anchor, positive, and explicit negative.

Hard negative mining finds negatives that are semantically close but wrong, which teaches the model to discriminate where it matters. For the refund question, a hard negative is a chunk about shipping times: same topic area, wrong answer.

Domain adaptation

For specialized domains (medical, legal, code, your internal product catalog), fine-tune on labeled or weakly labeled in-domain pairs. The contrastive losses and hard-negative mining above still apply, with the pairs drawn from your own queries and documents.

What Is Embedding Dimension in an LLM?

Dimension means two different things here. Inside an LLM such as Qwen3, the token embedding size equals the hidden size: 1,024 for Qwen3-0.6B and 4,096 for Qwen3-8B. An embedding model’s output dimension is the length of the vector it returns for a passage, and some models let you set it: Voyage 4 defaults to 1,024 and Cohere Embed 5 defaults to 2,048.

Output dimensions vary by model; the models named in this guide span 32 to 4,096. Tradeoffs:

Matryoshka representation learning (MRL) trains a single model so its full-length vector still works when truncated to a shorter length. You ship one model and let downstream consumers pick their dimension based on speed and storage budget. Several current models let you pick the output size: Qwen3-Embedding supports user-defined sizes from 32 to 4,096 (its model card lists MRL support), Cohere Embed 5 offers 256 to 2,048, and Voyage 4 offers 256 to 2,048.

Where Are Embeddings Stored for Retrieval?

Vector databases index embeddings and answer nearest-neighbor queries. Options include Pinecone, Weaviate, Qdrant, Milvus and pgvector on Postgres. They can use approximate nearest-neighbor (ANN) indexes for lower-latency search.

You can also skip a dedicated vector database. pgvector adds vector similarity search, both exact and approximate, to Postgres, so you can store vectors with the rest of your data. Whichever store you use, embed the stored documents and the incoming queries with the same embedding model snapshot, as the first pitfall below explains.

For more on this side of the stack, see our best vector databases for RAG guide.

Where embeddings show up in modern AI

Retrieval-augmented generation

The basic pattern:

1. Chunk the source corpus.
2. Embed each chunk.
3. Store embeddings in a vector index.
4. At query time, embed the user question.
5. Retrieve the top K most similar chunks.
6. Inject them into the LLM prompt as context.

Quality bottlenecks: chunk size, embedding model match to domain, hybrid retrieval, reranker presence, prompt template.

Classification, clustering, and deduplication

Embed every record once, then cluster or classify in vector space without retraining a custom model. Useful for deduplication, topic discovery, intent classification, and tag suggestion.

Recommendation and personalization

User and item embeddings sit at the core of two-tower recommender systems. The user vector plus the item vector predict relevance with a simple dot product.

Agent memory and tool routing

Agents embed past conversation turns to fetch relevant context, and they embed tool descriptions to pick the right tool from a large catalog without exhausting the context window with full schemas.

Multimodal search

Multimodal embedding models go beyond text: Google gemini-embedding-2 maps text, images, video, audio and documents into one unified embedding space, and Cohere embed-v4.0 and Embed 5 accept text, images and mixed PDFs. You search images with text queries, find similar audio clips by description, or compare a screenshot to a body of documents.

Common Pitfalls With Embedding Models

• Mismatched embedding model between indexing time and query time. Embed both with the same model snapshot or your similarity scores are nonsense.
• Skipping normalization when your distance metric requires it. Match the normalization expected by your embedding model and index configuration, or you can get silently wrong results.
• Pooling mismatch. Some models expect mean pooling, others CLS pooling. Read the model card.
• Chunk size too long or too short. Long chunks lose precision; short chunks lose context. Tune for your domain.
• Stale corpora. Stored embeddings go out of date as your source documents change. Schedule re-embeds for high-churn content.
• Single-language fine-tuning on a multilingual model can degrade the other languages. Use multilingual training data if you need multilingual support.

To evaluate an embedding model, measure retrieval on your own query set instead of trusting a public leaderboard. Build a small labeled set of real queries with the chunks that should come back.

Run each candidate model and compare Recall@K, Precision@K, NDCG@K and MRR at the K your pipeline retrieves. Rerun the same set whenever you change the model, the dimension or the chunking. Here is how to evaluate an embedding model on your own queries.

How to Evaluate and Trace Embedding Retrieval With Future AGI

Whichever embedding model you pick, Future AGI is the eval and observability layer that tells you whether your retrieve-then-generate pipeline is working. Evals score each answer against the chunks it was given, and traces show which chunks came back for which query.

• Future AGI has custom evals: write your own grading rule, pick an LLM judge or a deterministic check, set a pass threshold and run it as a CI gate. For retrieval, the built-in Groundedness, Context Relevance, Chunk Utilization, Precision@K, Recall@K and Embedding Similarity checks score retrieval and grounding. Run them on a regression set every time you change the embedding model or chunking strategy. The eval SDK, Agent Learning Kit, is Apache-2.0.
• traceAI (Apache-2.0, built on OpenTelemetry) records retrieval and generation calls as spans in one trace. Observe lets you open any failing query as a trace, see which retrieval span returned which chunks, and score that trace with evals.

Here is a groundedness check on the refund example:

from fi.evals import evaluate

# Score whether the LLM answer is grounded in the retrieved context.
result = evaluate(
    "groundedness",
    input="How long is the refund period?",
    output="Our return window is 30 days.",
    context="Returns must be initiated within thirty days of purchase.",
    model="turing_flash",
)

print(result.score, result.reason)

turing_flash is the fast default. Switch to turing_small for higher fidelity at lower cost, or turing_large for maximum accuracy and audio inputs. Configure with the FI_API_KEY and FI_SECRET_KEY environment variables. For the wider metric set, see RAG evaluation metrics like faithfulness and context relevance.

Future directions

• Stronger long-context embeddings that represent whole documents without chunking.
• Better multimodal joint spaces for text, image, audio, and video.
• Quantized and distilled embeddings that run on-device for privacy-sensitive workloads.
• Task-aware embeddings that produce different vectors depending on the downstream task (retrieval vs classification vs ranking).
• Continued vector-search upgrades in pgvector and OpenSearch.

Catch Embedding Drift Before Your Users Do

Go back to the support bot and its refund question. It keeps finding the thirty-day policy only while three things hold: the same embedding model snapshot indexes the documents and embeds the queries, retrieval is scored every time you change the model or the chunking, and failing queries are traced so you can see which chunks came back.

Score your retrieval on every embedding change with Future AGI evals.