How Multimodal LLMs Work in 2026: Vision Encoders, Fusion, and Cross-Attention Explained

TL;DR

A multimodal large language model takes more than one kind of input, such as text plus images, audio or video, and generates text from it. Open-weight designs usually join a vision encoder to a text decoder through a connector; native models such as Llama 4 fuse text and vision tokens in one backbone. This guide covers each layer, the training recipes that protect text-only skill, and production evaluation.

What Multimodality Means in 2026

A “modality” is a kind of input or output: text, image, audio, video, code. A multimodal LLM accepts at least two of these as input. In practice that usually means text plus image (vision-LLMs). Some frontier models go wider: Google lists text, image, video, audio and PDF inputs for Gemini 4 Argon, while OpenAI serves speech through separate audio models such as GPT-Live 1.

The core architectural pattern is the same across all of them:

1. Encode each non-text modality with a domain-specific encoder.
2. Project the encoder’s features into the language model’s token space.
3. Decode with a text-only LLM that sees both the projected tokens and the user’s text prompt.

The main design choices live in step 2 and in how the three components are trained together. Native models such as Llama 4 instead train text and vision tokens together in one backbone from pretraining (Option D). Steps 1 to 3 below take each component in turn.

Multimodal LLM vs LLM vs VLM

A text-only LLM reads and writes text tokens. Most multimodal LLMs keep that decoder and add an encoder plus a connector, so images, audio or video arrive as tokens the decoder can attend to. A vision-language model (VLM) is the image-and-text subset: LLaVA, NVLM 1.0 and BLIP-2 are VLMs, while Gemini 4 Argon also takes video, audio and PDF input.

Step 1: Vision Encoders

The vision encoder is a Vision Transformer or a Vision Transformer variant. It takes an image, splits it into patches (typically 14x14 or 16x16 pixels), and produces a sequence of patch embeddings. Common encoders in the models covered here are:

• CLIP ViT-L/14 (Radford et al. 2021). 24 transformer layers, 224 or 336 px inputs. Still common in LLaVA-family models.
• SigLIP (Zhai et al. 2023). Same idea as CLIP but with a sigmoid loss; better data efficiency.
• InternViT-6B-448px (Chen et al. 2023). 6B parameter encoder used by NVLM and InternVL.
• Pixtral-ViT (Mistral). 400M parameter encoder trained from scratch for Pixtral 12B to read images at native resolution. Pixtral Large uses a 1B parameter vision encoder.

The output of the encoder is a sequence of vectors, one per patch. A 224x224 image with 14x14 patches produces 256 patch tokens. Higher-resolution inputs multiply that count, which is why NVLM splits large images into tiles and downsamples each tile’s 1,024 tokens to 256.

Step 2: Fusion / Connector Strategies

The connector turns visual patch embeddings into something the LLM can read. Three connector options cover the adapter-based models here, and native models such as Llama 4 take a fourth path (Option D). Option A adds projected visual tokens to the context, Option B compresses patches into 32 query tokens, and Option C adds cross-attention layers instead.

Option A: Projection-based connector (LLaVA, NVLM, Pixtral)

A small feed-forward network maps each patch embedding to an LLM token. The most common choice is a two-layer MLP with a GeLU non-linearity; LLaVA 1.5, NVLM-D and Pixtral 12B all use one. Pixtral also inserts break tokens between image rows. The projection is small, fast, and easy to train.

• Token count tracks patch count: 256 patches means 256 extra tokens in LLaVA, while NVLM downsamples each tile’s 1,024 tokens to 256 first.
• Used by LLaVA 1.5+, NVLM 1.0 and Pixtral.

Option B: Q-Former (BLIP-2, InstructBLIP)

A learnable transformer compresses N patch embeddings into K query tokens (K is much smaller than N), and BLIP-2 sets K=32. The Q-Former is itself a 12-layer transformer trained in two phases. The design comes from BLIP-2 (Li et al. 2023).

• Saves context tokens: BLIP-2 passes 32 query tokens to the LLM instead of the full patch sequence.
• Adds parameters and a separate training phase.
• Used by BLIP-2 and InstructBLIP.

Option C: Cross-attention adapter layers (Flamingo, OpenFlamingo)

Insert new cross-attention layers between the existing self-attention layers of a frozen LLM. Each cross-attention layer takes language tokens as queries and visual tokens as keys and values. The design comes from DeepMind’s Flamingo (Alayrac et al. 2022), which also trains a Perceiver Resampler that shrinks the vision encoder output to a fixed size.

• Keeps the LLM frozen entirely, so text-only performance is preserved by construction.
• Adds parameters per inserted layer.
• Used by DeepMind Flamingo and the open-source OpenFlamingo.

Option D: Native early fusion (Llama 4)

A natively multimodal model trains on text and images together in one backbone from the start. Meta’s model card describes Llama 4 as mixture-of-experts models that incorporate early fusion for native multimodality, with Scout pretrained on about 40 trillion tokens of multimodal data.

• Used by Llama 4 Scout (17B active, 109B total parameters) and Llama 4 Maverick (17B active, 400B total).

Step 3: The Language Decoder

The decoder is a standard autoregressive text LLM. Common open-weight decoder families include:

• Llama 3 / Llama 4 family
• Qwen2 / Qwen3 family
• Mistral / Mixtral family
• Vicuna (LLaMA-derived instruction-tuned)

The decoder reads the concatenated sequence of projected visual tokens and the user’s text tokens, then generates a response one token at a time. For Flamingo-style cross-attention, the LLM is frozen and unmodified; the cross-attention layers do the integration.

Reference Architectures

LLaVA and LLaVA-NeXT

LLaVA is the canonical open-source vision-language model.

• Vision encoder: CLIP ViT-L/14 (336 px in LLaVA-NeXT, with AnyRes for higher resolutions)
• Connector: Two-layer MLP projector
• Decoder: Vicuna 7B or 13B (originally), with newer LLaVA-NeXT variants on Llama 3 backbones
• Training: Two-stage. Stage 1 trains the projector on 558K image-caption pairs. Stage 2 fine-tunes on 760K visual instruction samples, with both the projector and the LLM unfrozen.

LLaVA’s key contribution was showing that high-quality visual instruction tuning data plus a tiny connector can beat Gemini Pro on several benchmarks, per the LLaVA-NeXT release post (LLaVA-NeXT-34B, under 1M visual instruction samples). A May 2024 release scaled it to Qwen-1.5 72B and 110B backbones.

NVIDIA NVLM 1.0

NVLM 1.0 is NVIDIA’s open-source vision-language model.

• Vision encoder: InternViT-6B-448px-V1-5
• Connector: MLP projector with a tile-tagging mechanism that splits high-resolution images into tiles and tags each tile with positional text
• Decoder: Qwen2-72B-Instruct
• Training: Two-stage. Pretrain the projector on diverse VQA, OCR, and reasoning data. SFT unfreezes the LLM while keeping the vision encoder frozen, blending multimodal and high-quality text-only data.

NVLM reports an average 4.3-point gain on text-only math and coding benchmarks after multimodal training (Dai et al. 2024), addressing the catastrophic-forgetting problem. The decoder-only NVLM-D described here is one of three NVLM variants, next to cross-attention NVLM-X and hybrid NVLM-H.

Mistral Pixtral Large

Pixtral Large is Mistral’s 124B parameter open-weight vision-language model.

• Vision encoder: 1B parameter vision encoder
• Connector: In Pixtral 12B, a two-layer GeLU network plus [IMAGE BREAK] tokens between image rows
• Decoder: 123B multimodal decoder built on Mistral Large 2
• Key tricks (Pixtral 12B report): Block-diagonal attention masks and 2D Rotary Position Embedding (RoPE-2D) handle variable image sizes; context length up to 128k tokens.

At its November 2024 launch Pixtral Large led GPT-4o and Gemini-1.5 Pro on ChartQA and DocVQA, per Mistral. Mistral retired the Pixtral Large API on May 31, 2026, so read it here as an architecture reference; the open weights are still on Hugging Face. Its model card lists 88.1 on ChartQA with chain-of-thought and 93.3 on DocVQA.

BLIP-2 / InstructBLIP

BLIP-2 is the reference for Q-Former-based architectures.

• Vision encoder: Frozen pretrained ViT-L/14 (or EVA-ViT)
• Connector: 12-layer Q-Former that compresses patches into 32 query tokens
• Decoder: Frozen Flan-T5 or OPT, depending on the BLIP-2 variant
• Training: Two-phase. Phase 1 trains the Q-Former on image-text representation tasks against the frozen vision encoder. Phase 2 connects the Q-Former to the frozen LLM and trains for vision-to-language generation.

Both the encoder and the decoder are frozen throughout. Only the Q-Former and one fully connected projection layer are trained. The BLIP-2 paper reports that it beat Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters.

OpenFlamingo

OpenFlamingo is the open-source replication of DeepMind Flamingo.

• Vision encoder: CLIP ViT-L/14
• Connector: Cross-attention adapter layers inserted between self-attention layers in a frozen LLM
• Decoder: MPT or RedPajama, kept frozen
• Training: Fine-tune the cross-attention layers on interleaved image-text web data (Multimodal C4, LAION-2B).

OpenFlamingo’s freezing strategy preserves the underlying LLM’s text-only capabilities by construction, at the cost of a more complex parameter layout. OpenFlamingo-9B, the MPT-7B variant, scores 52.7 on zero-shot VQAv2 in the paper’s results, slightly above DeepMind’s Flamingo-9B at 51.8.

Benchmark Snapshot (Scores Reported at Release)

Scores come from each model’s paper, card or release post under different setups: Pixtral Large scores MathVista, MMMU and ChartQA with chain-of-thought and names no MMMU split, and BLIP-2 and OpenFlamingo are zero-shot. LLaVA-NeXT-34B’s OCRBench score is the figure NVLM’s paper reports for it. A dash means the source does not report that benchmark, so treat comparisons as directional.

How Multimodal Training Avoids Catastrophic Forgetting

The biggest failure mode of naive multimodal fine-tuning is that the LLM loses text-only ability. Two techniques mitigate this in 2026:

1. Selective unfreezing: freeze the vision encoder during SFT; only unfreeze the connector and the LLM. NVLM and LLaVA both follow this.
2. Mixed data: blend high-quality text-only fine-tuning data into the multimodal SFT mix. NVLM reports a 4.3-point gain on text-only math and coding after multimodal training using this recipe.

A third option, used by Flamingo and OpenFlamingo, is to never unfreeze the LLM at all and only train the cross-attention adapters. That preserves text-only performance by construction, though NVLM’s frozen-LLM ablation lost some accuracy on OCR-heavy vision tasks. Mistral says Pixtral Large extends Mistral Large 2 without compromising text performance.

Evaluating Multimodal LLMs in Production

Multimodal models are harder to evaluate than text-only models because:

• The input is high-dimensional and not easily diffable
• Hallucinations can be visual (the model invents content not in the image) or textual (the model misreads chart values)
• Standard heuristic metrics (BLEU, ROUGE) miss most failure modes

A practical 2026 evaluation stack:

1. Public benchmarks as a starting baseline: MMMU, MathVista, DocVQA, ChartQA, OCRBench, VQAv2.
2. Task-specific deterministic checks on production data: JSON-schema validation, output length caps, citation regex.
3. LLM-judge metrics for the open-ended failures: faithfulness to the image, instruction adherence, tone, safety.
4. Tracing so every score is tied to a replayable run.

Future AGI fits at steps 3 and 4 as the eval and observability layer. The evaluate() function in agent-learning-kit (pip install agent-learning-kit) runs evals that read the media itself, such as Caption Hallucination and Image Instruction Adherence for images, and OCR Evaluation for PDFs.

traceAI records the image URLs sent to the model as span attributes next to the prompt and output. The multimodal eval cookbook walks through image, audio and PDF evals end to end. Each scored span then shows its result in the Observe dashboard.

from fi.evals import evaluate

result = evaluate(
    "caption_hallucination",
    image="https://raw.githubusercontent.com/future-agi/cookbooks/main/ecom_agent/observe/generated_products/nike_air_max_sneakers.png",
    caption="A pair of white sneakers with a wavy sole design.",
    model="turing_flash",
)
print(result.score, result.reason)

For custom rubrics, such as whether the model read a chart legend correctly, use a custom LLM judge that receives the chart image itself:

from fi.evals.metrics import CustomLLMJudge
from fi.evals.llm import LiteLLMProvider
from fi.opt.base import Evaluator

judge = CustomLLMJudge(
    provider=LiteLLMProvider(),
    config={
        "name": "chart_legend_read",
        "model": "openai/gpt-6-astra",
        "grading_criteria": "Score 1 if every legend label is named correctly, else 0.",
    },
)
evaluator = Evaluator(metric=judge)

def score_legend(chart_url: str, answer: str) -> float:
    [result] = evaluator.evaluate([{"image_url": chart_url, "output": answer}])
    return result.score

Future AGI’s SDK evals docs list hosted Turing evals at about 1 to 3 seconds and custom LLM-as-Judge calls at about 2 to 5 seconds. Every Turing model reads images and Turing Large adds audio, per the evaluator models page. Set FI_API_KEY and FI_SECRET_KEY, then attach the same evals to production traces in Observe.

Open-Source Multimodal LLMs and Why They Matter

Open-weight multimodal models matter because:

• They let researchers and engineers reproduce results without relying on a closed API.
• They allow domain-specific fine-tuning (medical imaging, satellite imagery, manufacturing defect detection) without sending images to a third-party.
• They run on hardware you control, including air-gapped and compliance-bound environments, with no calls to an outside API.

The trade-off is engineering overhead: serving a 72B or 124B vision-language model requires multi-GPU inference and careful batching. The frontier API products (GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon) are easier to start with; open-weight models become attractive once you have a steady production workload to amortize the serving cost.

From an Image to an Answer You Can Trust

Follow one image and one question through the stack: the encoder turns the image into patch embeddings, the connector or a native backbone turns those into tokens, and the decoder writes the answer one token at a time. Whether that answer matches the image is the job of an eval that reads the image and a trace that keeps the run replayable.

Shipping a vision feature? Run your first multimodal eval on Future AGI and score answers against the images your users send.