Guides

Evaluating Modal LLM Inference Apps in 2026: Cold/Warm/Concurrent Latency Parity, Tail Quality Under Burst, and Shutdown-Determinism

Evaluate Modal LLM inference: score cold, warm and concurrent latency separately, test answer quality under burst, and verify scale-down and shutdown timing.

· Updated
· 14 min read
modal serverless-gpu llm-evaluation cold-start ai-gateway llm-observability 2026
Cover image for evaluating Modal LLM inference across cold, warm and concurrent traffic
Table of Contents

Modal LLM inference means serving an open-weight model, usually through vLLM’s OpenAI-compatible server, on Modal’s serverless GPUs with per-second billing and scale-to-zero. That elasticity is why a fixed-deployment eval misses its failures: one Llama 3.3 70B has one latency distribution on an H100 box and at least three on Modal. Evaluating it means scoring cold, warm and concurrent traffic separately, quality under burst, and shutdown behavior.

Cold start, warm start, and concurrency limits create eval blind spots. Per-invocation-type latency parity, p1 tail quality under burst, and shutdown-determinism are the three checks Modal-specific eval has to cover. Skip them and you ship a deployment that’s fast when warm and slow or wrong under load. Get them right and you have a serverless inference stack that holds its SLO across the regimes real traffic actually hits.

The shape draws on the agent-learning-kit SDK, the traceAI OpenAI instrumentor, and the Agent Command Center gateway. The latency and tracing samples below call an OpenAI-compatible Modal URL, so you can swap in your own deployment.

Why Modal LLM inference needs its own eval

Modal LLM inference needs its own eval because scale-to-zero, autoscaling and input concurrency give one model three latency regimes and a quality distribution that shifts under load. A fixed self-hosted deployment has predictable latency, predictable throughput, and a predictable hourly bill. A managed API like OpenAI or Anthropic has a vendor SLA and a per-token price. Modal sits between them and inherits the awkward parts of both.

Cold starts are user-facing latency, not vendor latency. Modal scales containers down to zero between bursts to save GPU-seconds. The first request after an idle period has to pull the container image, attach a GPU, load model weights into VRAM, and warm any compile caches before it can decode a single token.

Modal’s vLLM example puts vLLM’s compile step alone at tens of seconds to a few minutes, or about ten seconds from a cache. Either way, the cold tail is not something you can disclaim; it’s an attribute of your application that a user feels on the first request of every session.

Warm latency drifts with the scaler, not the GPU. When traffic ramps from one request per second to a hundred, Modal boots more containers, which Modal’s cold-start guide says takes seconds to minutes. If the scaler is conservative, requests queue behind the existing replicas and the warm p99 you reported in benchmarks gets buried under queue time.

Concurrency changes the answer as well as the cost. Modal bills GPU-seconds. A Modal Function container serves one request at a time unless it opts into Modal’s input concurrency, while a Modal Server, like the vLLM example, takes concurrent requests.

Running inputs concurrently can lower costs, Modal says. It also shifts the content-quality distribution because KV-cache pressure rises and the scheduler interleaves token generation. Free-form chat often holds. Long-context retrieval, strict-JSON outputs, and three-step tool composition often don’t.

These three properties compose into three eval checks. The next sections walk each one with the eval primitive that runs it. For the broader observability picture across self-hosted runtimes, see the self-hosted LLM observability guide.

TL;DR

CheckWhat it measuresMisses if you skip it
Per-invocation-type latency parityp50/p95/p99 first-token and total, per cold/warm/concurrent regimeCold tail buried in warm aggregate; burst queueing hidden in p50
Tail quality under burstContext adherence and tool-call correctness at p1 under realistic concurrencyQuality drift when several large-context calls share a replica
Shutdown-determinismWhether min_containers, buffer_containers, and scaledown_window behave as configuredCold starts inside supposedly warm windows; SLO becomes a lottery
Content quality baselineGroundedness, ContextAdherence, TaskCompletion, LLMFunctionCalling against golden setModel-level regressions across releases
Per-concurrency costBilled GPU-seconds per token, per concurrency tierCost per token that ignores GPU sharing

Ship only when the three Modal-specific checks pass on the exact GPU class, the exact concurrency setting, and the exact traffic shape you’re routing to. Anything else is the warm benchmark and the production behavior pretending to be the same artifact.

The Modal vLLM setup these checks assume

Modal’s vLLM example installs vLLM in an image, caches weights in Modal Volumes, and launches vllm serve from the @modal.enter() method of an @app.server class. modal deploy returns an OpenAI-compatible URL; with no container running, requests get a 503 and trigger a cold start.

Three settings in the example map onto the checks below: scaledown_window (set to 15 minutes there) decides when a container goes cold, target_concurrency (100, “tune carefully”) decides how many requests share a GPU, and FAST_BOOT trades generation speed for faster boots. Record all three with each eval run.

Check 1: per-invocation-type latency parity

Aggregate p99 is the wrong number for a serverless GPU runtime. Tag attempt spans with the intended modal.invocation_type=cold|warm|concurrent. Record the final observed modal.saw_503 on an invocation summary span, and compute percentiles from its measured timing attributes or the returned records per regime and availability path. Three regimes, three distributions, three SLOs.

Cold. Build a cold-start scenario in your load generator: idle long enough to force scale-to-zero (Modal’s default scaledown_window is 60 seconds), then fire one request and record end-to-end latency. Repeat across times of day and across container-image sizes. The variance matters as much as the mean.

Warm. Drive the function with steady traffic at the concurrency you expect in production. Record p50, p95, and p99 separately for time-to-first-token and total time. The traceAI OpenAI instrumentor captures token counts and timing automatically once you register the project, so the warm distribution rides on the trace tree with no extra wiring.

Concurrent. Ramp to the burst shape your traffic actually hits. Two burst shapes are worth testing, and you should score both: smooth ramp (1 RPS to 50 RPS over a minute) and step burst (sustained 5 RPS punctuated by 30 RPS spikes for 10 seconds). The step burst is where the scaler’s reaction time shows up as queue time and the warm replicas pack concurrent requests until the new replicas arrive.

import asyncio, time
import openai
from openai import AsyncOpenAI
from opentelemetry import trace
from fi_instrumentation import register, using_attributes
from fi_instrumentation.fi_types import ProjectType
from traceai_openai import OpenAIInstrumentor

trace_provider = register(
    project_type=ProjectType.OBSERVE,
    project_name="modal-llama-3-70b",
)
OpenAIInstrumentor().instrument(tracer_provider=trace_provider)
tracer = trace.get_tracer("modal-probe", tracer_provider=trace_provider)

client = AsyncOpenAI(
    base_url="https://<workspace>--llama70b.modal.run/v1",
    api_key="modal-token",
)

async def one_call(prompt: str, invocation_type: str, deadline_s: float = 120):
    # Python 3.11+: the deadline covers every attempt, wait and stream read.
    t0 = time.perf_counter()
    saw_503, attempt_ttft_ms, end_to_end_ttft_ms = False, None, None
    completion, finish_reason, status, error = "", None, "failed", None
    try:
        async with asyncio.timeout(deadline_s):
            while True:
                attempt_start = time.perf_counter()
                try:
                    with using_attributes(
                        metadata={"runtime": "modal", "modal.invocation_type": invocation_type,
                                  "modal.saw_503_before_attempt": saw_503},
                        tags=[f"modal-{invocation_type}"],
                    ):
                        stream = await client.with_options(max_retries=0).chat.completions.create(
                            model="meta-llama/Llama-3.3-70B-Instruct",
                            messages=[{"role": "user", "content": prompt}],
                            max_tokens=512,
                            stream=True,
                        )
                        async with stream:
                            async for chunk in stream:
                                if not chunk.choices:
                                    continue
                                choice = chunk.choices[0]
                                content = choice.delta.content
                                if content:
                                    if attempt_ttft_ms is None:
                                        now = time.perf_counter()
                                        attempt_ttft_ms = (now - attempt_start) * 1000
                                        end_to_end_ttft_ms = (now - t0) * 1000
                                    completion += content
                                if choice.finish_reason is not None:
                                    finish_reason = choice.finish_reason
                    if attempt_ttft_ms is None or finish_reason is None:
                        raise RuntimeError("Stream ended without content or a finish reason")
                    if finish_reason != "stop":
                        status, error = "incomplete", f"Incomplete answer: finish_reason={finish_reason}"
                    else:
                        status = "ok"
                    break
                except openai.InternalServerError as exc:
                    if exc.status_code != 503:
                        raise
                    saw_503 = True
                    await asyncio.sleep(1)
    except TimeoutError:
        status, error = "timeout", "Retry/stream deadline exceeded"
    except Exception as exc:
        error = f"{type(exc).__name__}: {exc}"
    record = {
        "invocation_type": invocation_type,  # intended load-test regime
        "saw_503": saw_503,  # observed availability path, not proof of a cold start
        "status": status, "error": error,
        "attempt_ttft_ms": attempt_ttft_ms,
        "end_to_end_ttft_ms": end_to_end_ttft_ms,
        "total_ms": (time.perf_counter() - t0) * 1000,
        "completion": completion, "finish_reason": finish_reason,
    }
    # Summary span is emitted after the probe, not a timer for the invocation.
    with tracer.start_as_current_span("modal.invocation.summary") as summary:
        summary.set_attribute("runtime", "modal")
        for key in ("invocation_type", "saw_503", "status", "error",
                    "attempt_ttft_ms", "end_to_end_ttft_ms", "total_ms", "finish_reason"):
            if record[key] is not None:
                summary.set_attribute(f"modal.{key}", record[key])
    return record

Modal does not send a cold-start header. A 503 is an observed unavailable path, consistent with having no ready container, not proof of a cold start. Keep the intended regime in invocation_type and record saw_503 separately. attempt_ttft_ms measures the served attempt to its first content-bearing chunk, a client-side token-arrival proxy; end_to_end_ttft_ms includes retries and waits. total_ms includes the entire completion, not startup time. Only finish_reason="stop" with nonempty content is ok; other terminal reasons, including length and content_filter, retain partial output with status="incomplete" and an explicit error. These are not evidence of container shutdown. The deadline produces an explicit timeout record rather than losing never-ready requests. Attempt metadata records modal.saw_503_before_attempt, not the outcome of that attempt. The post-probe modal.invocation.summary span records the final observed path, status and measured timing fields, including terminal failure; its own span duration is only summary emission time, not invocation latency. Split results by both regime and saw_503, and report failures alongside successful latency distributions. Aggregate p99 across all three is the number you stop reporting.

A defensible SLO looks like: a cold p95, a warm p99 TTFT and a concurrent p99 TTFT at the concurrency you ship. The exact budgets depend on your model and GPU class. The shape (three numbers, not one) does not.

Cutting Modal cold start time with memory snapshots

enable_memory_snapshot=True reuses a container’s memory state on later boots, and GPU memory snapshots, an alpha feature, add experimental_options={"enable_gpu_snapshot": True}. Modal reported median cold starts for Ministral 3 3B on vLLM falling from about 118 seconds to about 12 seconds with GPU snapshots.

Re-run the cold scenario after enabling them. Modal says that GPU snapshots are generally incompatible with multi-GPU code, so a 70B model split across GPUs may not qualify.

Check 2: tail quality under burst

Latency is half the eval. Content quality under burst is the other half, and a golden set run at concurrency=1 never measures it.

When requests share a GPU, vLLM’s continuous batching runs them through shared forward passes, and Thinking Machines showed that output depends on batch size. That batching is what makes serverless economical.

Batching also shifts the answer-quality distribution: KV-cache pressure rises with concurrent token counts, prefix-cache reuse drops when prompts diverge, and the scheduler interleaves token generation across requests. Strict-JSON outputs, tool arguments and long-context grounding are the outputs to score first. Run your golden set under realistic concurrent load and score the p1 of the quality metric, not the p50.

import statistics
from fi.evals import evaluate

# Score the same golden set at three concurrency tiers
for tier in [1, 10, 50]:
    cases = load_golden_set_at_concurrency("modal_rag_v2.jsonl", concurrency=tier)
    scores = [evaluate("context_adherence", context=c["context"], output=c["output"], model="turing_flash").score for c in cases]
    print(tier, statistics.quantiles(scores, n=100)[0])  # worst 1%

The shape that matters is the p1 delta across tiers. A mean that holds across tiers is fine; a worst-1% score that collapses at concurrency=50 is a tail-quality regression you cannot ship. The same shape applies to LLMFunctionCalling for tool-using agents: argument-shape drift under load is silent at p50 and visible at p1.

For long-context-specific failures, add ChunkAttribution and ChunkUtilization from the standard template set. They catch the retrieval-pattern degradations where the model starts ignoring later chunks under shared-replica pressure.

Check 3: shutdown-determinism

The third axis is the one teams underweight. Modal scales containers down to zero by default. min_containers (formerly keep_warm), buffer_containers and scaledown_window change when shutdown fires, and the autoscaler can end a container early.

GPU Functions also cannot opt out of preemption. The deployment ships, looks fine in staging, and starts cold-starting users in production because the warm window doesn’t behave the way the config implies.

Shutdown-determinism is whether the model stays loaded in VRAM between requests in the way you configured. Verify it on every release, not at deploy time.

The test is a strict timing sweep. Fire requests at intervals that straddle the configured scaledown_window: timeout minus 10s, timeout exactly, timeout plus 10s, timeout plus 60s. Record modal.saw_503 from Check 1 and its status for each request. A 503 or timeout inside a supposedly warm window flags availability drift to investigate; correlate it with server lifecycle logs before calling it a cold start. This client probe does not measure container startup duration.

The fix is usually one of three: raise scaledown_window so idle containers live longer, set min_containers=1 so a replica is always hot (cost-up, latency-flat), or set buffer_containers so bursts find a warm container. Preemption can still restart a GPU container, so keep the sweep in every release.

Score the sweep with a plain code check:

def shutdown_deterministic(sweep, scaledown_window_s):
    # sweep rows: idle_s plus saw_503 and status from one_call
    bad = [r for r in sweep if r["idle_s"] < scaledown_window_s
           and (r["saw_503"] or r["status"] != "ok")]
    return not bad, bad

Run it weekly. Shutdown behavior drifts silently with autoscaler and SDK changes you don’t control. A check that passes today can fail next month after a Modal release rolls out a different default. The check catches the drift before the on-call rotation does.

Shutdown order with exit_grace_period and @modal.exit

On Modal Servers, a container that is scaling down stops taking new requests but keeps serving in-flight ones for up to exit_grace_period seconds (maximum 3600). Modal then sends SIGTERM and runs your @modal.exit() handlers, which get 30 more seconds before SIGKILL. exit_grace_period defaults to 0, so a long generation can be cut off at scale-down.

Modal container shutdown order: a container serving a long generation is scaled down, finishes in-flight requests within exit_grace_period, then gets SIGTERM with 30 seconds for exit handlers, then SIGKILL.

Add that case to the sweep: start a long generation, let traffic drop so the container scales down mid-stream, and check the response completes. A truncated answer fails shutdown determinism even when latency looks fine.

Standard content quality baseline

Before any of the three Modal-specific checks earn their place, run the standard rubric set against your golden set. Groundedness, ContextAdherence, ContextRelevance, Completeness for RAG. TaskCompletion and LLMFunctionCalling for agents. AnswerRefusal, Toxicity, PromptInjection, DataPrivacyCompliance for any user-facing surface.

Install the evals with pip install agent-learning-kit and set FI_API_KEY and FI_SECRET_KEY. The code below runs Groundedness, ContextAdherence and TaskCompletion; the others take the same evaluate() call with their own inputs.

from fi.evals import evaluate

cases = load_golden_set("modal_rag_v2.jsonl")
for c in cases:
    g = evaluate("groundedness", input=c["input"], context=c["context"], output=c["output"], model="turing_flash")
    ca = evaluate("context_adherence", context=c["context"], output=c["output"], model="turing_flash")
    tc = evaluate("task_completion", input=c["input"], output=c["output"], model="turing_flash")
    print(g.score, ca.score, tc.score)

If the baseline regresses, fixing latency parity or shutdown-determinism won’t save the deployment. The baseline is the floor; the three Modal-specific checks are the ceiling. For rubric design and dataset shape, see the 2026 LLM evaluation playbook.

The traceAI integration for Modal

A Modal deployment exposes an HTTPS endpoint, usually served by an OpenAI-compatible server (vLLM’s OpenAI server or SGLang), so the traceAI OpenAI instrumentor covers it with base_url pointing at the Modal endpoint. Attempt metadata and tags passed through using_attributes identify the intended regime and prior-attempt history; Check 1 also emits a standard OpenTelemetry summary span with the final lifecycle record.

from fi_instrumentation import register, using_attributes
from fi_instrumentation.fi_types import ProjectType
from traceai_openai import OpenAIInstrumentor
from openai import AsyncOpenAI

trace_provider = register(
    project_type=ProjectType.OBSERVE,
    project_name="modal-llm-prod",
)
OpenAIInstrumentor().instrument(tracer_provider=trace_provider)

client = AsyncOpenAI(
    base_url="https://<workspace>--llama70b.modal.run/v1",
    api_key="modal-token",
)

async def call(prompt: str, invocation_type: str):
    # Reuse Check 1's streaming probe and its measured lifecycle record.
    return await one_call(prompt, invocation_type)

Every chat completion emits an LLM span with model name and token counts. Attempt metadata (runtime, modal.invocation_type, modal.saw_503_before_attempt) and the modal-cold, modal-warm or modal-concurrent tag identify the request context in Observe. Query the modal.invocation.summary span attributes or returned records for final modal.saw_503, status and retry-inclusive timing; an attempt context cannot describe a later outcome. The same pattern works in TypeScript via @traceai/openai. For more on the underlying schema, see traceAI’s OpenTelemetry-native tracing.

Future AGI trace tree and span-detail view; the application supplies Modal invocation regime and observed 503 path as metadata

The Agent Command Center for Modal eval

The gateway treats a Modal deployment as another OpenAI-compatible upstream alongside 100+ other providers. Add the Modal endpoint as a provider with api_format: "openai" and its base_url, attach budgets and guardrails, and gateway responses carry the header set that makes per-call observability ride on the trace tree without extra instrumentation:

  • x-agentcc-model-used: the underlying model that actually served the request, which may differ from the requested model after a fallback
  • x-agentcc-cost: estimated USD cost, sent when the model has pricing data (0 on exact cache hits)
  • x-agentcc-latency-ms: end-to-end latency measured at the gateway
  • x-agentcc-fallback-used: boolean indicating whether the call hit a fallback (so cost and parity scoring attribute correctly)
  • x-agentcc-routing-strategy: the strategy that selected the upstream (round-robin, weighted, least-latency, cost-optimized, adaptive, fastest)
  • x-agentcc-guardrail-triggered: true when a guardrail fired

Shadow mode is the routing primitive that matters most for Modal eval. The gateway sends production traffic to your Modal endpoint and silently copies the same prompt to a backup: a different Modal function with min_containers=1 for the cold-tail fallback, or an API provider like Fireworks for the burst-overflow path. The backup response is logged but not returned to the caller, which makes per-invocation-type parity scoring run on real production traffic without affecting users.

Race mode picks whichever upstream returns first (right for latency-sensitive paths, wrong for parity eval). The gateway also ships as a Go binary and Docker image, so you can self-host it. For the broader gateway story, see the best self-hosted AI gateways comparison.

How Future AGI ships Modal eval

Future AGI ships the eval stack as a package. Start with the SDK and traceAI for code-defined gates. Add the gateway when shadow routing becomes the workflow. Add the Platform for evals on live traces, alerts and failure grouping.

  • agent-learning-kit SDK (Apache-2.0, pip install agent-learning-kit). Eval templates covering Groundedness, ContextAdherence, TaskCompletion, LLMFunctionCalling, AnswerRefusal, PromptInjection, Toxicity, and DataPrivacyCompliance. Custom LLM-as-Judge and code evals cover Modal-specific rubrics.

  • traceAI. OpenTelemetry-native, 80 integration families across 5 languages. The OpenAI-compatible instrumentor covers Modal out of the box. The probe supplies attempt context and a final invocation summary with measured lifecycle fields for the three Modal-specific checks.

  • Agent Command Center. Go binary and Docker image, part of the Apache-2.0 core, 100+ providers including Modal as an OpenAI-compatible upstream. Shadow mode for parity scoring; the gateway README reports about 29k req/s on a t3.xlarge with P99 at or under 21 ms. Returns the header set above; conditional headers appear when they apply. Built-in guardrails cover safety regardless of which upstream fired.

  • Future AGI Platform. Observe runs evals on traced spans and alerts on latency, token usage or eval scores. Error Feed groups traces that failed the same way into one issue and points at the fix layer.

Traces feed the eval set, failures group into named issues, and the next traces show whether a routing or Modal config fix held. You then judge every Modal config change on the same traces and the same evals as the release before it.

Anti-patterns to avoid

Reporting one aggregate p99. The warm number looks healthy; the cold tail sits in the long tail and users who arrive after an idle period feel it on the first request of every session. Tag attempts with intended modal.invocation_type, keep final observed modal.saw_503 on the invocation summary, and compute percentiles from summary timing attributes or returned records per regime and availability path.

Single-concurrency testing. Real traffic is bursty. Content quality drifts when several requests share a replica. Run the golden set at three concurrency tiers (1, 10, 50) and score the p1 of the quality metric.

Treating min_containers as set-and-forget. Shutdown behavior drifts between Modal SDK versions and infrastructure changes you don’t control. Verify shutdown-determinism on every release with the strict timing sweep above.

No per-concurrency cost tracking. GPU-second billing means cost-per-token varies sharply with concurrent batch size. Divide billed GPU-seconds by the tokens your spans recorded, per tier, and you get a true per-concurrency cost curve, not the Modal monthly bill arriving a month after the regression.

Making a Modal deployment hold up under real traffic

The three-check pattern works for any serverless GPU runtime, Modal or otherwise. Per-invocation-type latency splits translate directly to Replicate, RunPod Serverless, and Beam. Tail-quality-under-burst applies to any runtime that packs concurrent requests onto shared GPUs. Shutdown-determinism is universal across scale-to-zero platforms.

For the sibling story on managed open-weight providers, see evaluating Fireworks AI and Together AI inference. For the parity reference against a fixed deployment, see evaluating vLLM self-hosted LLM apps. For the catalog of serverless GPU alternatives, see best Modal LLM serving alternatives.

A Modal-served LLM is two systems pretending to be one: the model and the runtime. Score cold, warm and concurrent traffic apart, run the golden set under burst, and sweep the shutdown window each release, so the deployment holds its SLO across every regime real traffic hits.

Trace your Modal endpoint and score it on Future AGI.

Sources

Frequently Asked Questions

What is Modal LLM inference?

Modal LLM inference means serving an open-weight model on Modal's serverless GPUs, usually through vLLM's OpenAI-compatible server, with per-second billing and scale-to-zero. You define the image, GPU and scaling settings in Python and run `modal deploy`. Evaluating it means scoring cold, warm and concurrent traffic separately.

How do you run vLLM on Modal?

Modal's vLLM example builds an image with vLLM, caches weights in a Modal Volume, and starts `vllm serve` inside an `@app.server` class from an `@modal.enter()` method. Running `modal deploy` then returns an OpenAI-compatible URL. Point the OpenAI SDK, traceAI's OpenAI instrumentor, or a gateway at that URL.

How do you reduce Modal cold start time for an LLM?

Keep containers warm with `min_containers` or a longer `scaledown_window`, cache weights in a Volume, and enable memory snapshots to cut Modal cold start time for LLM servers. Modal reported GPU snapshots cutting Ministral 3 3B median cold starts from about 118 to about 12 seconds. Re-measure after every change.

How does Modal inference pricing work for LLMs?

Modal bills inference per second of GPU time, so you pay while a container runs, including idle time inside the scaledown window. Modal lists an H100 SXM5 at $0.001097 per second and an H200 SXM at $0.001261. The Starter plan includes $30 of free credit per month.

What does scaledown_window do on Modal?

`scaledown_window` sets how many seconds a Modal container may sit idle before it shuts down, from 2 to 3600 seconds, with a 60-second default. It is a ceiling: the autoscaler can scale down sooner when a Function is over-provisioned, so verify it with a timing sweep.
Related Articles
View all