Evaluating Fireworks AI and Together AI Inference in 2026: Bit-Fidelity, Arena-Judge, and Latency Parity for Managed Open-Weight LLMs

The launch ships on Fireworks AI in five minutes. meta-llama/Llama-3.3-70B-Instruct, OpenAI-compatible base URL, copy-paste from the docs. The eval ran clean on the Hugging Face weights last week. By Wednesday the on-call thread reads: JSON-mode adherence drops 6 points on the support agent, the second tool call in three-step chains starts dropping arguments, p99 first-token latency triples on the 9am burst. The model card said Llama 3.3 70B. The bytes coming back over the wire said a quantized variant served on a different kernel with a non-default top_p. The eval suite never checked.

Managed inference for open-weight models is the fastest five minutes in your roadmap, and the quietest place to ship a regression. The OpenAI-compatible API is a contract about request and response shape, not about weights, precision, kernel, or sampler defaults.

The opinion this post earns: the eval that matters for managed inference is provider-bit-fidelity + per-provider arena-judge + per-provider latency parity, run on your own data, against the variant the provider actually serves. Headline benchmarks won’t catch quantization drift. Pricing pages won’t catch burst-load throughput collapse. The press release won’t disclose that the FP16 model card is being served as an FP8 or AWQ-INT4 build. Without these three checks plus quantization disclosure and a sampler audit, you’re trusting the press release.

This guide is the playbook for evaluating apps on Fireworks AI, Together AI, Modal, and Replicate end to end in 2026. It’s shaped against the ai-evaluation SDK, the traceAI OpenAI-compatible instrumentor, and the Agent Command Center gateway. If you’re choosing between providers, the Fireworks alternatives, Together alternatives, Modal alternatives, and Replicate alternatives round-ups cover the catalog side.

TL;DR: managed inference is a serving problem dressed as an API call

Ship only when the five checks pass on the exact provider, the exact model identifier, and the exact traffic shape you’re routing to. Anything else is the model card eval and the production behavior pretending to be the same artifact.

Why managed inference needs provider-specific eval

When you call OpenAI or Anthropic, you evaluate one thing: the prompt and the application logic. The vendor owns the weights, the kernels, the scheduler, the quantization, and the SLA. Content quality is enough.

Managed inference for open-weight LLMs inverts every assumption underneath the OpenAI-compatible interface.

The weights aren’t really shared. Fireworks, Together, Modal, and Replicate all advertise meta-llama/Llama-3.3-70B-Instruct, but only one of them is serving it at the precision Meta published. Fireworks runs an FP8 build tuned for throughput. Together ships an INT8 variant on some tiers, FP8 on others. Modal serves whatever container you deployed, usually a vLLM image with quantization flags set during build. Replicate runs Cog containers, and the community template might be AWQ-INT4 without saying so in the README. The model card is shared. The bytes are not.

The kernels diverge. FlashAttention-3, PagedAttention, speculative decoding, prefix caching, and KV-cache layout decisions all shift the output distribution. Two providers running the same nominal weights with different kernels produce different generation distributions for the same prompt and sampler setting. None of this is malicious. It’s the cost of competing on speed at the same headline price.

The sampler defaults aren’t aligned. OpenAI’s API spec sets temperature=1.0, top_p=1.0, n=1, presence_penalty=0, frequency_penalty=0. Most providers follow it. Some override silently. A provider that defaults top_p to 0.95 returns systematically different samples than one that defaults to 1.0 (same user-supplied temperature, divergent outputs). Your eval baseline on one provider does not transfer to the other until you audit defaults.

The result is a five-check protocol, not a content-quality pass. The next sections walk each check with the eval primitive that runs it. For the self-hosted baseline you’ll want as the parity reference, the evaluating vLLM self-hosted LLMs guide covers the same checks against your own cluster.

Check 1: provider bit-fidelity against Hugging Face weights

The headline test. Send a deterministic prompt at temperature=0 with logprobs enabled. Compare the top-k token probabilities the provider returns against a self-hosted BF16 reference of the same model. Run on 50 to 100 prompts covering JSON, tool calls, reasoning, and long-context retrieval.

from openai import OpenAI

def fidelity_probe(provider_client, ref_client, prompt: str, k: int = 5):
    p_kwargs = dict(
        model="meta-llama/Llama-3.3-70B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        temperature=0, max_tokens=1, logprobs=True, top_logprobs=k,
    )
    prov = provider_client.chat.completions.create(**p_kwargs)
    ref  = ref_client.chat.completions.create(**p_kwargs)

    prov_top = {l.token: l.logprob for l in prov.choices[0].logprobs.content[0].top_logprobs}
    ref_top  = {l.token: l.logprob for l in ref.choices[0].logprobs.content[0].top_logprobs}

    top1_match = next(iter(prov_top)) == next(iter(ref_top))
    overlap = len(set(prov_top) & set(ref_top)) / k
    kl_proxy = sum(abs(prov_top.get(t, -20) - ref_top.get(t, -20)) for t in ref_top) / k
    return {"top1_match": top1_match, "topk_overlap": overlap, "kl_proxy": kl_proxy}

Aggregate the probes into three numbers per provider: top-1 match rate, top-k overlap, KL-proxy mean. A provider serving the published weights cleanly sits at top-1 match above 0.95 with KL-proxy under 0.5. An FP8 build often sits at 0.85–0.92 with KL-proxy around 1.0. INT8 and INT4 variants typically land below 0.80 with KL-proxy above 2.0, and the structured-output and tool-call cliffs follow. Numbers move with the model and the prompt set; the shape holds.

The check doesn’t tell you what precision the provider serves. It tells you how far the distribution has drifted from the reference. Pair it with check 4 to put a label on the drift, and you have a defensible answer to “is this the model the docs claim it is?”

Check 2: per-provider arena-judge on your own data

Headline benchmarks (MMLU, HellaSwag, IFEval, GPQA) don’t represent your traffic. Per-provider arena-judge does: the head-to-head LLM-as-judge protocol from Chatbot Arena, run on your prompts, scored by a stronger judge model.

from fi.evals import Evaluator
from fi.evals.metrics.llm_as_judges.custom_judge.metric import CustomLLMJudge
from fi.evals.metrics.llm_as_judges.types import CustomInput
from fi.evals.llm.providers.litellm import LiteLLMProvider
from fi.testcases import TestCase

provider_arena_judge = CustomLLMJudge(
    provider=LiteLLMProvider(),
    config={
        "name": "provider_arena_judge",
        "model": "claude-sonnet-4-5-20250929",
        "grading_criteria": (
            "You are judging two answers to the same prompt from different inference providers. "
            "Pick the better answer against the task: JSON validity, tool-call correctness, "
            "factual grounding, instruction adherence. Return 'A', 'B', or 'TIE'. "
            "Reward semantic correctness and structured-output fidelity. "
            "Penalize hallucinated fields, dropped tool args, and refusal on safe prompts."
        ),
    },
)

def arena_pair(prompt: str, answer_fw: str, answer_tg: str):
    return provider_arena_judge.compute_one(CustomInput(
        question=prompt, answer_a=answer_fw, answer_b=answer_tg,
    ))["output"]

Run 200 to 500 paired prompts across your traffic mix: JSON outputs weighted as you actually use them, tool-call sequences at the length you ship, RAG answers with the chunks your retrieval returns, long-context retrieval at the position the lost-in-the-middle effect cares about. The output is a win-rate per provider per workload slice, which is the number that tells you which provider serves your app better than any headline benchmark can.

Cluster the cases where one provider systematically wins or loses through Error Feed (HDBSCAN soft clustering plus a Sonnet 4.5 Judge writing the immediate_fix). Typical clusters: “Fireworks Llama 3.3 70B loses 11 points on JSON-mode versus Together FP16, fix: route JSON to Together,” “Together top_p default 0.95 versus Fireworks 1.0 leads to divergent samples, fix: pin sampler params client-side,” “long-context past 64k loses retrieval on Replicate community Cog, fix: pin to the official Meta deployment.” Each immediate_fix becomes a routing rule at the gateway.

Check 3: latency parity across cold, warm, and burst

Latency parity covers three regimes, and most teams measure one. Tag every span with the request regime; compute p99 per regime, not in aggregate.

Cold start. Endpoint sleeping, time from request hit to first token. Replicate and Modal territory, where both platforms scale to zero by design. A Replicate community model on a cold container can take 60 to 300 seconds to wake. Modal apps with min_containers=0 and a cold GPU image sit in the 15 to 60 second range. Fireworks and Together don’t surface a cold-start regime because they multi-tenant the GPU pool; the closest analog is a model spin-up on a tier you haven’t used recently.

Warm. Endpoint hot, normal concurrency. TTFT (time-to-first-token), ITL (inter-token latency), and total completion latency are the three numbers. Fireworks and Together sit here for most queries. Modal sits here once at least one container is warm. Replicate sits here when you’ve pinned min_workers above zero (which trades latency for cost).

Burst. Concurrent requests overwhelm allocated capacity. The tail where p99 triples and the cheaper provider at concurrency 1 stops being the cheaper provider at concurrency 50. A throughput sweep with realistic concurrency catches it.

import asyncio, time
from openai import AsyncOpenAI
from fi_instrumentation import using_attributes

async def one_call(client, prompt, regime, quant):
    with using_attributes({
        "llm.system": "fireworks",
        "llm.serving.quantization": quant,
        "llm.request.regime": regime,
    }):
        t0 = time.time()
        resp = await client.chat.completions.create(
            model="accounts/fireworks/models/llama-v3p3-70b-instruct",
            messages=[{"role": "user", "content": prompt}],
            max_tokens=512,
        )
        return {
            "ttft_ms": None,                              # streaming: capture first chunk timestamp
            "total_ms": (time.time() - t0) * 1000,
            "tok_per_s": resp.usage.completion_tokens / (time.time() - t0),
        }

async def sweep(client, prompts, regime, quant):
    return await asyncio.gather(*[one_call(client, p, regime, quant) for p in prompts])

# Run cold (after deliberate idle), warm (after a min of traffic), burst (concurrency 50+)
for regime in ["cold", "warm", "burst"]:
    results = asyncio.run(sweep(fireworks_client, golden_set[:50], regime, "fp8"))

A representative finding on Llama 3.3 70B (illustrative): Fireworks warm TTFT 280ms, burst p99 1.4s; Together warm TTFT 240ms, burst p99 1.1s; Modal warm TTFT 320ms, cold 28s, burst p99 950ms; Replicate warm TTFT 410ms with min_workers=1, cold 95s. The right provider depends on your regime mix.

The gateway records x-agentcc-latency-ms per call, so per-regime distributions ride on the trace tree without extra instrumentation. The LLM observability tracing guide walks the standard OpenInference attribute set.

Check 4: quantization disclosure (the provider won’t tell you)

Provider model cards rarely name the precision. Fireworks lists “FP8” on a handful of pages and stays silent on others. Together exposes precision on some tiers and not others. Modal serves whatever the container builder set, opaque unless you control the Dockerfile. Replicate community models often ship AWQ-INT4 or GPTQ-INT4 with no README mention.

Detection is empirical. Three signals stack to a defensible label.

Signal one: the bit-fidelity check from above. Top-1 match above 0.95 with KL-proxy under 0.5 is consistent with full-precision serving. 0.85–0.92 with KL-proxy around 1.0 is consistent with FP8. Below 0.80 with KL-proxy above 2.0 is INT8 or INT4 territory.

Signal two: the structured-output cliff. Free-form generation often stays within one point of the reference across FP8, INT8, and INT4. JSON-mode strict outputs drop 3 to 8 points on INT4. Tool-call argument correctness drops 4 to 10 points on three-or-more-tool composition. A per-axis delta shaped like a structured-output failure puts you on a quantized variant.

Signal three: the cost-versus-throughput ratio. Providers serving FP16 on H100s at a Llama 3.3 70B catalog price near the cheapest tier are an arithmetic problem. The economics force quantization. If a provider charges 0.6x the published H100-hour rate for a 70B model, they aren’t running BF16.

Codify the three signals as a QuantizationDisclosure rubric. Run it weekly. The output is a per-provider per-model precision label (FP16 / FP8 / INT8 / INT4 / UNKNOWN) with confidence. Pair it with check 1 to know when a provider silently swaps variants, and you have a record the procurement team can take to the vendor.

Check 5: sampler default parity

The quietest source of cross-provider divergence. Send the same prompt with no sampler parameters to all four providers, ask for logprobs on top tokens. If the top-token distributions differ for a deterministic-looking prompt like “The capital of France is”, the defaults are not aligned.

Codify the audit as a SamplerDefaultParity rubric, run weekly. If providers shift defaults between releases (they sometimes do, especially after a model rev), the rubric catches it before the rest of your eval suite drifts. The fix is client-side: always pass temperature, top_p, n, presence_penalty, and frequency_penalty explicitly, even when you’re using spec defaults. The bytes match better when you stop trusting implied state.

Wiring multi-provider traceAI in five lines

Fireworks, Together, Modal, and Replicate all expose OpenAI-compatible chat completion endpoints (Modal and Replicate via their OpenAI-compatible deployment templates), so the traceAI OpenAI instrumentor covers them with base_url pointing at each provider.

from fi_instrumentation import register, ProjectType, using_attributes
from traceai_openai import OpenAIInstrumentor
from openai import OpenAI
import os

trace_provider = register(
    project_type=ProjectType.OBSERVE,
    project_name="managed-inference-eval",
)
OpenAIInstrumentor().instrument(tracer_provider=trace_provider)

providers = {
    "fireworks": OpenAI(api_key=os.environ["FW_KEY"],  base_url="https://api.fireworks.ai/inference/v1"),
    "together":  OpenAI(api_key=os.environ["TG_KEY"],  base_url="https://api.together.xyz/v1"),
    "modal":     OpenAI(api_key=os.environ["MD_KEY"],  base_url=os.environ["MODAL_BASE_URL"]),
    "replicate": OpenAI(api_key=os.environ["RP_KEY"],  base_url="https://openai-proxy.replicate.com/v1"),
}

def call(name, model, messages, regime="warm", quant="unknown"):
    with using_attributes({
        "llm.system": name,
        "llm.serving.quantization": quant,
        "llm.request.regime": regime,
    }):
        return providers[name].chat.completions.create(model=model, messages=messages)

Every chat completion now emits fi.span.kind=LLM spans with llm.model_name and token counts, plus the three custom attributes (llm.system, llm.serving.quantization, llm.request.regime) that feed the per-provider checks. Add llm.serving.region if you route geographically. Those five attributes plus the standard token counts give you everything the bit-fidelity, arena-judge, latency-parity, quantization-disclosure, and sampler-parity rubrics need.

How the Agent Command Center routes Fireworks, Together, Modal, and Replicate

The gateway routes all four providers through the OpenAI-compatible preset path alongside 100+ other upstreams. You add each as an upstream with its base URL and API key, attach budgets and guardrails, and every gateway response returns a header set that gives you per-call observability without extra instrumentation:

• x-agentcc-model-used: the underlying model that actually served the request, which may differ from the requested model after a fallback or routing override
• x-agentcc-cost: computed from per-provider pricing tables refreshed against vendor announcements
• x-agentcc-latency-ms: end-to-end latency measured at the gateway
• x-agentcc-fallback-used: boolean indicating whether the call hit a fallback (so the cost and parity scoring can attribute correctly)
• x-agentcc-routing-strategy: the strategy that selected the upstream (round-robin, weighted, race, mirror, shadow)
• x-agentcc-guardrail-triggered: the scanner that fired if the call was blocked or modified

The two routing modes that matter for cross-provider eval are shadow and mirror. Shadow sends the primary request to one provider and silently copies it to a second; the second response is logged but not returned to the caller. Mirror returns both responses (or all four) to the caller so the provider_arena_judge rubric can score them inline. Race mode picks whichever provider returns first, which is the right mode for latency-sensitive paths and the wrong mode for parity eval. Deploy the gateway BYOC in the same cluster as your application and the hop adds nothing meaningful to the warm-path latency. For the broader gateway story, see the best LLM gateways round-up.

How Future AGI ships managed-inference eval

Future AGI ships the eval stack as a package. Start with the SDK and traceAI for code-defined gates. Graduate to the Platform when self-improving rubrics and per-cluster routing become the bottleneck.

• ai-evaluation SDK (Apache 2.0). 50+ EvalTemplate classes covering Groundedness, ContextAdherence, TaskCompletion, EvaluateFunctionCalling, AnswerRefusal, PromptInjection, Toxicity, and DataPrivacyCompliance. CustomLLMJudge is the primitive for provider_arena_judge, QuantizationDisclosure, and SamplerDefaultParity rubrics. Local heuristic metrics (regex, JSON schema, BLEU, ROUGE) run offline at sub-second latency, which matters when you score every provider against the same 500-case set on every release.
• traceAI. 50+ AI surfaces across Python, TypeScript, Java, and C#. The OpenAI-compatible instrumentor covers all four providers out of the box. Every span carries llm.model_name and token counts; add the five custom attributes above as using_attributes context and the four runtime rubrics share one span tree.
• Agent Command Center. Single Go binary, Apache 2.0, 100+ providers including all four here. Shadow, mirror, and race modes for cross-provider routing; eval-gated canary rollback as the default; benchmarked at ~29k req/s, P99 ≤ 21 ms with guardrails on, on t3.xlarge. Returns cost, latency, model, and routing-strategy headers on every call. 18+ built-in guardrail scanners cover safety regardless of which provider fired.
• Future AGI Platform. Self-improving evaluators that retune from production traces; in-product authoring agent that writes rubrics from natural-language descriptions; classifier-backed evals at lower per-eval cost than Galileo Luna-2. Error Feed clusters failing traces with HDBSCAN and a Sonnet 4.5 Judge writes the immediate_fix per cluster. Typical clusters: “Fireworks FP8 Llama 3.3 70B drops 6 points on JSON-mode versus Together FP16, fix: route JSON to Together,” “Replicate community Cog AWQ-INT4 loses long-context retrieval past 32k, fix: pin to official Meta deployment,” “Modal cold start adds 28s on first request, fix: set min_containers=1 on the budget tier.” Each immediate_fix flows back into the routing policy at the gateway.

Drop ai-evaluation and traceAI into the gate this afternoon. Add the gateway when shadow and mirror routing become the workflow. Turn the Platform on when per-cluster routing across providers is the bottleneck.

Ready to evaluate your first managed-inference stack? Run pip install ai-evaluation traceai-openai, scaffold the five checks against your golden set, point your OpenAI client at https://gateway.futureagi.com/v1 with Fireworks, Together, Modal, and Replicate configured as upstreams, and shadow-route to a second provider for parity scoring. The provider that survives all five checks on your traffic shape is the one worth shipping. Everything else is the press release.

Related reading

• Evaluating vLLM Self-Hosted LLMs (2026)
• Evaluating Modal LLM Inference (2026)
• Evaluating Anyscale Ray LLM (2026)
• Evaluating LiteLLM Multi-Provider Apps (2026)
• Evaluating Cheap Frontier Models (2026)
• LLM Eval Shadow Traffic and Canary Patterns (2026)
• Best LLM Gateways (2026)
• Top 11 LLM API Providers (2025)