What Is Autoformalism?

What Is Autoformalism?

Autoformalism is the task of converting natural-language statements — usually mathematical claims, specifications, or contracts — into a formal language that a mechanical checker can verify. In LLM systems it appears as a translation step: a prompt asks for a Lean, Coq, Isabelle, or TLA+ rendering of an English claim, and the resulting code is fed to a proof assistant. Production teams use autoformalism whenever a model’s answer must be machine-verified rather than rubric-judged. FutureAGI treats it as a structured-output reasoning task and evaluates it with schema, reasoning-quality, and groundedness metrics.

Why Autoformalism matters in production LLM and agent systems

The point of autoformalism is to replace “the model sounded confident” with “the proof checker accepted it.” Without that closure, a math or contract agent shipping into production has no objective failure signal — the LLM hallucinates a lemma and the user trusts a wrong answer. Autoformalism is the bridge between probabilistic generation and deterministic verification.

The pain shows up in three places. ML engineers see parser failure rates of 30–60% on first attempts, where the model produces Lean that doesn’t even parse. Verification engineers see proofs that parse but don’t close — sorry placeholders, missing hypotheses, broken type unifications. Product teams see latency spikes when an agent retries autoformalism in a tool-call loop until the checker accepts.

In 2026 agent stacks, autoformalism rarely runs as a one-shot prompt. It is wired into a multi-step trajectory: planner emits a sub-goal, autoformalism step renders Lean, proof-assistant tool returns success or error, agent revises. Unlike MiniF2F pass@k or a LeanDojo benchmark score, production autoformalism also has to track retry budget and trace context. A trajectory-level failure mode looks like the agent re-emitting the same broken Lean for nine iterations before timing out — an infinite-loop pattern that single-call evals miss.

How FutureAGI evaluates autoformalism

FutureAGI’s approach is to score autoformalism as a structured-output reasoning task and wire its failures into agent-trajectory observability. There is no Autoformalism evaluator in fi.evals and we do not invent one — instead the workflow combines existing evaluators on the right trace surfaces.

The setup looks like this. The autoformalism step is instrumented with traceAI-langchain (or traceAI-openai if the model call is direct). On every span where the formal output is emitted, we attach SchemaCompliance against the proof-assistant grammar and ReasoningQuality against the natural-language claim plus the formal rendering. Parser-pass and proof-pass results from the external checker are written back as span_event attributes — formal.parser_pass, formal.proof_closed, formal.unifier_errors — so they sit next to the LLM call.

In an Agent Command Center configuration, a routing policy can send “first attempt” traffic to a cheaper model and cost-optimized-routing only escalates to a stronger reasoning model when formal.parser_pass is false. A pre-guardrail rejects prompts that would cause prompt-injection contamination of the formal output. The team’s release gate is eval-fail-rate-by-cohort on SchemaCompliance and a manual diff of ReasoningQuality traces flagged for human annotation in the FutureAGI annotation queue. Regression evals run against a canonical Dataset of past benchmark theorems before each model swap.

How to measure autoformalism

Layer parser-level, proof-level, and reasoning-level signals:

• Parser-pass rate: percentage of generated formal outputs the target language parses without error.
• Proof-closed rate: percentage that close the goal in the proof assistant — the only fully objective signal.
• SchemaCompliance score: structural agreement between the formal rendering and the expected grammar.
• ReasoningQuality score: rubric-judged faithfulness of the formal claim to the natural-language original.
• FactualConsistency and contradiction detection: catch when the formal rendering subtly weakens or strengthens the claim.
• Trajectory signals: tool-retry count, time-to-proof, infinite-loop detection on the agent span tree.

A minimal structural check on the rendered formal output:

from fi.evals import SchemaCompliance

metric = SchemaCompliance()
result = metric.evaluate(
    response="theorem add_even (a b : Nat) (...): Even (a + b) := by ...",
    expected_schema="lean4_theorem_grammar",
)
print(result.score, result.reason)

Common mistakes

• Trusting parser-pass as proof success. Code that parses can still leave sorry in place and never close the proof.
• Evaluating only on famous theorems. Public-benchmark contamination inflates numbers; sample your own cohort.
• Letting the same model be the autoformalism generator and the eval judge. Self-judging masks subtle weakening of the claim.
• Ignoring retry budget. A 90% solve rate at 50 attempts is operationally a different system than 60% at one attempt.
• Treating contract autoformalism as math autoformalism. Legal text has open-world semantics that proof assistants do not model.