What Is an AI Software Factory and How to Measure Its Output

A platform team runs an AI software factory where agents write and review most pull requests. On a Tuesday, someone trims the reviewer agent’s prompt to cut noisy comments. Two weeks later an incident traces back to a change that reviewer approved, and the obvious question comes up.

Did the Tuesday edit make the reviewer worse, or was that change always going to slip through? Nobody can say. Each gate decision lived as a click on a pull request, and nobody recorded which reviewer prompt version was live when the agent left its review.

That scene is the measurement problem of an AI software factory. No one scores the reviewer agent, and teams often throw away the cheapest measurement data they already have: the approve and request-changes verdicts their human reviewers hand out every day.

TL;DR

• An AI software factory uses agents to turn specs into shipped code, checked by tests, reviewer agents and human gates.
• The term dates to the late 1960s, and Hitachi’s 1969 Software Works was the first facility to carry it. The agentic version took shape between 2024 and 2026 through Dan Shapiro’s levels, StrongDM’s rules, 8090’s Software Factory and Uber’s report on agent-attributed pull requests.
• A dark factory removes human review. The human verdicts other factories keep are the main outside reference for the other checks, as long as humans review a random sample without seeing the agent’s call.
• Measure throughput, quality after merge, review accuracy, gate load and cost together.
• Score each reviewer prompt version against those human verdicts with Cohen’s kappa and missed rejections, and promote only once the labeled set is big enough to trust.

What is an AI software factory?

An AI software factory is a development pipeline where agents turn a written spec into shipped code, with other agents and automated tests checking the work at each step. People write the specs, design the checks, and decide where a human must still approve a change before it ships.

The difference from one engineer using a coding agent is the hand-off. A developer with an agent checks each result personally. A factory is a standing line: work moves from spec to writer to tests to reviewer to gate without a person carrying it, and each station records what it decided.

AI software factory versus AI factory

NVIDIA’s definition of an AI factory is “a specialized computing infrastructure designed to create value from data”, which is compute, not a software delivery process. This post does not cover that meaning.

Where the software factory idea came from

The term is older than coding agents. Its agentic meaning took shape between 2024 and 2026 through four public sources.

8090, launched in 2024, sells a Software Factory product. StrongDM formed its AI team in July 2025 and wrote rules that ban humans from writing or reviewing code. Dan Shapiro published five levels that end in a dark factory, and Uber put agent-attributed pull requests above 70%.

From Hitachi’s Software Works to Kessel Run

R.W. Bemer urged GE to build a software factory in 1968.

By founding its Software Works in 1969, Hitachi became the first company to call a software facility a factory, and System Development Corporation opened the first US one in 1975 to 1976, according to Michael Cusumano’s history of the software factory. Those factories standardized tools, methods and reuse for human programmers.

The US military revived the name for in-house teams with their own DevSecOps pipelines. The Air Force stood up its first, Kessel Run, in 2017, years before coding agents could open a pull request on their own.

Dan Shapiro’s five levels and the dark factory

Dan Shapiro’s five levels (January 23, 2026, “The Five Levels: from Spicy Autocomplete to the Dark Factory”) borrow the levels of driving automation.

The scale runs from Level Zero to Level 5. At Level 3 the developer manages agents: “Your life is diffs.” Shapiro adds that “almost everyone tops out here.” Level 5 borrows its “dark” from the Fanuc Dark Factory, “a place where humans are neither needed nor welcome.”

The quotes are Shapiro’s. The two right-hand columns are our reading of each level, not his words. They matter for this post because human review, the step that disappears at Level 5, is where the only outside check on a factory’s reviewer agents comes from.

StrongDM’s rules for a factory without human review

StrongDM’s software factory page (February 6, 2026, by co-founder and CTO Justin McCarthy) states two rules: “Code must not be written by humans” and “Code must not be reviewed by humans”.

Its definition is “non-interactive development where specs + scenarios drive agents that write code, run harnesses, and converge without human review.” Scenarios are end-to-end user stories “often stored outside the codebase (similar to a “holdout” set in model training)”. Its spending bar, “$1,000 on tokens today per human engineer”, is StrongDM’s own benchmark.

8090’s Software Factory and the engineer sign-off

8090 Software Factory, which 8090 calls “The AI-native SDLC platform that refines intent, governs execution, verifies outputs and preserves the complete audit trail,” moves work through Requirements, Blueprints, Work Orders and Feedback.

Blueprints are “the technical design your engineers review and sign off on.” Work Orders then go “to your engineers and their agents through MCP, in Cursor, VS Code, or Claude Code.” A human signs off before agents build, so this is a factory with a gate by design.

Uber’s report on pull requests attributed to agents

Uber Engineering’s software factory report, published August 27, 2026 by Distinguished Engineer Uday Kiran Medisetty, states in its introduction: “More than 70% of pull requests are attributed to local or cloud agents.”

It also counts over 3,600 agent skills and more than 30K skill executions per day. Its managed agents handle code review, CI failures, on-call triage and code maintenance, the last “with human reviews/escalations”. So read the 70% as attribution, not autonomy: an agent-attributed pull request can still pass a human reviewer.

AI software factory architecture from spec to merge

Most AI software factory architectures share five stations: a spec that defines done, writer agents that produce the change, tests or scenarios that check behavior, reviewer agents that read the diff, and human gates that approve what the checks cannot settle. Each station emits a verdict that later stations, and your measurements, depend on.

Release, rollout and monitoring sit after the merge. This post meets them only through the quality-after-merge metrics further down.

The spec and the definition of done

The spec carries the acceptance criteria and the risk level of the change, and everything downstream is judged against it. StrongDM’s scenarios and 8090’s Requirements and Blueprints are two forms of this station: one written as user stories, the other as a design a person signs.

Writer agents and the harness they run in

Each writer agent runs inside a harness that gives it tools, a turn limit and a budget. The harness decides what the agent can touch and when it must stop, so two factories on the same model can ship different code. See what an agent harness does.

Tests, scenarios and CI

Tests live in the repository next to the code. Scenarios, in StrongDM’s sense, are often kept outside it, so the writer agent cannot edit them to make a change pass. CI returns the first automatic verdict before any reviewer reads the diff.

Reviewer agents and routing

Reviewer agents read each diff and return approve or request changes, and a router decides whether the change merges or waits for a human. A reviewer’s behavior comes from its instructions and its model, so editing the instructions changes what it approves while no application code changes.

Human gates

The human gate is where a person approves what the checks could not settle: 8090’s Blueprint sign-off before the build, Uber’s reviews and escalations after it. Each gate decision records a verdict, a reviewer and a commit, the raw material for every review accuracy number below.

Dark factory software development and how far to take the humans out

Dark factory software development removes human code review entirely and trusts tests, scenarios and agent judges to decide what merges. StrongDM runs this way by rule. 8090 and Uber keep a human in the loop, which matters because a factory with no human verdicts has no outside human reference to check its checkers against.

That is the case against Level 5. A human gate costs reviewer time but produces labels. Remove it and the graders left are other agents, plus tests and scenarios the factory’s own team wrote. Keep it on a random share of changes, chosen independently of the agent’s verdict, as an unbiased sample of ground truth.

Large public factories keep that gate. Stripe says over a thousand pull requests merged each week are fully produced by its Minions agents, human-reviewed but with no human-written code. Spotify teams had merged over 1,500 pull requests from its background coding agent by November 2025.

For open source AI software factory examples, start with StrongDM’s Attractor spec, published under Apache-2.0 as the “nlspec of StrongDM’s Attractor, a non-interactive Coding Agent sufficient for use in a Software Factory”, and the smaller super-simple-software-factory project is MIT-licensed.

Where an AI software factory breaks

An AI software factory breaks where its checks stop being independent of the work they check. Tests in the repo can be rewritten to pass, reviewer agents drift when someone edits their prompt, judge models share blind spots with the writers, and specs that leave gaps get filled with confident guesses.

Tests that bend to the code

StrongDM hit this in its own build: “return true is a great way to pass narrowly written tests, but probably won’t generalize to the software you want.” It also lists “Tests can be reward hacked.” Its answer: scenarios run against behavioral clones of services such as Okta, Jira and Slack.

Reviewer prompts that change without a measurement

This is the opening scene. A reviewer prompt edit is a behavior change, yet it ships with no test, no version stamp on the reviews it produces and no comparison against the prompt it replaced. Code changes get CI, while reviewer changes usually get a quick read.

Judges grading their own kind

When one model writes the code and a similar model reviews it, their mistakes can line up, and the reviewer approves exactly what it should catch. Our view is that agreement with the human gate is the main outside reference most factories have for this.

Spec gaps and cost

Vague acceptance criteria let writer agents fill gaps with guesses that pass every check, because the checks came from the same spec. Cost breaks quietly too: retries, long context and failed runs never show in a seat price. Uber’s report tracks outcome-denominated cost, including cost per merged PR.

How do you measure the output of an AI software factory?

Measure an AI software factory on five things together: how much merges, how much holds up after release, how often reviewer agents agree with the human gate, how much work still waits on a human, and what each merged change costs. Throughput alone rewards a factory that ships more broken changes faster.

The first, second, fourth and fifth areas come straight from logs your Git host, router and bills already keep. Review accuracy is the one that needs a labeled set, so the rest of this post builds it.

Gate load is where factories back up. Across 8.1 million pull requests from 4,800 teams, LinearB’s 2026 benchmarks found that AI pull requests wait 4.6 times longer before review, and are accepted 32.7% of the time against 84.4% for manual ones.

Uber’s report lists “cost per merged PR, cost per review” for each managed agent, next to revert rate. Its uReview benchmark uses real pull requests with known bugs and scores “precision, recall, and F1 against those bugs, plus cost per review, latency, timeouts, and noise.”

Why review accuracy needs its own measurement

Raw agreement flatters a reviewer that approves almost everything. If 80% of pull requests deserve approval, a reviewer that always approves scores 80% while catching nothing. Cohen’s kappa removes the agreement expected by chance from each side’s approve and reject rates, so that reviewer scores zero.

In our reading of published factory-metrics guides, as of November 2026, most count review capacity, such as queue depth, or grade agent sessions with an LLM judge. Few check whether that reviewer or judge agrees with a human. Without that check, a reviewer that drifts after a prompt edit looks the same on a dashboard as one that improved.

Report missed rejections and needless blocks as separate counts. A missed rejection is a bad change the reviewer approved and the human gate stopped. A needless block is a good change the reviewer held up. One ships defects, the other costs an engineer a wait.

How this differs from scoring a code review tool

Grading a tool’s individual review comments is a separate job, covered in scoring AI code review tools on precision and recall, while here the unit is the factory’s approve or request-changes decision, scored against the human gate.

Turning human review verdicts into a labeled dataset

Your human gate already decides approve or request changes on pull requests the reviewer agent has read. Store each decision with the PR number, the commit SHA the agent read, and the reviewer prompt version. Join them on commits both a human and an agent judged, and you have a labeled dataset.

The log needs one line per verdict. These are the first three lines of the sample file, which is synthetic: 67 events across pull requests 4101 to 4120, from a human gate and two reviewer prompt versions, v3 and v4.

{"pr": 4101, "commit": "07ee33e", "reviewer": "agent:reviewer@v3", "verdict": "approve"}
{"pr": 4101, "commit": "07ee33e", "reviewer": "agent:reviewer@v4", "verdict": "request_changes"}
{"pr": 4101, "commit": "07ee33e", "reviewer": "human:gate", "verdict": "request_changes"}

Join verdicts on the PR and commit together

The script keys every verdict on the pair of PR and commit, so a human approval on a later fix commit never pairs with an agent review of the earlier one. It keeps only commits judged by the human and both versions. In the sample, seven human approvals land on fix commits no agent reviewed, and the join drops them.

Record the prompt version on every agent verdict

The reviewer field carries the prompt version, as in agent:reviewer@v4. Without it, the opening scene’s question has no answer. With it, you can split any month of reviews by version and see what changed after each prompt edit, and when.

Sample the gate at random and keep it blind

Two things bias these labels. If the router sends a human only some changes, such as the ones the agent flagged, the bad changes the agent approved never reach a person, and missed rejections stay invisible. A human who can see the agent’s verdict also tends to agree with it, which inflates agreement.

So send a fixed share of changes to the human gate at random, chosen independently of the agent’s verdict. Hide the agent’s verdict until the human has decided, and score only that sample.

Score agreement against the human label with Cohen’s kappa

Observed agreement is the share of commits where agent and human gave the same verdict. Chance agreement is what two raters would reach by luck, given how often each approves. Kappa is the share of the remaining room above chance that the reviewer earns.

For v3, observed agreement is 0.70. The agent approves 15 of 20 commits and the human approves 13, so chance agreement is 0.75 times 0.65 plus 0.25 times 0.35, or 0.575. Kappa is (0.70 minus 0.575) divided by (1 minus 0.575), about 0.29.

What the dataset should hold

Keep both outcomes, with enough rejections to score. A set where the human approves 98% of commits says almost nothing about missed rejections. The script will not promote on fewer than 50 labeled rows or 10 rejections, and the sample holds 20 rows and 7 rejections. For what else belongs in the set, see designing a golden set.

"""Turn human review verdicts into a labeled dataset and score reviewer-agent prompt versions.

Input: a JSONL log of review events, one per verdict, from the random, blind human sample:
  {"pr": 4101, "commit": "07ee33e", "reviewer": "human:gate", "verdict": "request_changes"}
Output: labeled_reviews.csv plus a per-version scorecard and a promote/hold decision.
"""
import csv
import json
import random
import sys
from collections import defaultdict

BASELINE, CANDIDATE = "agent:reviewer@v3", "agent:reviewer@v4"
MIN_ROWS, MIN_REJECTIONS = 50, 10  # below either floor, report the scores but never promote


def build_dataset(events_path):
    by_commit = defaultdict(dict)  # (pr, commit) -> {reviewer: verdict}
    with open(events_path) as f:
        for line in f:
            if not line.strip():
                continue  # tolerate blank lines
            e = json.loads(line)
            by_commit[(e["pr"], e["commit"])][e["reviewer"]] = e["verdict"]
    rows = []
    for (pr, commit), verdicts in sorted(by_commit.items()):
        human = verdicts.get("human:gate")
        if human and all(v in verdicts for v in (BASELINE, CANDIDATE)):
            rows.append({"pr": pr, "commit": commit, "label": human,
                         BASELINE: verdicts[BASELINE], CANDIDATE: verdicts[CANDIDATE]})
    return rows


def cohen_kappa(pairs):
    n = len(pairs)
    if not n:
        return None
    observed = sum(a == b for a, b in pairs) / n
    cats = {x for pair in pairs for x in pair}
    expected = sum(
        (sum(a == c for a, _ in pairs) / n) * (sum(b == c for _, b in pairs) / n) for c in cats
    )
    return None if expected == 1 else (observed - expected) / (1 - expected)  # undefined, not perfect


def interval(values):
    values = sorted(v for v in values if v is not None)
    return (values[int(0.025 * len(values))], values[int(0.975 * len(values)) - 1]) if values else None


def kappa_interval(pairs, draws=2000, seed=0):
    """95% bootstrap interval for one version's kappa: resample commits with replacement."""
    rng = random.Random(seed)
    return interval(cohen_kappa(rng.choices(pairs, k=len(pairs))) for _ in range(draws))


def kappa_gain_interval(rows, draws=2000, seed=0):
    """95% paired bootstrap interval for candidate kappa minus baseline kappa on the same commits."""
    rng = random.Random(seed)
    gains = []
    for _ in range(draws):
        sample = rng.choices(rows, k=len(rows))
        b = cohen_kappa([(r[BASELINE], r["label"]) for r in sample])
        c = cohen_kappa([(r[CANDIDATE], r["label"]) for r in sample])
        gains.append(None if b is None or c is None else c - b)
    return interval(gains)


def score(rows, version):
    pairs = [(r[version], r["label"]) for r in rows]  # both versions use the same matched commits
    if not pairs:
        return None  # no common labeled commits
    return {
        "agreement": sum(a == h for a, h in pairs) / len(pairs),
        "kappa": cohen_kappa(pairs),
        "interval": kappa_interval(pairs),
        "missed_rejections": sum(a == "approve" and h == "request_changes" for a, h in pairs),
        "needless_blocks": sum(a == "request_changes" and h == "approve" for a, h in pairs),
    }


def decide(rows, b, c):
    rejections = sum(r["label"] == "request_changes" for r in rows)
    if not b or not c or b["kappa"] is None or c["kappa"] is None:
        return f"insufficient data; keep {BASELINE}"
    if len(rows) < MIN_ROWS or rejections < MIN_REJECTIONS:
        return (f"too few labels ({len(rows)} rows, {rejections} rejections; "
                f"need {MIN_ROWS} and {MIN_REJECTIONS}); keep {BASELINE}")
    gain = kappa_gain_interval(rows)
    if c["missed_rejections"] <= b["missed_rejections"] and gain and gain[0] > 0:
        return f"promote {CANDIDATE}"  # no more missed rejections, and kappa gain clears zero
    return f"keep {BASELINE}"


if __name__ == "__main__":
    assert cohen_kappa([("a", "a"), ("b", "b")]) == 1.0  # perfect agreement
    assert cohen_kappa([("approve", "approve")]) is None  # no variation to measure
    rows = build_dataset(sys.argv[1])
    with open("labeled_reviews.csv", "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=["pr", "commit", "label", BASELINE, CANDIDATE])
        w.writeheader()
        w.writerows(rows)
    print(f"labeled rows: {len(rows)} (human rejected {sum(r['label'] == 'request_changes' for r in rows)})")
    s = {v: score(rows, v) for v in (BASELINE, CANDIDATE)}
    for v, m in ((v, m) for v, m in s.items() if m):  # skip versions with no labeled commits
        kappa = "insufficient data" if m["kappa"] is None else f"{m['kappa']:.2f}"
        if m["interval"]:
            kappa += f" (95% CI {m['interval'][0]:.2f} to {m['interval'][1]:.2f})"
        print(f"{v:<20} agreement {m['agreement']:.0%}  kappa {kappa}  "
              f"missed rejections {m['missed_rejections']}  needless blocks {m['needless_blocks']}")
    gain = kappa_gain_interval(rows)
    if gain:
        print(f"kappa gain, candidate minus baseline: 95% CI {gain[0]:.2f} to {gain[1]:.2f}")
    print("decision:", decide(rows, s[BASELINE], s[CANDIDATE]))

Comparing reviewer prompt versions before you promote one

Run the current reviewer prompt and the candidate over the same labeled commits, score both against the human verdicts, and promote the candidate only if it misses no more rejections, gains kappa by more than chance resampling explains, and the labeled set is big enough to trust.

Write that rule down before the run, so the numbers cannot talk you into it.

The candidate has never reviewed live traffic, so replay it over the stored diffs of those commits, on the same model and settings as the live prompt.

Run the script with python -I score_reviewer_prompts.py review_events.jsonl. It writes labeled_reviews.csv. A synthetic log with the verdict counts above produces the scorecard below, though the intervals shift by a few hundredths with which commits carry which verdicts.

A hand check agrees: for v4, observed agreement is 0.90 and chance agreement is 0.545, which gives a kappa of 0.78.

labeled rows: 20 (human rejected 7)
agent:reviewer@v3    agreement 70%  kappa 0.29 (95% CI -0.18 to 0.69)  missed rejections 4  needless blocks 2
agent:reviewer@v4    agreement 90%  kappa 0.78 (95% CI 0.39 to 1.00)  missed rejections 1  needless blocks 1
kappa gain, candidate minus baseline: 95% CI 0.11 to 0.92
decision: too few labels (20 rows, 7 rejections; need 50 and 10); keep agent:reviewer@v3

Reading v3 and v4 side by side

On the same 20 commits, v4 lifts kappa from 0.29 to 0.78 and cuts missed rejections from 4 to 1. Needless blocks fall from 2 to 1. Missed rejections lead the rule, because each one is a change the reviewer waved through.

Twenty rows cannot settle it, though. The bootstrap resamples the commits 2,000 times and scores both versions on each draw.

V4 gains kappa on almost every draw, with a 95% interval for the gain of 0.11 to 0.92, but with 7 rejections one relabeled commit moves kappa a lot. The script keeps v3 until the set clears its floors.

Choosing a promotion rule

The rule in the script is no more missed rejections than the live version and a kappa gain whose 95% interval sits above zero, on at least 50 labeled rows with at least 10 rejections.

Ties on missed rejections are allowed on purpose. The Tuesday edit aimed to cut needless blocks, and a prompt that does that without missing more rejections raises kappa and should win.

The floors are starting points, not a standard. Raise them for a high-risk repo. The gain interval is the real test: if its lower bound is at or below zero at 50 rows, keep labeling.

Both versions must be scored on the same human-labeled commits. Missing reviews are excluded from both, not just one version. If no common commits remain, or both raters give the same single verdict throughout so kappa is undefined, the script reports insufficient data and keeps the current prompt.

Keep auditing the gate

Human verdicts are labels, and labels err. Reviewers tire, skim large diffs and disagree. We recommend sampling a few gate decisions each month for a second human read, and treating disagreement there as a sign that your labels need attention too.

Measuring reviewer agents with Future AGI Evaluation

In Future AGI Evaluation, the labeled commits become a dataset, an experiment runs the live and candidate reviewer prompts over it, and the Production label points the factory at whichever version passes the promotion rule above.

The labels can arrive two ways. If your gate lives on the Git host, upload the script’s CSV, with each commit’s diff added as a column, since the reviewer prompt needs the diff next to the human label. CSV and JSONL files up to 25 MB are accepted. Load only the random, blind sample.

If people label inside Future AGI instead, a Categorical label in an annotation queue holds approve or request changes, and you can export a review queue to a dataset with no item cap.

Then run an experiment with the live reviewer prompt and the candidate as two configurations on the same model. Every row runs against every configuration, and every eval you attach scores each one.

Have the reviewer prompt end with a single verdict, approve or request_changes. Two custom evals of the Code type, both Pass/fail, can then flag each row: one fails on a missed rejection, the other on a needless block. Their fail counts are the two numbers the rule needs.

Per-row matches only add up to raw agreement, the number that flatters an always-approve reviewer. Kappa needs the whole column, so a per-row kappa means nothing.

Take each configuration’s verdicts, with the human labels, into the scoring script above. It returns kappa, the intervals and the promote-or-hold decision in one pass.

If you use the built-in cohen_kappa metric instead, score the whole column at once. It reports kappa rescaled to 0 to 1 as (kappa + 1) / 2, so v4’s 0.78 shows as 0.89.

Its default pass mark of 0.5 means only no worse than chance: the always-approve reviewer above scores exactly 0.5 and passes.

Choose winner weighs response time and tokens alongside the evals, so promote by the rule rather than by its pick.

In prompt versions and labels, each prompt commit is a numbered version, and Production is a label on one version. Promoting the winner means moving that label.

The core is open source in Future AGI on GitHub under Apache-2.0, with code in ee/ folders under a separate enterprise license.

Catching the next reviewer regression before it merges

The fix for the opening scene is a recording habit. Send a random share of changes to a human who cannot see the agent’s verdict, and log every verdict with the commit and the reviewer prompt version. Score each prompt edit against that sample before it goes live, and keep the current version until a candidate beats it.

Replay the Tuesday edit with that habit in place. The team scores the trimmed prompt against the last month of gate verdicts, sees it miss rejections the old prompt caught, and leaves the old version on Production. The regression shows up as a number before it ships.

Start on your next pull request by recording the verdict, the commit and the prompt version. Run the scoring script once you have 50 labeled commits and 10 rejections, and make “beats the current prompt on the labeled set” the rule for every reviewer change.

If Future AGI is your tool, begin with how prompt versions and labels work.