Human in the Loop AI: Designing Review Queues That Improve LLM Quality

Your team added a human review step three months ago. Reviewers approve almost everything that reaches them, the dashboard glows green, and the queue feels like progress. Then someone checks output quality against where it started, and the numbers have not moved at all.

That review step is human in the loop AI, and on its own it changes nothing. A queue only improves quality when it routes the right work to people, defines what a good judgment is, and checks that reviewers agree. Approval rates feel like signal and rarely are.

This is a build guide for that queue, not a definitions page. We cover the vocabulary once, in the next section, and spend the rest on decisions. What to send to a human, how to write a rubric two people can agree on, how to measure that agreement, and how to prove the queue worked.

The parts most guides skip are the measurable ones. Anyone can say send the hard cases to a person. Far fewer show you the agreement number that tells you whether those people judge consistently, or the before and after that tells you the queue was worth its cost.

What Human in the Loop AI Means Once the Model Is in Production

Three postures get grouped under the same phrase, and they are not interchangeable. Human in the loop blocks the action until a person approves it. Human on the loop lets the system act while a person watches and can step in. Human in command keeps final authority over the system as a whole with a person, not a single output.

The choice is a product decision before it is an ops decision. Blocking review adds latency to every action it touches and needs staff on the critical path. Monitoring after the fact is cheaper and faster, and it accepts that some bad output reaches users before anyone catches it. Pick the posture per action, not per system.

A review queue does two jobs, and mixing them causes half the confusion in this area. The first is catching bad output before a user sees it, a safety job measured in incidents avoided. The second is producing labelled data that improves the system, a learning job measured in later quality. Keep them separate.

The two jobs pull in different directions. Safety wants the riskiest actions reviewed now, synchronously, whatever the cost to speed. Learning wants a representative spread of cases, including the ordinary ones, so the labels teach you something general. A queue built for only one of them quietly fails at the other.

Most teams discover this the slow way. They build a blocking queue for risk, fill it with scary edge cases, and end up with labels that describe the tail and say nothing about the median experience. The fix is to design routing for both jobs at once, which the next section does.

What Belongs in the Review Queue

Most implementations go wrong at the routing decision, either reviewing everything until reviewers burn out, or reviewing at random until the queue catches nothing that matters. The useful middle sends a human the cases where human judgment actually changes the outcome. That means routing on a small set of explicit triggers rather than a vibe.

Six triggers cover most systems. High business risk earns a human regardless of model confidence. Low model confidence or a low eval score earns one because the system is already unsure. Disagreement between two automated scorers earns one because something is genuinely ambiguous. User reported problems and novel inputs unlike anything in your golden set round out the set.

Several of those triggers come straight off your dashboards. Low scores, user complaints, and strange inputs are exactly what production LLM observability is meant to surface, so the queue can subscribe to signals you already collect rather than inventing new ones. The routing layer is where monitoring turns into action.

One trigger is non negotiable and usually missing: a random sample of everything else. Without it you only ever see the cases your other triggers already suspected, so your quality estimate is biased by construction. The random draw is the only part of the queue that can tell you about failures you did not predict.

The table below sets out each trigger, whether it blocks the action or runs asynchronously, and the reason it earns a person. Treat blocking as the exception, reserved for the irreversible and the expensive, and let everything else flow through an async lane.

Writing a Rubric Two Reviewers Can Agree On

Before you measure agreement, you have to make agreement possible, and that is a property of the rubric long before it is a property of the reviewers. Two careful people will still diverge if the question they are answering is vague. Fix the rubric and most disagreement disappears on its own.

A measurable rubric has five properties. One dimension per question, so a score means one thing. A small ordinal or categorical scale, not a raw zero to a hundred. A written definition of each level, with at least one worked example per level, and an explicit rule for when the input itself is broken.

The failures are as consistent as the fixes. Compound questions that ask two things at once, so a single score cannot answer them. Unbounded scales where one reviewer’s seven is another’s nine. Subjective adjectives like helpful or fluent with nothing to anchor them, and no instruction for the broken input, so everyone improvises differently.

Treat the first draft of a rubric as a hypothesis. Run it on a small shared set, measure agreement, and read every case where two reviewers split. Rewrite the definitions that caused the split, then run it again. The queue opens only after the calibration set holds together, never before the first pass.

This section looks like the boring one and quietly decides everything downstream. The agreement scores in the next section are only as meaningful as the rubric behind them, and a judge you calibrate against human labels inherits the rubric’s clarity or its mess. Time spent here pays back more than any other hour in the whole build.

Measuring Whether Your Reviewers Agree

Start with the number most teams reach for and should not stop at: raw percent agreement. Two reviewers who both approve almost everything will agree most of the time by chance alone, so a high percentage can describe two people rubber stamping in parallel. You need a measure that subtracts chance.

That measure is kappa. Cohen’s kappa handles two raters and Fleiss’ kappa extends the same idea to more than two, both comparing observed agreement against what chance would produce. A kappa near zero means your reviewers agree no better than coin flips. The further above zero it sits, the more the agreement is real.

Interpreting kappa is where most articles go wrong. The quoted bands, where 0.61 to 0.80 reads as substantial and 0.41 to 0.60 as moderate, are what McHugh attributes to Cohen and then rejects, warning they let too little agreement count as substantial.

Landis and Koch are often credited for them, but McHugh points to Cohen.

McHugh’s own table is stricter, and it is the one worth adopting. She reads .40 to .59 as weak, .60 to .79 as moderate, .80 to .90 as strong, and only above .90 as almost perfect, with everything under .40 minimal or none. Under her scale, a lot of published agreement would not clear the bar for use.

For a live queue, Krippendorff’s alpha fits the shape of the problem better than kappa.

It handles any number of observers, any measurement level, and incomplete or missing data, with no minimum sample size, which matches a queue where different reviewers see different items. Krippendorff suggested requiring alpha at or above .800, with .667 where only tentative conclusions are acceptable.

He paired those figures with a warning that there are no magical numbers, and the warning matters. A threshold is a decision you make for your risk tolerance, not a law of statistics. Write it down before the queue opens, so the number cannot be quietly negotiated after the results are in.

One practitioner caveat saves a common mistake: the unit of agreement must be chosen on purpose. Take span annotation, where most characters sit outside any labelled span. Counting no annotation as its own class artificially inflates the agreement, because the reviewers are in agreement on not annotating most of the text.

Designing the Queue Mechanics

With routing and rubric settled, the queue itself has settings that decide whether it produces trustworthy numbers. How many independent annotations each item needs. How items get assigned to reviewers. How long an item stays reserved before it returns to the pool. Whether a second pass reviews the first, and what written guidance reviewers see while they work.

The setting with real cost is overlap. More annotations per item buys you a measurable agreement number and costs throughput, since every extra review is time not spent on new work. The sensible pattern is a higher overlap on a calibration subset, where you actually need the agreement figure, and single annotation on the steady state.

The tools differ in ways worth knowing before you pick one. Langfuse annotation queues need a score configuration defined before use, and its documented uses include aligning an LLM-as-a-judge evaluation with human annotation. Argilla is Apache-2.0 but in maintenance mode, with its maintainers saying they will not add new features. Braintrust stores each reviewer’s scores as dedicated review spans.

That Braintrust detail matters more than it looks. Storing scores per reviewer means the raw material for agreement is there, even though the tool will not compute the number for you. Label Studio sits in the same build or buy spot, open source under Apache-2.0 with its agreement statistics reserved for the Enterprise tier.

For agent workflows that need a blocking pause, LangGraph’s interrupt() stops execution and Command(resume=…) continues it, given a checkpointer and thread identifier. The documented patterns cover approve or reject, editing state, interrupts inside tools, and multiple interrupts. One rule prevents a nasty bug: any side effect before an interrupt must be idempotent, since that code can run twice.

The table pairs each setting with a typical choice, what it protects against, and the cost of getting it wrong. Read the last column as the reason each setting exists, and set every row on purpose before the first reviewer logs in.

Turning Human Labels Into Something the System Learns From

A review queue that only catches bad output is a filter. A queue that feeds what it learns back into the system is a quality programme, and the difference is entirely in what happens to the labels after review. Four uses, in rising order of effort, cover most of the value.

First, fix the individual output the reviewer caught, the immediate save. Second, add the case to the golden evaluation set so the same failure is scored automatically from then on. Third, calibrate the automated judge against the human scores. Fourth, revise the prompt, retrieval, or policy behind the failure class, which retires it at the source.

Judge calibration is the highest leverage of the four. Your human labels are the reference an automated scorer is measured against, so agreement between judge and human is the number that says whether the judge can run unsupervised. This is where a review queue connects to your wider LLM evaluation work rather than sitting beside it.

Order the uses by leverage, not by ease. Fixing individual outputs feels productive and scales badly, since you are patching symptoms one at a time. The compounding wins are the golden set and the calibrated judge, because both keep working on every future request without another minute of human time. Spend the review budget accordingly.

The failure mode that wastes the whole effort is simple: labels that land in a spreadsheet nobody opens again. The fix is a rule you set before the queue opens, that every label has a named destination, a dataset, a metric, or a ticket. A label with nowhere to go should not be collected in the first place.

How Do You Prove Human in the Loop AI Improved Quality?

Here is the question a manager will eventually ask, and most queues cannot answer it: did the review step actually improve anything? Answering needs a design set up before the queue runs, because you cannot reconstruct a baseline after the fact. Fix the rubric and the golden set first, then measure.

The design is a before and after on a held out sample. Score a fixed, representative sample with the rubric and golden set in place, and keep that as the baseline. Let the queue run for a set period. Then score the same sample again, using the random baseline draw from routing as the unbiased read.

Three numbers are worth tracking over that window. The rate at which reviewers change the output, which tells you whether the queue is catching anything. The movement in the golden set score, which tells you whether quality is rising. And the agreement between the automated judge and the humans, which tells you whether the cheap scorer is trustworthy.

One honest caveat keeps the claim credible. A prompt change, a model version bump, and a retrieval tweak all move the same numbers the queue does, so an uncontrolled before and after cannot prove the queue caused the gain. A change log, kept alongside the scores, is part of the measurement design, not paperwork.

Proof here is modest by design. You are showing a defensible before and after with the confounders written down, short of a randomized trial. That is enough to justify the queue’s cost to a reasonable manager, and far more than the approval rate everyone quotes and no one should lean on.

Running Review Queues with Human Feedback in Future AGI

Everything above needs a place to actually run. Future AGI Annotations is built for this exact job: human-in-the-loop feedback with queues and an analytics view that reports reviewer agreement. It works over three primitives, Labels for the judgments, Queues for the work, and Scores for the results.

The labels match the rubric shapes from earlier. You can define Categorical, Numeric, Text, Star Rating, or Thumbs Up or Down labels, and attach them to the units you actually work with: traces, spans, sessions, and dataset rows. That range means the same queue can score a whole agent session or a single retrieved chunk without leaving the tool.

The queue settings map onto the mechanics section almost one to one.

Annotations required per item run from 1 to 10, assignment is Manual, Round Robin, or Load Balanced, reservation timeout offers presets from 15 minutes to 4 hours, review is an optional pass, and a markdown guidelines panel keeps the rubric visible. Those are the knobs Table 2 argued for.

The analytics view is where it pays off for this post’s hardest requirement. It reports completion rate, throughput over 30 days, annotator performance, and label distribution, alongside the inter-annotator agreement itself, computed with Cohen’s and Fleiss’ kappa. The agreement number the whole post has been building toward is a panel, not a script you maintain.

One gap is worth naming rather than leaving for you to discover. This post argued that Krippendorff’s alpha fits a production queue best, because it tolerates missing data and unequal reviewer coverage. The analytics view computes Cohen’s and Fleiss’ kappa, not alpha. For a queue where most items are seen by one reviewer and only a calibration subset is double-reviewed, kappa on that subset is the right read anyway. If your queue is sparser than that, export the labels and compute alpha yourself.

Where Human in the Loop AI Earns Its Cost

Go back to the team whose reviewers approved everything. Nothing was wrong with the reviewers. The rubric never gave them a reason to disagree, so approval was the path of least resistance, and a queue that cannot produce disagreement cannot produce learning either. The fix sat upstream of the people the whole time.

A human in the loop AI queue earns its cost when it routes selectively, measures whether reviewers agree, and sends every label somewhere specific. The smallest version that does real work stays small: one rubric dimension, fifty overlapping items, one agreement number, and one named destination for the labels.

From there you scale the parts that earn it. If you would rather run the queue and read the agreement analytics in one place instead of wiring it together yourself, that is the shape Future AGI Annotations is built to hold.