Open-Source AI Agent Architecture: An Agent Harness Build

A hand-rolled agent harness usually starts life as a while-loop around a model call, and it works fine for six turns. Then it runs long enough to fill a context window, loses track of what it already tried, and nobody can tell whether the model or the harness caused the failure.

This piece picks up where Build a Minimal Agent Harness in Python leaves off. That guide builds the smallest runnable loop on purpose, skipping memory, tool registries, and subagents to keep the code readable. This post builds those production layers, using three real open-source harnesses (Hermes from NousResearch, OpenHands, SWE-agent) as evidence instead of a single toy example.

Every Hermes-specific implementation detail below is sourced from Arize’s teardown of Hermes’s harness design, which is cited at each point of use and quoted directly where the wording is theirs.

What Is an Agent Harness Architecture?

An agent harness is the code layer between a raw model call and a finished agent: the loop, tool dispatcher, memory manager, and control-flow rules. It is not the prompt, and it is not the model. It is everything that decides whether a model’s proposed action actually runs.

A harness is not the same thing as a framework like LangGraph or CrewAI, and it is not a protocol like MCP. For how a harness relates to MCP specifically, see Agent Harness vs MCP rather than re-deriving that distinction here.

Every working harness owns four jobs, and this article is structured around them: control flow, tool calling, memory and context, and state and session. Real open-source projects (Hermes from NousResearch, OpenHands, SWE-agent) implement all four in different ways, and each shows up as evidence in the sections below.

Those four jobs also give you a checklist for judging any harness, whether you’re building one or evaluating an existing project before you adopt it. A harness that’s thin on any one of the four usually fails in a predictable way.

A weak control loop runs past the point of usefulness. A missing tool registry lets a subagent see tools it shouldn’t. A harness with no durable session store loses a multi-hour run the moment the process restarts.

The Core Loop: Control Flow Before Anything Else

The harness’s non-negotiable spine is a loop: call the model, dispatch whatever tools it asked for, append the results, repeat. Arize’s teardown describes Hermes as running that same cycle until the model returns a final response or an interrupt stops it. Teams that add memory or subagents before this loop is solid end up debugging two problems at once.

Stop conditions matter as much as the loop itself. A harness needs a model-declared completion signal, a hard turn ceiling, and an explicit timeout or abort path. Skip any of these and you get runaway loops that burn API budget for hours before anyone notices.

Parallel tool calls need careful handling too. A single model turn can request several tool calls at once, and the harness has to dispatch all of them and map each result back to the correct call ID, in order. Get the mapping wrong and the model starts reasoning from results attached to the wrong action.

Where Teams Cut Corners

The most common shortcuts show up early and compound later. No turn ceiling means a confused model can loop until the process is killed manually. Silently swallowing tool errors means the model reasons from a false “success” and builds an entire plan on bad information. Skipping argument validation before dispatch means malformed calls reach application code instead of failing fast with a clear message.

Registering a Tool vs. Exposing It to a Run

Tool registration and tool exposure are two different jobs, and conflating them is a common design mistake. Registration means a tool’s schema and handler function exist somewhere in the codebase. Exposure decides which of those registered tools a given run can actually see and call.

Hermes keeps the two apart. Per Arize’s teardown, “tools register into a central registry at import time” while “a separate toolset layer decides what the model actually sees in a given run.”

That separation is what lets one codebase serve many agent personas without duplicating code. A support agent and an ops agent can share the same registry while seeing completely different toolsets.

Schema validation before dispatch is one of the cheapest reliability wins available. Checking a tool call’s arguments against its schema before it touches application code catches malformed calls early, though the exact error-catch rate depends heavily on how sloppy your model’s tool calls tend to be, and no single, verifiable industry figure covers that broadly enough to state as fact.

Scoping tools down for delegated or child runs follows the same least-privilege logic teams already apply to service accounts. Arize reports that the toolset Hermes exposes narrows by platform and by scenario, and narrows again when a run is delegated to a child. A subagent spawned to search documentation has no reason to see a file-delete tool, and a harness that enforces that by default catches a whole category of mistakes before they happen.

Designing a Tool Registry

A minimum viable tool registry needs five fields per entry: a name, a JSON schema for its arguments, a handler function, scope tags describing which run types can see it, and a danger flag for actions that need extra confirmation or logging before they execute.

Memory and Context Management

Raw message history does not scale. Every additional turn adds tokens, cost climbs linearly with session length, and long contexts measurably degrade model performance well before you hit a hard context-window limit.

The fix most production harnesses converge on is compaction, not truncation. Hermes summarizes older turns with an auxiliary model call while protecting the most recent turns verbatim, so the model never loses immediate context. Arize puts the numbers on it: the summary budget scales with the compressed content at “roughly twenty percent, with a two-thousand-token floor and a twelve-thousand-token ceiling.”

Compaction is a session lifecycle event, not a silent rewrite. In Hermes, per Arize’s account, compaction closes the current session, opens a child session seeded with the summary, issues a new session ID, and writes down which session the new one came from. Old history is never destroyed; it is chained off into a session a harness can walk back to.

Short-term memory and long-term memory solve different problems and usually need different stores. Short-term memory is the current run’s live state: the goal, recent tool results, what’s already been tried this session. Long-term memory is cross-session facts, typically kept in a separate store keyed by user or tenant, and pulled in only when relevant.

Rather than dumping full history back into context, Hermes ships a tool the model itself can call to search prior sessions and pull back only the fragments that matter, described by Arize as “a model-facing session_search tool for focused recall over prior sessions.” That pattern, giving the model controlled recall instead of forcing it to re-read everything, holds up regardless of which harness you build on.

Where Session State Actually Lives

Sessions need a durable backing store, not just in-memory state that vanishes on crash. Arize reports that Hermes keeps session state in SQLite, using FTS5 for search and WAL journaling for durability, with a fallback path for filesystems where WAL coordination isn’t available. That durability buys three things: crash recovery, the ability to resume a multi-hour agent run where it left off, and an audit trail you can actually query.

Parent-child run lineage tracking, the same mechanism compaction uses to chain sessions together, is also the foundation subagent delegation depends on. Without a lineage record, there’s no reliable way to trace which child run belongs to which parent task once a session has been rotated a few times.

A harness starts to feel like one coherent system, rather than a pile of disconnected scripts, once multiple entry points share the same session plane. Hermes is built for that: a command line, a chat platform, and a cron job can all read and write the same session state instead of each keeping a private copy.

What Breaks When a Parent Run Ends First

The delegation pattern is simple to describe: a parent run spawns a child run with its own scoped toolset and its own context budget, and the child returns a summarized result to the parent rather than dumping its full transcript back up. In Hermes, each child is issued its own task identifier and works in its own terminal context.

Lifecycle ownership is the failure mode that matters most here. Most open-source harnesses, Hermes included, run child work inside the parent’s call stack. As Arize puts it, “when the parent is done, the child is done.” Their analysis frames the missing piece as “a durable, externally steerable child-run plane with independent control semantics,” and treats “promoting child runs to first-class control-plane objects” as the next architectural step rather than a solved problem. That framing is Arize’s read on the gap, not a statement from Hermes’s own team, though it is consistent with how Hermes’s session model works today.

This matters more than it sounds like on paper. A subagent researching a long-running task can lose its work entirely if the parent process crashes, restarts for a deploy, or simply finishes before the child does. Teams building on any of today’s open-source harnesses should treat that as a known constraint, not a bug to file against the project.

Recursion depth capping is the practical guardrail every harness needs regardless of how sophisticated the delegation logic gets. A child that spawns a child that spawns a child, with no ceiling, is a runaway loop wearing a different costume.

Tool Calling and Orchestration Frameworks Compared

Every builder eventually faces the same fork: hand-roll the loop, or adopt an orchestration framework that already handles tool-call plumbing. Hand-rolled gives full control over memory and session design but costs real build time. Frameworks give you a working loop faster but impose their own state model on top of it.

No single row wins outright. A framework saves weeks of plumbing work; a full custom harness costs those weeks back but removes the constraint of someone else’s state model.

Build vs. Buy: When to Roll Your Own Harness

The decision comes down to four real signals, not a preference for control. Custom tool permission models, durable subagent needs, compliance or data-residency constraints, and available engineering bandwidth all point in different directions depending on the team.

Forking an existing harness (Claude Code, OpenHands, Hermes) is the cheaper default; building the loop from zero is the exception you should have to justify. For a full comparison of named, currently maintained harnesses and which task each one fits, see Best Agent Harnesses 2026: A Builder’s Field Guide instead of re-listing options here.

OpenHands is a useful reference point for this decision. It’s MIT-licensed, has more than 84,000 GitHub stars as of August 2026, and runs as an event-driven platform where agents edit files, execute shell commands, and browse the web inside sandboxed containers, with reproducible baselines on SWE-bench.

SWE-agent, built by researchers at Princeton, takes a narrower approach: a single-agent framework built specifically for software engineering, using what it calls an agent-computer interface to navigate codebases, edit files, and run tests. Both are real projects a team can fork today rather than reasons to build from scratch.

Instrumenting Your Harness for Evaluation and Observability with Future AGI

A hand-built harness has no visibility into tool-call correctness, compaction quality, or subagent failures unless someone adds it. This is the part builders skip, usually until the first production incident forces the question.

traceAI, Future AGI’s open-source OpenTelemetry tracing library, turns a custom loop into a span tree: each model call, tool dispatch, and compaction event becomes a span instead of a line in an unstructured log file. Those spans land in Observe, where the span graph, latency, and token cost sit on one timeline, so the call-dispatch-compact sequence for a session reads in order rather than being reassembled from separate log lines.

Evaluate scores the traced sessions. Tool Call Accuracy is the relevant metric for a harness: it is code-based rather than judge-based, greedy-matching each actual call to the best unused expected call, scoring 1.0 for an exact name-and-arguments match and 0.5 for a name-only match, then dividing by max(expected, actual). Trajectory Match scores the step sequence itself in strict, unordered, subset, or superset mode. Both catch a silently failed tool call sitting underneath an otherwise fluent model response, which a text-quality check never would.

The Error Feed covers the failure modes this article describes. It reads a sample of production traces, decides unaided what failed, and groups traces sharing the same failure into one issue, so a compaction event that dropped something important or a subagent that died with its parent surfaces as a named issue rather than as a log-grep exercise.

Which Layer Should You Build First?

Build in the order this article covers them: control flow first, tool scoping second, memory third, session persistence fourth. Skipping ahead to subagents or memory compaction before the core loop has explicit stop conditions just moves the debugging problem one layer deeper without actually solving it.

Durable subagent control is the frontier problem here. Every harness examined, Hermes included, still runs child work inside the parent’s call stack rather than as an independently steerable process. That’s not a bug so much as an open research question the field hasn’t solved yet.

Before shipping a custom harness to production, instrument it. A harness with no tracing and no tool-use evaluation is a harness nobody can debug once it starts failing in ways a demo never surfaced, and by the time that happens, guessing which layer broke costs far more than the instrumentation would have.