Let me tell you about the Harness

The same model scored 30.98% in one harness and 46.34% in another.

One model, MiMo-V2.5. Five harnesses. A 15.36-point spread on identical weights. The harness accounts for all of it.

A model produces tokens. A harness produces consequences.

Most of the argument I hear is about models: which one reasons better, which one costs less, which one to standardize on. The thing that decides whether the model's work actually lands sits one layer out, in the loop that runs it, the tools it reaches, the permissions it holds, the memory it keeps, and the checks that catch its mistakes.

That layer has a name, a market, and about 15 points of measurable difference.

Here is how it works, what you can buy today, and how to match one to your work.

What a harness actually is

Let's define 3 important terms.

An agent loop is the cycle where the model decides what to do next, does it, reads the result, and decides again. Ask it to fix a bug: it reads code, runs a test, reads the error, edits, runs again. Every agentic system has this shape.

A tool call is the model asking the surrounding software to act. Read a file. Run a command. Send a message.

The harness is everything around the model that turns text into work that landed. Six parts:

1. The loop that decides what to do next and when to stop
2. The tools the model can reach
3. What it remembers and what it forgets
4. What it does without asking
5. How it learns whether an action worked
6. What survives the process exiting

That is the mechanism, and it is smaller than the marketing around it suggests.

Five of those six are ordinary engineering under new labels: permissions, idempotency, verification, state, blast radius. I have shipped all five before anyone called it a harness, and most of you have too.

The sixth is genuinely new. The component making decisions now carries a theory about its own success. My CI reports what it observed. My agent reports what it believed. Those are different claims, and the difference shows up the first time a shell variable goes empty mid-run.

So the first practical requirement is verification inside the loop, measured against the world rather than the transcript.

The data: one model, six different scores

Let's start with the strongest case.

On the Terminal-Bench 2.0 leaderboard, hold the harness fixed and swap models: the median spread is 33.6 points. Hold the model fixed and swap harnesses: 13.6 points, or 11.2 among frontier models alone. The model is worth roughly 2.5 times the harness.

That number is real, and the model companies are right to like it. It also describes a decision you make a few times a year, while the harness describes one you make every working day.

Now hold the model still and watch the harness move.

EPAM AI/Run ran SWE-bench Verified three times against a pinned checkpoint, claude-3-5-sonnet-20241022. Scaffold v20241029 scored 39.6%. Scaffold v20241212 scored 55.4%. Scaffold v20250219 scored 62.8%. That is 23.2 points of movement on frozen weights, from scaffold work alone.

DataSpace, published August 2026 by HKUST and Tsinghua as the official benchmark of the KDD Cup 2026 data agents competition, fixed MiMo-V2.5 and ran five production harnesses: Grok Build 46.34%, Claude Code 44.63%, DataSpace-Agent 39.27%, Codex 34.88%, Smolagents 30.98%. A 15.36-point spread, across harnesses you can install this afternoon.

Widen the frame. On the swebench board's claude-sonnet-4-20250514 group: Atlassian Rovo Dev and EPAM AI/Run at 76.8%, Tools at 72.4%, SWE-agent at 66.6%, mini-SWE-agent at 64.93%, Artemis Agent v2 at 57.0%. Best to worst, 19.8 points, same model group. I have watched teams spend three weeks debating a model migration while that gap sat available on the model they already pay for.

The same choice moves your invoice. Terminal-Bench 2.1, matched model and matched reasoning effort: Codex scores 83.1% and costs $2,059.19. Terminus 2 scores 78.0% and costs $493.85. Five points of accuracy for 4.2 times the spend.

One caveat belongs here, because it cuts against my own argument. Models absorb harness features. Chain-of-thought was scaffolding. ReAct was scaffolding. Both live in the weights now. mini-SWE-agent makes the point in 100 lines of bash, scoring 76.8% with Claude 4.5 Opus and beating far more elaborate scaffolds (the 64.93% above is that same harness on claude-sonnet-4-20250514). Every line of scaffolding you write today is a bet on current model limits, and those limits move every release.

The harness still deserves your attention, for a reason that has more to do with your calendar than the leaderboards. Model choice is procurement. Harness choice is Tuesday morning, in every prompt, every permission, every approval. Ten to twenty points sit within reach on the subscription you already pay for.

What the leaderboards measure, and what they leave out

Every number above measures one thing: whether an agent completes a scored task. That is a real measurement with narrow edges.

Four failures live outside those edges.

Instruction fidelity against a wrong spec. The a16z researchers watched an agent pass a spec test by writing a 2,900-line "compiler" that memorized the test cases. Their words: "The loop converged, but just on the verifier, not the user's intent".

Credential handling. I told an agent to delete a private key. It reported success. The key stayed on disk another ten minutes, because a shell variable had gone empty across an interrupted call, so the cleanup ran against nothing and exited zero.

State loss at approval boundaries. The agent pauses for permission, you grant it, and the work resumes with half its context gone.

Failure awareness. Whether the agent can distinguish an action that worked from a command that returned quietly.

You can test all four this week with one experiment. Swap the model, change nothing else, run the same work. The failures that survive belong to the harness. Leaderboards tell you whether your agent can do the work; your own logs tell you what happens when it does the wrong work perfectly.

The market in August 2026

More harnesses ship today than the leaderboards track, and they sort into six kinds. I run most of them, and I have broken something in each. Learn the kinds first, then the names.

The CLI coding agents. The crowded aisle: Claude Code from Anthropic, the Codex CLI from OpenAI, OpenCode (MIT), Cline (Apache 2.0), Aider (Apache 2.0), Google's Gemini CLI, and Cursor's CLI. Point one at a repository and it works there. These are the most mature and most benchmarked, which is exactly why the spreads above matter so much. The real difference between them is the permission model: Codex ships three built-in profiles from read-only to full access, and Claude Code sandboxes with Seatbelt or bubblewrap. Read the sandbox documentation before you treat either as a hard boundary. On cost this aisle splits three ways: Claude Code and Cursor are closed subscriptions, Codex and Gemini are open CLIs that bill per token, and OpenCode, Cline, and Aider are free.

The autonomous agents. Devin from Cognition, OpenHands (now branded Agent Canvas), and Cursor's background agents. They take a task and work for an hour without you. The trade is structural: a loop that survives unattended plays conservatively, and conservative play costs score. Your attention is the variable. I send one-off chores here and keep credentials out of reach. Devin is paid and closed; OpenHands is MIT and free.

The frameworks. CrewAI, AutoGen, LangGraph, Google ADK. You assemble the loop yourself from agents, tools, handoffs, and memory. Multi-agent systems live here, and so do a lot of lost quarters. A framework is a kit: you build the harness from it, then you maintain it forever. Reach for this aisle when the shipping harnesses genuinely cannot do the job. Every name here is open source.

The research harnesses. mini-SWE-agent, a hundred lines of bash that beats bespoke scaffolds, plus Terminus and Meta-Harness from Stanford IRIS. These are instruments rather than products, built to isolate one variable at a time, and they keep the commercial harnesses honest. All open source.

The open-source workspaces. Buzz from Block, Apache 2.0, launched July 2026: agents hold their own identities, keys, and audit trails, working beside humans in shared rooms with every action signed to a decentralized log. QM from Y Combinator, MIT, open-sourced July 31 2026: the multi-agent harness YC runs its own accounting, legal, events, and engineering on, cloud-first with Slack and web interfaces. OpenClaw: a local-first personal harness with a heartbeat loop, running on your own hardware, now organized as a nonprofit foundation. The shape is newer than some of the names: agents treated as durable infrastructure instead of sessions. All three are free and self-hostable.

The personal automation harnesses. Hermes Agent is the open-source one I run my operations on: scheduled jobs, subagent delegation, tool access, and delivery to my phone. Its job is to keep working after I close the laptop. If your work lives outside a repository, start shopping here.

Six kinds, a handful of names each. The whole market fits on one page.

The licenses tell you who is selling what. The subscription harnesses are the closed ones: Claude Code, Cursor, Devin. You rent them monthly and the spec inside stays private. Everything else runs free or per-token: Codex and Gemini bill usage on open CLIs, OpenCode and OpenHands are free, the frameworks are free, and Buzz, QM, OpenClaw, Hermes, and the research harnesses are free and self-hostable. You rent the model either way. The harness is available to own.

Which harness for which workload

Here is the assignment, workload by workload. I run all seven concurrently. One rule governs every entry: match the harness to the worst thing that can happen when it gets something wrong.

Coding: one CLI agent, learned deeply.

Claude Code, the Codex CLI, Cursor. Pick one, learn its permission model cold, and run it read-only until the plan is on the table. Most of a coding day is reversible work, and a read-only profile makes the irreversible part visible before it happens.

Data and analytics: the agent with a shell.

DataSpace proved this case: the same model swings 15 points depending on which agent runs the task. CSVs, SQLite, PDFs, joins across artifacts. That is CLI-agent territory with the tools mounted, or a framework like Smolagents when the pipeline is yours to build. Mount the tools, keep a deterministic evaluator, and verify the table that comes back. When the deliverable is the decision on top of the table, that becomes a decision-intelligence problem, and Syntaxia is the harness I reach for there.

Imaging and media: workflow software.

Image generation is a pipeline problem: model, prompt, post-processing, review. Node-based tools like ComfyUI fit the shape of the work better than an agent loop does, because the failure mode is a bad render that costs you a retry. Use the tool built for the medium.

Legal and compliance: the open harness with an audit trail.

Contracts, discovery, regulatory filings. The record matters as much as the output. Requirements: a spec you can read and version, permissions that stop short of signing authority, a citation on every claim, and human sign-off before anything commits. A hosted agent with a private system prompt fails the first requirement, which settles the question.

Operations: the harness that outlasts the session.

Business operations run on a schedule: invoicing, client status, meeting notes, partner compliance. That work belongs in a persistence-first harness with cron jobs, file artifacts, and a delivery channel that reaches your phone. Personal automation harnesses like Hermes Agent cover exactly this. If the work has to happen while your laptop is closed, the harness has to run while your laptop is closed.

Research: read widely, write it down.

Web search, papers, competitor moves. The deliverable is a notes file that outlives the session. Any harness with search and file access will do; the discipline is writing each finding to disk while the context window still holds it.

Writing: keep the human in the seat.

Writing stays manual here. Voice is the product, and the judgment that produces it is the work itself.

One rule covers all seven: route by blast radius. Cheap and reversible work goes to the cheap loop. Keys, money, contracts, and production go to the harness with the real permission profile, even at 4.2 times the spend.

If your harness arrived by way of a platform team or an IDE mandate, run the seven anyway. The output becomes a list of what you compensate for outside the harness, which is worth having either way.

Where to start

Model selection is procurement. Pick the strongest one your budget allows, then revisit it next quarter.

Harness selection is daily practice, and that is where the ten to twenty points sit. Pick one this week from the aisle that matches your work. Learn its permission model well enough to predict what it will do before you press enter. Give it the narrowest access that still finishes the job. Then require proof of every action that matters: the file on disk, the exit code, the row count.

Your agent will tell you the task succeeded. Your harness is what finds out.