Confounded by Default

A task fails at two in the morning. A container exits non-zero, a scan times out, a tool writes nothing to its output. Cazi, the orchestrator that ran it, notices and writes it down. It has done this for a long time: every failed run becomes a learning, a short note that later runs read before they repeat the same mistake.

For most of that time the notes carried a quiet lie. A learning recorded what broke, and once someone filled it in, a cause and a fix. What it never recorded was how much any of that was actually known. A guess and a proven remedy looked exactly the same on the page. The library of lessons grew, and some of what it held was folklore wearing the clothes of fact.

Then we read harness², a framework from Google Research. Its subject is harnessing: making an agent better without touching the model, by improving the system around it — the prompts and tools and memory and the lessons it carries forward. The machinery is clever. But the part that stayed with us was the discipline.

The loop we already had

harness² keeps a library of lessons that past work writes and future work reads, under a strict rule about what may go in. Cazi already had the shape of this. Its learnings are that library; failed runs write them, and a run that matches an old pattern reads the advice before it starts. We did not need to build the loop. We needed to make it honest.

What a failure proves

Here is the whole problem in one line. A failed run proves that something went wrong. It does not prove what went wrong, and it certainly does not prove that the fix a person typed into the note an hour later is the cure. Treating a capture as a solved case is how a library fills with confident advice that was never tested, then matches that advice against future work as if it were settled law.

A failed run proves one thing. That something went wrong. Not what, and not the cure.

So we gave every learning two new things, borrowed straight from harness². The first is a cue: the shape of the situation that should bring the lesson to mind next time, written as a shape and not as this one task’s facts. Not “subfinder failed on acme.com at 14:03,” but “a scanner pinned to :latest with no input list.” A note with no cue is a note about one task. A note with a cue might transfer, which is the only reason to keep it.

The second is a verdict, and it is where the honesty lives. It says how far the evidence actually goes:

• helped — a later run showed the fix worked
• hurt — the change made things worse
• no effect — no observed difference
• confounded — something went wrong, but the record does not show what fixes it (the default)

A fresh capture starts at confounded. Not helped, not even no effect, but the honest shrug: we know a thing broke, and we do not yet know the cure. helped has to be earned, by a later run that actually shows the fix working. Above both fields sits one rule: cite a concrete run, a task, a line of output, or write nothing at all. harness² puts the reason sharply — an invented attribution poisons the library for every later task.

The loop we didn’t build

harness² has a second, inner loop that we left on the page. Its idea is to refine the harness for the task in hand: run the same task several ways, and compose a better approach out of the contrasts between the runs. In Cazi that would be a candidate sweep, a handful of approaches raced against one job.

We designed it and did not switch it on. Probing means running real executions, and Cazi’s executions spend a budget, start containers, and reach out to live targets, sometimes the same one several times over. That is not a default you flip without a cost and a safety review behind it. So the inner loop is specified, documented, and waiting for that review rather than quietly running.

What actually changed

Not much, by the diff. Two nullable columns, a reworked instruction to the model that writes the notes, and a verdict that starts at confounded. But it changed what the system is allowed to believe about itself. A lesson now has to state how far its evidence reaches, or admit that it does not know. For a library that future work leans on, being allowed to say “I’m not sure” turns out to be most of the point.

---

Cazi is a deterministic task orchestrator: it runs tools as containers and lets AI agents commission that work. harness² is open source from Google Research, under Apache 2.0; the quoted lines are from its documentation.