Updated July 14, 2026. New research now backs what was previously anecdotal, but also suggests a refinement to the three model review panel.

Over the past year I have shipped around twenty major revisions modernizing a 25-year-old, 550,000-line codebase, refining an automated multi-agent/multi-LLM workflow that has greatly reduced AI coding errors. It’s impossible to quantify, but my unscientific impression is I’ve made more changes to this product in the past 12 months than previous teams of four could have achieved over several years. With AI-driven coding everyone is finding their own development process, but I’m convinced that this unique combination of spec-driven project control, multiple agent/multiple LLM collaboration, and hand-curated testing (or something similar) will become the norm for AI-driven software development.

The human decides intent, picks the hard test cases, and signs off on what ships. Everything in between is automated, and the review and build stages fan out across multiple models and agents.

The workflow has three aspects: spec-driven planning, multi-agent/multi-LLM review, and hand-curated tests with golden datasets. Each aspect catches a class of failure the other two can’t. Everything that follows is a defense of why you need all three.

Three independent guarantees. Skip one and the other two can still let the wrong answer through.

Spec-Driven Planning

The biggest mistake you can make with AI code generation is under-specifying: the model will happily build the wrong thing very fast. So you write down what the system does, how it does it, and how you’ll prove it worked, before any code is generated. With agile entrenched almost everywhere that sounds quaint, but this is not the old waterfall argument; it’s a response to speed. When a complex piece of code that used to take two weeks arrives in a matter of hours, the deciding has to happen before the typing. If you’re vibe coding a prototype, skip all this and see what the model comes up with. Production code needs a process.

For major projects, agentic coding flips your job: you’re less a coder and more a domain expert with an architect’s instincts and a testing manager’s paranoia. When coding is cheap and fast, you spend much less time sweating the details of a sort and more time worrying about everything else. This still requires all of the skills developed over years of experience in the software industry, it’s just those skills are everything else but how to solve LeetCode problems. For small, surgical bug fixes, then the amount of planning you need to do is greatly reduced, just like with non-AI programming. You match the amount of specification to the complexity of the task.

Spec-driven development tools like GitHub’s Spec Kit structure the up-front deciding into three layers:

1. Specification. What the system does.
2. Plan. How it does it.
3. Tasks. The individual work items that carry out the plan.

There are lots of other planning tools, and personal preference decides which one you pick. The important thing is that by the time a model is generating code, the hard problems have already been decided. Each task is small, self-contained, and doesn’t need the entire codebase in context to complete. Failure risk collapses because the scope of any one completion is a function or a class, not an app.

The mechanism behind small tasks is the same one behind the review jurisdictions later in this post: models measurably degrade as their input grows, so every token the writer doesn’t need is a tax on the tokens it does. The planning artifacts also do at planning time what runtime scaffolding recovers only expensively later: they separate discovery from writing. By the time implementation starts, the plan has already answered where the change goes and what it touches, so the writer writes instead of exploring. Getting those boundaries right is exactly why the task list goes through the same review panel as the code.

The most important part of this process is that it’s not done by a single agent, and gates aren’t optional. My constitution makes external review mandatory at four points, planning, architecture, code, and pre-commit, and each one routes out to a multi-agent/multi-LLM panel. The spec, the plan, and the task list go out for review before any of them advances, not just the finished code.

Automated Multi-Agent/Multi-LLM Review

No single model can be trusted to review its own work. If you use the same model to do a code review on what another agent just wrote, you’ll get AI-model bias: an LLM naturally favors the code it would have written. Every time I’ve submitted a specification, plan, or code to another LLM for review it has found major errors in logic, strategy or consistency. Fixing these problems up front is always faster than chasing a cascading flow of software bugs, only to discover it would have been impossible to make the program work as planned. In a way, this is no different than the process of having multiple people reviewing specs and code design up front. It’s simply a long-held practice with AI in the loop rather than a team of peers.

The other piece of the puzzle is automation. I’ve modified GitHub SpecKit using its hooks to automatically create multiple agents with multiple LLMs, each reviewing every single step of the development process, from spec to plan to task list to actual code. In my setup the router is PAL, a small MCP server that fires the panel at each Spec Kit gate, so the reviews cost me zero keystrokes. Each step invariably catches major problems that would have been impossible just using Claude to audit itself, and automatically so that a human only need get involved if there’s a major breakdown somewhere.

There is hard evidence to support this: on benchmark submissions every frontier model solves roughly the same set of easy problems, but on the hard ones, each model gets a different subset right. The overlap in failures is low, and that’s not a hunch. Organizations have to publish per-task results when they upload benchmark scores, which means you can compute the oracle: the score you would get if you always picked the best model’s answer for each problem. Across 39 models on SWE-Bench Verified, the oracle lifts the best individual model from 76.8% to 90.2%. On the harder SWE-Bench Pro, across nine models, it goes from 43.8% to 64.8%. The gains concentrate where they matter most. On the hardest SWE-Bench Pro problems, where the best individual model again scores 43.8%, a three-model oracle lifts that to 56.9%.

Harder benchmarks show larger oracle gains, concentrating where single models struggle most.

The Review Panel: Three Different Models, Three Different Jobs

The first version of this review panel was based solely on the gain you get from having outside models review the work, be it specification, plan, task list, or code. A February 2026 scaling study measured what happens as you add identical agents to an ensemble: clones of one model saturate almost immediately because their outputs correlate, and two agents from different model families matched or beat sixteen copies of one. PoLL found a panel of smaller judges drawn from disjoint families tracked human judgment better than a single GPT-4 judge across six datasets, at over seven times less cost.

The reason the model family boundary matters for review specifically: LLM evaluators recognize their own writing, and self-recognition scales linearly with self-preference. Great Models Think Alike (ICML 2025) should end the popular pattern of having a model’s cheaper sibling do the reviewing: as capability rises, models’ mistakes converge, so a same-family reviewer misses what its sibling misses and likes what its sibling likes. A panel of OpenAI, Gemini, and a rotating third seat, none of them the family that wrote the code, is exactly what this research prescribes.

In the past each reviewer was given the same prompt, relying on variations in model behavior to catch diverse problems, but evidence shows that approach does not give the best results. ChatEval’s ablation tested evaluator single-model ensembles where the prompts were identical, and the performance was no better than a single-agent performance. A study of 162 personas across four model families and 2,410 factual questions found “you are an expert” prompts produced no measurable improvement on the tested tasks, and even automatically picking the best persona per question did no better than random selection.

What works is restricting scope. Dividing up the review responsibilities into different areas shrinks what the reviewer has to read, and smaller, focused context is one of the best-replicated results in this literature. Chroma’s context-rot study ran 18 models against questions asked two ways, a focused prompt of about 300 tokens holding only the relevant material versus the full 113,000-token history it came from; the focused prompt won on every model tested, and the underlying benchmark shows 30 to 60 point absolute drops on long inputs.

Family diversity contributes more than role prompts, and the combined design ranks strongest on the available evidence; the combination itself is not yet benchmarked end to end. Three staffing patterns for the same reviewer slot.

So at every gate each reviewer gets a different prompt, because the ways a spec can be wrong are not the ways code can be wrong. What stays constant is the shape: one reviewer checks fidelity to what came before, one checks whether the thing holds together on its own terms, and one checks whether it will survive what comes after.

The caveat: nobody has published the end-to-end benchmark, jurisdiction-scoped cross-family panels versus generic-prompt panels, scored on bugs caught in real code review. The studies support each mechanism separately, so I’m instrumenting my own panels, tracking unique findings per family, per jurisdiction, per gate. One jurisdiction already has its priority set by data: spec adherence leads the code row because disobeying the spec is the dominant measured failure mode in multi-agent systems, as the trace data in the swarms section below shows.

Consulting is not reviewing, and the two get opposite context. When the main agent brings other models in at a planning gate for advice, they get the full context, forked, everything, because Cognition found a partially informed consultant gives confidently partial advice. A model reviewing finished work gets only the diff and the task’s slice of the spec. Cognition’s Devin Review averages about two flagged issues per AI-authored PR (internal vendor telemetry with unpublished methodology; treat it as directional), and it worked best when reviewer and coder shared no context at all. Their name for the pattern, “reviewers without egos”: reviewers that share no state with the coder.

None of this makes three reviewers a magic number. In my experience even one outside model reviewing every stage cuts the chance a bug ships, and three cuts it further; what the scaling study adds is where the marginal dollar goes, to a different family rather than another copy of one you already have. Diversity has a floor, though: heterogeneous teams sometimes underperform their best member, because a weak model added for variety drags the synthesis toward itself. The third seat rotates among near-frontier models only. And ignore the weekly model-quality panic online: any model fails sometimes, on a sample of one, which is exactly the failure this workflow is built to absorb.

Hand-curated Tests with Golden Datasets

Reviews check intent and implementation; only tests check output against a right answer, and with LLM-generated code the output is where the bugs hide. Neither of the first two pillars runs the code. Neither compares actual output against a correct answer on inputs a human deliberately chose to be nasty.

A human engineer writes code that looks wrong when it’s wrong, through weird structure, nervous comments, sloppy naming. An LLM writes code that looks right even when it’s wrong, because it has been trained to produce code that pattern-matches working code. Three models in a debate round can all agree the code is fine and all three can be fooled by the same plausible-looking mistake; the mistake-convergence result above says this risk grows, not shrinks, as models improve. The only discipline that reliably catches this class of failure is running the code against ground-truth output that a human curated by hand.

The hand-curated part is the whole point. Tests written by an AI against its own output are circular; they assert that the code does what the code does, not what the code was supposed to do. The point of a golden dataset is that a human sits down, picks the inputs most likely to trip the system, decides what the correct answer should be, and encodes both. The test harness then runs the code and checks. This is the one artifact in the pipeline that cannot be delegated to the model, because it is the definition of correctness, not a derivative of it.

Skip this pillar and the first two can still let wrong answers through. Keep it, and each pillar catches what the others miss: intent, implementation, and output. Sound like extra work compared to a traditional development process? It shouldn’t be since this has been common practice for as long as companies have been shipping complicated software. The only thing that’s changed is the AI model needs to use this up front to be able to design and validate the code it writes.

Quality Processes Still Rule

The three aspects in this article don’t replace lint, CI/CD, and human PR review; they depend on them, because any individual model will eventually generate something terrible with complete confidence. This is not different from managing a team of engineers, where any of them will have a bad day eventually and write something that looks fine until it isn’t. The answer is not a better prompt. The answer is the operational layer the industry settled on decades ago, and the three aspects above make that layer more important, not less.

When I was working at Sun Microsystems on the audio API team, one of my duties was to ship the old API while designing and coding the new one. Their approach to quality was obsessive: each possible variation of the OS was automatically tested on each possible variation of hardware. It was complicated and expensive, and completely worth the effort if your main goal is reliability. That lesson is much more important with AI-generated code. If you have to rely on complete human checking of every line of code and manual testing, that mostly means you haven’t done a thorough job of automating the testing process. At webperformance.com, we had 3,000+ tests for a single product, including the all important end-to-end tests of the hardest and most important use cases. After running the full suite of tests I could rest assured that the major AI-driven change I just made is guaranteed not to break anything. Human review of AI-generated code is useful; relying on your human review to prevent bugs means you need to revamp your test suite.

Recent research did argue me into one change here: the pull request is not another multi-LLM panel run. By the time a branch becomes a PR, every task on it has already survived a three-family review, and running the same detectors on the same code again adds correlated redundancy, not safety. The PR slot gets checks that fail differently. A small job verifies the spec, plan, and per-task review records actually exist for the diff. Deterministic tooling covers the failure modes models are worst at: tests, type checks, static analysis, secret scanning. A human reviews the assembled bundle against the spec’s intent, which is the first moment the whole feature exists to be judged. The one machine check worth adding at that boundary is a whole-feature coherence pass, because it reviews an object no per-task review ever saw.

One-shotting an app looks like magic because you press a button and something comes out the other end. But that’s the output you would get from a human engineer told not to run tests, not to exercise the app, and to ship as soon as it compiles. Of course it’s crap. If you use the same professional workflow you would use with humans, the hallucinations and bad-confidence moments get absorbed the same way a weak PR from a tired engineer gets absorbed: somebody catches it and it doesn’t ship.

What about Multiple Agents/Swarms?

Multi-agent frameworks failed 41% to 86.7% of their tasks in the largest failure study to date, and the dominant failure was agents ignoring the spec, not agents losing context. That is why, exciting as multi-agent programming has become, especially with features like Anthropic’s Agent Teams, swarms aren’t featured more prominently here: the first goal of any development process is reliability, each phase of a complex project delivered with minimal rework and maximum consistency. That position used to be instinct backed by scars; the research has since caught up, and the numbers are worse than even I expected.

The study behind those numbers is MAST, NeurIPS 2025: 1,642 execution traces from seven open-source multi-agent frameworks, every failure annotated and classified. Nearly four in five failures traced to organizational design rather than model capability, and the two dominant modes, agents repeating steps (15.7% of traces) and agents disobeying the task spec (11.8%), dwarf the one everyone architects against, losing context in a handoff (2.8%). The failures are organizational, not technical: the messages get through, and the instructions get ignored. For this workflow that’s quietly good news, because spec adherence is checkable. It’s what the review gates are for.

Spec-adherence failures dwarf context-handoff failures in multi-agent systems: 15.7% and 11.8% of traces versus 2.8% and 0.85%. From MAST (NeurIPS 2025), 1,642 annotated traces.

The economics point the same direction. Anthropic’s production numbers put a single agent at about 4x the tokens of a chat session and a multi-agent system at about 15x, and their own engineering write-up steers you away from the pattern for code: most coding tasks have fewer truly parallelizable pieces than research does. A seven-framework comparison running identical SWE-bench work on the same base model found single-agent frameworks repaired more issues while multi-agent frameworks took nearly twice the execution steps, with a 23x cost spread on the same work: $370.19 for the priciest orchestrator against $16.29 for the cheapest alternative. In one measured system the orchestrator alone burned 67.2% of all tokens, drowning in its own workers’ status reports until it started making bad decisions. Delegation doesn’t eliminate the context problem; it moves the problem up a level, into the orchestrator.

A multi-agent system uses about 15x the tokens of a chat session; a single agent about 4x. Anthropic production data, 2025, measured before prompt caching.

The two labs with the most skin in this game converged from opposite ends. Anthropic built the famous multi-agent researcher and warns you off using the pattern for code. Cognition wrote “Don’t Build Multi-Agents”, the most-cited argument against the whole idea, then partially reversed itself in 2026 with a sentence worth framing: multi-agent systems work best when writes stay single-threaded and the additional agents contribute intelligence rather than actions. One writer, many readers. The main agent holds every write. Sub-agents explore the codebase, chase dependencies, and hunt prior art, returning compressed summaries, because a search result compresses honestly (it found the thing or it didn’t) while a summary of someone else’s diff drops exactly the implicit decisions the next edit depends on.

Isn’t This Slow and Expensive?

Slow to first code, faster to shippable code, and the outside reviews free me up to run several major projects simultaneously. I spend most of the up-front budget on architecture, data model, and how to comprehensively test the output before any code is generated. But what gets generated then tends to run and match the spec on the first pass. It may not be what I stick with, but it works. Without this structure you end up with a buggy mess, the AI trying to fix bug after bug while you head to Reddit to post that your favorite model was “nerfed.” Model quality will always fluctuate. With this workflow, it doesn’t matter.

As for cost, reviewing plans and code is cheap. I’m on API key plans for everything but Claude Code, and those extra LLMs typically run $100-$200 a month combined. For a professional programmer, that’s one billable hour a month to save untold hours of work.

Isn’t This Overkill for Small Problems?

Of course. As any software project manager will tell you, you pick the level of process appropriate to the size and importance of the task. I divide my work into three tiers:

What you can’t skip on is the testing. “Just tell the AI to do it” without a plan only works if there’s a comprehensive testing framework that will catch regression, “tell the AI” has to include test driven development instructions to create the proof that what is created will actually work. The overhead of the full workflow only pays off when the work is substantial enough that catching mistakes early matters more than shipping fast. For small jobs, one-shotting is possible only if you have well-designed unit and integration test suites. Just don’t mistake a big job for a small one.

The Bottom line

Twenty major revisions in twelve months, each shipped in days instead of the month or two the old way would have taken, and no meaningful mistakes: that is the record this workflow has to show. Can it fail? Yes, the same as any development process; the record started only after I worked through the initial gotchas and folded the recovery patterns into the workflow. The pace is, frankly, astounding.

In a world in which you can press a button and get an article, people have become wary of reading AI slop. This article was written by hand and then edited/modified by AI; this revision folds in a machine-assisted research pass over the 2025-2026 literature, with every claim linked so you can check the sources yourself. The opinions, and almost all of the sentences remain my own.

Key Sources

The load-bearing studies, one line each, for readers who want to check my work:

• MAST (NeurIPS 2025): 1,642 multi-agent traces; spec disobedience (11.8%) and step repetition (15.7%) dwarf context loss (2.8% and 0.85%) as failure modes.
• BOAD (ICLR 2026): a hand-designed agent hierarchy loses to a single agent (47.4% vs 49.8%); the bandit-discovered one that wins runs its specialists one at a time.
• Agent scaling via diversity (2026): two diverse agents match sixteen clones; model plus prompt diversity wins.
• PoLL: judge panels from disjoint families track human judgment better than a single GPT-4 judge across six datasets, at over seven times less cost.
• LLM evaluators favor their own generations (NeurIPS 2024): self-recognition scales with self-preference.
• Great Models Think Alike (ICML 2025): capability convergence correlates mistakes and biases judges toward similar models.
• ChatEval (ICLR 2024): identical role prompts collapse an evaluator ensemble to single-agent performance.
• Personas don’t work (EMNLP Findings 2024): 162 expert personas, four model families, 2,410 questions, no improvement; scope jurisdictions instead.
• Context rot (Chroma): 18 models degrade as input grows; a 300-token focused prompt beat a 113,000-token history everywhere, with 30 to 60 point drops on the underlying long-input benchmark.
• Seven-framework comparison: single-agent frameworks repaired more, at as little as 1/23rd the cost, on identical work.
• Anthropic’s multi-agent write-up and Cognition’s reversal: the two labs that converged on single-threaded writes.