The Verifiability Thesis - Why AI Automates Some Jobs in Hours and Others in Never

A coding agent shipped a 12.5-million-line refactor in seven hours with 99.9% numerical accuracy.

A legal agent can't reliably summarize a 40-page contract without a human review pass.

Both are state-of-the-art in 2026. Why the gap?

The framework nobody put on a slide

Andrej Karpathy has been hammering one idea for months, and it's the most useful lens I have for thinking about where AI actually wins this year:

AI automates fastest in domains where a proposed output can be cheaply verified.

Not where the work is "easy." Not where there's lots of training data. Where you can write a checker.

Coding is the canonical case. A test passes or it doesn't. A type-checker is binary. A linter is mechanical. An agent proposes a 500-line diff; the build runs; the answer is green or red. The cost of verification is near zero, so the cost of trial-and-error iteration is near zero, so the agent can grind.

Legal work is the opposite case. A contract summary is "correct" only against ground truth nobody can read. Verification requires a human expert who already knows the answer - which means the verification cost exceeds the production cost. The agent loops forever or hallucinates confidently.

Why this is the actual story of 2026

Most AI trend pieces lump everything together: agents are getting better, models are getting bigger, productivity is exploding. That's a surface read. The verifiability thesis explains the shape of what's happening:

• Coding agents scale superlinearly. TELUS engineers saved 500,000 hours. Rakuten saw seven-hour autonomous refactors. The verification loop is the moat.
• Math agents scale similarly. Formal proof checkers (Lean, Coq) give a binary verdict. DeepMind and OpenAI both published breakthrough results this year - not because math got easier, because the checker got integrated.
• Writing agents improve unevenly. Email drafting: easy to verify (did it sound professional? did it cover the points?). Long-form creative work: nearly impossible to verify without re-reading the whole domain.
• Healthcare and law stay jagged. A diagnostic agent that proposes a treatment plan can't be cheaply verified. A radiologist is still in the loop.

The pattern isn't "AI is good at X and bad at Y." The pattern is "AI scales with verification cost, not task difficulty."

The jagged frontier, mapped

Here's how I'd categorize work in 2026 by verifiability:

| Verification type | Examples | Agent maturity | |---|---|---| | Deterministic checker | Code with tests, math proofs, SQL queries, type-checked migrations | Production-ready, multi-hour autonomy | | Cheap human spot-check | Email drafts, summaries, PR descriptions, marketing copy | Useful, needs review | | Expensive human review | Strategy docs, architecture decisions, hiring, design | Assistant-only | | Ground-truth-only | Diagnostics, legal advice, novel research | Augmentation, not automation |

The honest answer for most companies in 2026 is that they're operating in row 2, not row 1. They deploy agents for tasks where a human reads the output in 30 seconds - which is a real productivity win, but not the "AGI replaces my job" story.

How to use this as a decision framework

If you're deciding where to deploy agents in your own work:

1. Find your checkers. What's the cheapest signal that an output is correct? If you can't name it in one sentence, the workflow isn't ready for full autonomy.
2. Instrument the loop. Every minute spent on verification is a minute an agent can't iterate. The teams winning in 2026 are the ones who built the feedback loop, not the ones who picked the smartest model.
3. Resist the urge to skip row 2. Cheap human spot-check is not a failure state. It's a productive one. Don't pretend your agent is row 1 when it's actually row 2.
4. Watch for verification cost to drop. A new tool that makes verification cheaper - a linter, a test harness, a domain-specific checker - expands the automation frontier overnight. This is why coding led the wave and not because coding is special.

The uncomfortable implication

The verifiability thesis implies something a lot of AI hype pieces avoid: most human work is row 3 or row 4.

The things people get paid a lot to do - strategy, judgment, taste, persuasion, taste in design - are exactly the things that are hardest to verify. Which means exactly the things where AI iteration loops are slowest. Which means exactly the things where automation will arrive last, no matter how smart the model gets.

That's not a comforting story. It's an honest one. And it explains why, three years into the agent era, the visible productivity gains are concentrated in software engineering and mathematics, while the rest of the economy looks... fine.

Where this leaves us

If you're an engineer or PM in 2026: pick your battles by verifier, not by task. Find the workflows where you can write a checker in an afternoon, deploy an agent, and let it grind. Save your human judgment for the places where verification is itself the job.

The agents aren't going to take the unverifiable work. They're going to take the verifiable work - and that's a much smaller, more predictable, more useful revolution than the headlines suggest.

What's the cheapest checker you could write for one of your workflows tomorrow?