Analysis: what actually changed when AI agents moved from demo to production

There is a familiar arc to agent projects. A prototype does something startling in week one. By week six it is producing plausible nonsense on inputs nobody anticipated, and the team is arguing about whether to add another model or give up.

The projects that came out the other side did not find a better model. They changed the shape of the problem.

The arithmetic that kills demos

Take a task decomposed into steps, each of which the agent completes correctly 95% of the time. That is a good rate for an open-ended step involving judgement.

Chain three steps and roughly 86% of runs succeed. Chain twenty and about a third do. Chain fifty and you are at 8%.

The demo showed you one run of a five-step task, and a human quietly restarted the two attempts that went wrong. Production runs the twenty-step version ten thousand times with nobody watching.

Everything that follows is a response to this arithmetic.

What the working deployments have in common

A narrow domain. Not "handle operations" but "reconcile these two invoice systems". Fewer branches, fewer tools, fewer ways to be wrong. Almost every agent that survived contact with production is less ambitious than the demo that justified it.

Cheap verification. This is the strongest single predictor. Coding agents worked first because tests provide an objective, automatic signal: the agent can try, check, and retry without a human. Where verification is expensive — a legal opinion, a pricing decision — the agent's output has to be reviewed by a person, which caps the leverage and changes the business case.

A permission boundary outside the model. Read broadly, write narrowly. Approval gates on spending money, contacting customers, deleting data, deploying. This is enforced in the runtime, never by instructions in a prompt: an agent that reads untrusted content is an agent whose instructions can be contaminated.

Budgets. Maximum steps, maximum wall clock, maximum tokens. An agent looping on a malformed tool response is a billing incident with no natural end.

A replayable log. Every tool call and result, stored. When something goes wrong, "what did it actually do" must be answerable immediately — both to fix it and, increasingly, to satisfy an auditor.

Evaluation grew up

The most consequential change is the least visible. Early agent work was evaluated by watching it. That does not scale and it does not catch regressions.

The current practice is to record trajectories — the full sequence of steps, tool calls and results from real runs — and replay them against every change to the prompt, the model, or the tool set. Each trajectory carries an assertion about what a correct run looks like. A new model version is not adopted because it scores better on a public benchmark; it is adopted because it does not break the recorded set.

The second change is per-step scoring. End-to-end pass rates tell you that something is wrong. Step-level scoring tells you the retrieval tool returns stale results on Mondays.

Tools became infrastructure

For a period, every agent product wrote its own connectors. The Model Context Protocol turned tool exposure into a standard interface: a server describes what it offers, and any compatible client can use it. That has moved integration work from bespoke glue in each application to reusable servers maintained once.

The practical effect is mundane and large: the interesting question shifted from "how do I connect my agent to this system" to "what should my agent be permitted to do with this system". That is a governance question, and it is the right one.

Where it has not worked

Worth stating plainly, because the failures are less publicised:

• Long-horizon autonomous operation. Agents left to run for hours on open-ended goals still drift, and the cost of a wrong turn compounds silently.
• Tasks with expensive verification. If a human must read the whole output to know whether it is right, the agent saved typing, not work.
• High-variance environments. Websites that change layout, systems that return inconsistent errors, processes with undocumented exceptions.
• Swarms for their own sake. Multi-agent architectures help when subtasks need genuinely different tools and contexts. Applied to a task a single agent could do, they add coordination failures.

The honest summary

Agents work today where the goal is clear, the domain is narrow, the result is cheap to check, and the blast radius is bounded by something other than the model's good judgement. That is a real and growing category — and it is a smaller one than the word "agentic" in a product announcement implies.

The teams shipping successfully are, without much exception, the ones who accepted that first.