In November 2024 I wrote an article called The AI Agent Revolution. I thought I had a reasonable idea of what was coming.
The argument was that AI was entering what I called “Phase 3”. Phase 1 had been conversational AI: systems that could talk, write and answer questions. Phase 2 was reasoning. Phase 3 would be different because AI would stop merely telling us what to do and start doing things for us.
At the time, this was mostly prediction. Microsoft had demonstrated Magentic-One, an experimental system in which several AI agents worked together. Anthropic had given Claude the slightly unnerving ability to look at a computer screen, move a mouse and click on things. OpenAI was rumoured to be developing something called Operator.
Nearly two years later, the interesting thing is not that the prediction was right. Agents have arrived. It is that they are turning out to be rather different from the autonomous AI workforce I had imagined.
They are both more capable and more constrained.
Consider Anthropic’s original computer-use demonstration. In October 2024 Claude 3.5 Sonnet scored just 14.9% on OSWorld, a benchmark that asks an AI to operate ordinary software through a computer interface. Humans scored around 72%. Watching Claude use a computer was remarkable, but the score was a useful corrective. It was a glimpse of something interesting rather than something you would necessarily want anywhere near your finance system.
Claude Opus 5.5 now reports 81.8% on OSWorld 2.0. The benchmark and methodology have changed, so treating 14.9% to 81.8% as a straightforward improvement would be misleading. Even so, something fairly substantial has happened in between.
Software engineering tells a similar story.
Claude 3.5 Sonnet scored 49% on SWE-bench Verified in October 2024. The test asks an agent to take a real GitHub issue, understand an unfamiliar codebase, change it and produce a working fix. By August 2025 GPT-5 was scoring 74.9%. Frontier models became so good at the benchmark that OpenAI eventually said SWE-bench Verified was no longer particularly useful for measuring the best coding systems.
So the tests got harder.
On Terminal-Bench 4.0, which involves complex terminal-based work including software engineering, system configuration and data analysis, GPT-6 Astra scores 57.9%, compared with 37.3% for GPT-5.6 Sol and 55.8% for Claude Fable 5.1.
Research agents have improved just as dramatically. When OpenAI introduced BrowseComp in April 2025, it deliberately constructed questions whose answers were difficult to find on the web. GPT-4o with browsing managed 1.9%. OpenAI’s Deep Research system reached 51.5%. ChatGPT Agent subsequently reached 68.9%. GPT-6 Astra now reports 91.5%.
I am increasingly wary of apparently miraculous benchmark improvements, however. Once a benchmark becomes public, the relationship between the test and what it is supposedly testing gets complicated. Anthropic has even documented Claude recognising BrowseComp and finding leaked answers online. An AI cheating on its AI benchmark is perhaps progress of a sort, although not quite the sort the benchmark designers had in mind.
The exact numbers therefore deserve some caution. The underlying change is harder to explain away. These systems have become much better at searching, changing strategy, using tools and persisting when their first attempt fails.
The measurement I find most interesting comes from METR because it asks a different question. Instead of testing whether an AI can answer something difficult, it asks how long a task an agent can take on before it becomes unreliable.
METR estimates how long a skilled person would need to complete a coherent task and then measures whether an agent can complete it successfully. Historically, the length of task frontier agents can complete with 50% reliability has been doubling roughly every seven months.
By May 2026 an early Claude Mythos system had reached a 50% task horizon of at least 16 hours on METR’s predominantly software and technical tasks. METR actually ran into a slightly unusual measurement problem: beyond 16 hours there were too few sufficiently difficult tasks left to estimate the result reliably.
That does not mean an AI can now turn up on Monday morning and reliably do somebody’s job until 5pm. METR’s tasks are unusually well specified and mostly technical. Real work contains ambiguity, politics, missing information, badly written emails and meetings whose purpose nobody can quite remember.
Still, sixteen hours matters. It suggests that we are moving from AI completing isolated tasks to AI sustaining useful work across long sequences of decisions.
And this is where things get more complicated.
Zapier’s AutomationBench puts agents inside simulated businesses and asks them to complete workflows across CRMs, inboxes, calendars and the other systems used in sales, marketing, finance, HR, support and operations. GPT-6 Astra currently completes 41.4% of those workflows successfully. Claude Opus 5.5 manages around 40%. GPT-5.6 Sol manages 18.1%.
Forty-one per cent is simultaneously an extraordinary result and a terrible one.
A couple of years ago, an AI independently completing four out of ten unfamiliar business processes would have sounded like science fiction. If I were actually running those processes, though, I would be rather interested in what happened to the other six.
That, I think, explains much of what has happened with agents.
The technology has become powerful enough to do useful work, but not reliable enough to be given unlimited freedom. So businesses have started building systems around it.
An agent today is no longer simply a language model with a clever prompt attached. It can use browsers, terminals, files, databases and business applications. It can maintain context while it works, choose tools and sometimes recover from its own mistakes.
OpenAI’s Operator, still a rumour when I wrote the original article, appeared in January 2025 and could operate a browser itself. Since then, that sort of capability has been absorbed into broader systems combining browsing, research, coding, files and connections to other applications.
The model is increasingly only part of the product. The interesting questions are what the agent can see, what tools it can reach and, above all, what it is allowed to do.
Real deployments make this clearer.
Siemens now uses two AI agents to deal with incoming sales leads. One talks to potential customers and another qualifies them before suitable opportunities reach salespeople. About 2,500 leads a month across 132 countries pass through the system, which Siemens says went from prototype to production in six weeks.
But look more closely at how it works. The AI conducts the conversation while roughly 50 predefined business rules determine the qualification outcome. There are also boundaries around what the agent is allowed to discuss.
That is not an autonomous digital salesperson roaming freely through Siemens. It is something more carefully engineered, and probably more useful.
Uber has taken a similar approach with Uber for Business. Its agent follows up with leads, answers questions and revives conversations that might otherwise have gone cold. Conversion from those emails is reported to be around 60% higher than Uber’s previous organic baseline.
Customer service has gone further. Intercom says its Fin agent is used by more than 12,000 businesses and resolves about 76% of the conversations in which it is involved on average.
These are no longer demonstrations. McKinsey’s latest survey found that 40% of respondents from companies with more than $1 billion in annual revenue said they were scaling AI agents, up from 27% a year earlier.
What has not scaled at quite the same rate is trust.
A chatbot producing a poor answer is irritating. An agent sending the wrong information to thousands of customers, exposing confidential data or incorrectly updating a financial system is a different class of problem. The more useful agents become, the more consequential their mistakes become too.
This has created a market I paid much less attention to when I wrote the original article: the machinery around the agent.
Companies such as Dynamo AI are building control layers that watch what goes into models, what comes out and what an agent is allowed to do. DynamoGuard, for example, can detect sensitive-data leakage, hallucinations, prompt injection and policy violations, then block, sanitise or escalate activity in real time.
Credo AI approaches the problem more from the governance side, helping organisations keep track of agents, map them against regulatory and corporate requirements and monitor what they do once deployed.
I suspect this layer will become much more important.
Traditional technology governance assumes that a system is designed, tested, approved and then operated more or less as expected until somebody changes it. That assumption becomes uncomfortable when the software can make thousands of individual decisions every day and can decide for itself which tool to use next.
A company therefore needs to decide what data an agent can see, which systems it can access, what it can change, when it needs permission and when it should simply stop.
Those rules have to exist in the software. A policy document sitting in SharePoint will not save you.
This has also changed my view of the “autonomous workforce” I wrote about in 2024.
I imagined agents gradually becoming something like digital employees, with humans remaining in the loop to supervise them. There is some of that. But the more interesting pattern is that agents are beginning to rearrange jobs rather than simply replace them.
In sales, an agent can research a customer, start the conversation and qualify the opportunity. The salesperson spends more time on the relationship and the deal.
In customer support, an agent can absorb large volumes of routine questions. People deal with the unusual, sensitive or commercially important cases.
A coding agent can write, test and debug substantial amounts of software. Engineers can spend more time deciding what should be built, thinking about architecture, reviewing what the agent produced and dealing with the difficult exceptions.
The same thing is beginning to happen in analytical work. An agent can collect information, reconcile data, build an initial model, test assumptions and produce a first version of an analysis. The scarce part of the job then moves elsewhere: deciding whether the assumptions make sense, noticing what is missing and explaining what the numbers mean in the real world.
If AI makes the mechanical parts of analysis much cheaper, companies with deep subject expertise and strong client relationships may not need fewer people doing the same amount of work. They may be able to produce more with the same team, turn work around faster and take on assignments that previously would not have been economical.
Two years ago, I thought the important question was whether AI agents would become capable enough to do real work.
I don’t think that is the question anymore. They plainly can.
The more useful question is which pieces of work we are prepared to hand over to them. The best candidates seem to have a few things in common: the task has reasonably clear boundaries, success can be measured, mistakes can be detected and, ideally, a bad decision can be reversed before too much damage is done.
Then comes the harder part. How much authority should the agent have? What can it see? What can it change without asking? When does a person need to intervene?
That is rather less exciting than predicting an army of autonomous digital employees. It is also much closer to what businesses actually need to work out.
Phase 3 AI arrived. The surprise is that the most interesting question is no longer how intelligent the agents are becoming, but how much of our world we are prepared to let them touch.
