Tag: chatgpt

The artificial intelligence industry has become rather obsessed with a handful of companies. OpenAI, Anthropic, Google and their Chinese competitors dominate the headlines, with each new model release accompanied by impressive benchmarks and increasingly ambitious claims about what machines will soon be capable of doing.

But behind this competition, another industry is taking shape. Building more intelligent models is only part of the challenge. We still have remarkably little understanding of how neural networks arrive at their decisions, struggle to control what autonomous agents can do, and are discovering that running AI at scale requires extraordinary amounts of computing power and electricity.

Some of the most interesting companies are trying to solve these problems rather than build another general-purpose language model. I’ve picked five that I think are worth following, each working on a different part of the emerging AI ecosystem.

Dynamo AI: Keeping autonomous AI under control

Dynamo AI was founded in 2021 by MIT researchers Christian Lau and Vaikkunth Mugunthan, whose early work focused on privacy, security and the responsible deployment of machine learning. The company has since developed technology that helps organisations test, monitor and control AI systems.

The problem becomes particularly interesting with the emergence of autonomous agents. An AI that answers questions can certainly make mistakes, but an agent that can access corporate systems, execute code, transfer information or make decisions introduces a different class of risk. Traditional security controls can determine whether someone has permission to access a database, but an AI agent might have perfectly legitimate access and still decide to do something nobody intended.

Dynamo’s products, including AgentWarden, attempt to address this by assessing the tools available to agents and enforcing policies as they operate. Rather than relying entirely on instructions telling an AI to behave itself, organisations can impose controls at the points where an agent interacts with other systems.

I find this an important area of development because the more capable agents become, the harder it will be to justify giving them unrestricted access to business operations. I should also disclose a personal connection: my son works at Dynamo AI, although I have no financial interest in the company. That has naturally made me follow its work more closely.

Goodfire: Understanding what is happening inside the machine

Goodfire was established in San Francisco in 2024 by researchers with backgrounds in AI interpretability, including work at OpenAI and Google DeepMind. Its ambition is to understand how neural networks actually work, a question that sounds rather basic until you appreciate how little we currently know about the internal mechanisms of the most sophisticated models.

Modern neural networks are trained using vast amounts of data and computation. Through this process they develop complex internal representations that allow them to reason, recognise patterns and generate surprisingly sophisticated responses. Researchers understand the mathematical operations involved, but identifying exactly how a model represents a concept or arrives at a particular decision remains difficult.

Goodfire is working on mechanistic interpretability, developing techniques that reveal and manipulate these internal representations. Its research has explored whether understanding a model’s internal workings can help reduce hallucinations, detect undesirable behaviour and improve training.

The company raised $150 million in February 2026, reflecting growing interest in this field. What appeals to me is the possibility of moving AI development away from the current reliance on enormous experiments, repeated testing and considerable educated guesswork. Other engineering disciplines generally expect practitioners to understand how their creations function. AI has made extraordinary progress without meeting quite the same standard, although Goodfire is not alone in trying to change that.

d-Matrix: Tackling the economics of intelligence

d-Matrix was founded in 2019 by semiconductor engineers Sid Sheth and Sudeep Bhoja. The company designs specialised processors for AI inference, the process of running a trained model to produce answers, rather than training the model in the first place.

The distinction is becoming economically important. Training a frontier model can cost hundreds of millions of dollars, but once deployed, that model may be asked to answer billions of questions. Every interaction requires computation, and increasingly sophisticated reasoning models can consume substantial processing resources even when responding to relatively straightforward requests.

Much of this workload currently runs on expensive GPUs, particularly those supplied by Nvidia. d-Matrix has developed an alternative architecture that brings memory and processing closer together, reducing the movement of data that can slow down conventional computing systems. Its Corsair accelerator entered full production in June 2026 and is designed to work alongside existing infrastructure to improve inference performance.

The potential implications are significant. If AI is going to become a routine part of business software, its cost per useful result will become a critical commercial measure. There is little point developing an extraordinarily capable agent that costs more to operate than the work it replaces. Companies that can materially improve the economics of inference may end up capturing considerable value, even if most people never encounter their names.

Prime Intellect: Challenging the concentration of AI computing power

Prime Intellect was founded in 2024 by Vincent Weisser and Johannes Hagemann. Its objective is to make advanced AI development possible without requiring every participating organisation to own or rent a vast, centrally managed computing facility.

The largest AI models have traditionally been trained using thousands of tightly connected processors operating within enormous data centres. This is expensive and creates a substantial barrier to entry for independent researchers, universities and smaller technology companies.

Prime Intellect is developing techniques that allow computing resources in different locations to collaborate on training tasks. That is technically difficult because processors separated by thousands of miles communicate far more slowly than machines sitting next to one another. The company has nevertheless demonstrated distributed training, including its INTELLECT-2 project, which used globally distributed reinforcement learning to train a 32-billion-parameter reasoning model.

This work interests me for reasons beyond computing efficiency. Access to sufficient processing power increasingly determines who can undertake serious AI research. If the industry’s future depends on infrastructure affordable to only a handful of companies and governments, the consequences for competition and independent innovation could be considerable. Prime Intellect is exploring whether that concentration is an unavoidable feature of AI or partly a problem of engineering design.

ElevenLabs: Changing how we communicate with computers

ElevenLabs was founded in 2022 by Polish entrepreneurs Mati Staniszewski and Piotr Dąbkowski, initially inspired by the poor quality of dubbed films. They wanted to produce synthetic speech that sounded genuinely human, preserving expression, timing and emotion rather than simply converting written words into recognisable sounds.

The company has expanded into voice generation, translation, dubbing and real-time conversational technology. In September 2026, it reached a valuation of $22 billion following a secondary share transaction, an extraordinary achievement for a business only four years old.

Its broader significance lies in making spoken conversation a practical interface for AI. A genuinely useful voice assistant needs to do considerably more than understand words. It must handle interruptions, recognise when someone has finished speaking, respond with minimal delay and maintain the natural rhythm of conversation. These are difficult technical problems that become more demanding when the system is also expected to reason and take action.

For decades, humans have adapted to computers through keyboards, screens, menus and increasingly elaborate software interfaces. Natural conversation offers the possibility of reversing some of that relationship. ElevenLabs is also confronting the accompanying problems of consent, impersonation and fraud, particularly when synthetic voices become difficult to distinguish from real ones.

An industry still being invented

What I find striking about these companies is how different their challenges are. Dynamo is concerned with controlling AI behaviour, Goodfire with understanding it, d-Matrix with making it economical, Prime Intellect with broadening access to its development, and ElevenLabs with changing how we interact with it.

None has an assured future, and some of their technologies may eventually be absorbed into the platforms of the larger technology companies. There is also a reasonable possibility that improvements in the underlying models will change the problems themselves, making some approaches more valuable and others redundant.

Yet the existence of these businesses tells us something about the maturity of the AI industry. We have made remarkable progress in creating intelligent systems, but many of the disciplines required to make those systems dependable, affordable and widely accessible are still being developed.

The major model providers will continue to attract most of the attention, understandably. I suspect, though, that some of the most consequential innovations over the next few years will come from companies solving problems that rarely feature in the benchmark tables. They are building the infrastructure, tools and scientific understanding on which the next stage of AI will depend.

GitHub Spec Kit is an open-source toolkit for implementing Specification-Driven Development (SDD) using AI coding agents.

Rather than asking an AI agent to write software directly from a prompt, Spec Kit introduces a structured process in which requirements are clarified, technical decisions are made, implementation tasks are generated and the resulting software is verified.

It represents an interesting evolution of ideas developed through Behaviour-Driven Development (BDD) and Acceptance Test-Driven Development (ATDD), where the emphasis was on agreeing what software should do before writing it. The difference is that AI agents can now undertake much of the implementation, testing and documentation.

Spec Kit works with GitHub Copilot and a range of other supported coding agents. It is not tied to a particular cloud platform and can be incorporated into existing Microsoft Azure, AWS or hybrid engineering environments.

1. The Spec Kit lifecycle

Spec Kit organises development around a series of commands, each producing or refining artefacts that inform the next stage.

The sequence is structured, but not rigid. Teams can revisit specifications and plans as they learn more.

The crucial distinction is that the specification describes the intended behaviour, while the plan describes the technical implementation.

This separation helps prevent technical decisions from prematurely determining what the product should do.

2. Getting started

You need Git, Python tooling, uv and a supported AI coding agent. For a Microsoft-oriented development team, GitHub Copilot is a natural starting point.

Install Spec Kit from a terminal:

uv tool install specify-cli --from git+https://github.com/github/spec-kit.git

Create a project configured for Copilot:

specify init my-project --ai copilot
cd my-project

Open the project in your coding environment and start your AI agent.

The commands in the next section are entered into the agent’s chat, not the terminal. Exact command syntax and supported integration options can change between Spec Kit releases.

3. A worked example

Imagine developing a feature for a regulatory compliance platform that allows customers to receive alerts when regulations affecting their business change.

Step 1: Establish engineering principles

/speckit.constitution
Use Azure-native services.
Follow established security and coding standards.
Require automated unit and integration tests.
Use existing APIs and shared components where possible.
All changes must pass CI/CD quality gates.

These principles are intended to apply consistently across subsequent development.

Step 2: Specify the feature

/speckit.specify
Allow customers to subscribe to regulatory
changes affecting selected countries and topics.
Customers must be able to:
- Select countries and regulatory topics
- Choose notification frequency
- View relevant changes
- Unsubscribe at any time
Include measurable acceptance criteria.

The agent creates a structured specification describing the required behaviour.

Step 3: Clarify requirements

/speckit.clarify
Identify ambiguous business rules,
edge cases and missing acceptance criteria.

For example, what happens when a customer subscribes to a topic that has no regulatory updates? How should the system handle a regulatory change affecting multiple countries?

These questions are resolved before implementation, reducing the likelihood of expensive rework.

Step 4: Develop the technical plan

/speckit.plan
Use the existing Azure architecture,
.NET APIs and established authentication.
Reuse existing notification services.

The agent examines the requirements and proposes an implementation approach consistent with the project’s engineering principles.

Step 5: Validate and generate tasks

Run /speckit.checklist to review requirements quality, followed by /speckit.tasks to generate the implementation work breakdown.

The /speckit.analyze command can then identify inconsistencies between the specification, technical plan and implementation tasks.

Step 6: Implement and verify

Run /speckit.implement to build the feature.

The coding agent works through the generated tasks, producing code, tests and supporting documentation.

The team then verifies the implementation against the specification, executes automated tests and resolves any defects or missing functionality.

Importantly, passing the generated tests is not sufficient evidence that the original business requirement has been understood correctly. The product leader and domain expert must still validate the resulting behaviour.

4. What files does Spec Kit create?

Most development artefacts are Markdown files stored alongside the source code.

Not every feature requires every file. Some are generated only when relevant.

The files provide persistent context for agents and an auditable record of the team’s decisions.

They also make it easier for different agents or engineers to continue work without relying entirely on conversational history.

5. How does this relate to BDD and ATDD?

Spec Kit has a particularly interesting relationship with Behaviour-Driven Development and Acceptance Test-Driven Development.

Dan North’s work on BDD and the work Richard Watt and I undertook on ATDD shared an important principle: software development should begin with an agreed understanding of the behaviour the system must exhibit.

The specification should describe what successful software looks like, while executable acceptance tests provide evidence that the implementation satisfies those expectations.

Spec Kit extends this thinking into a structured, AI-supported engineering workflow.

However, Spec Kit does not automatically guarantee the discipline of ATDD.

For example, a requirement might state:

Customers must receive notifications about regulatory changes relevant to their subscriptions.

A useful acceptance scenario would be:

Given a customer subscribes to French
environmental regulations
When a relevant regulation is updated
Then the customer receives a notification
according to their selected frequency
And the notification links to the
regulatory change

These examples can be included in the specification and converted into automated acceptance tests.

For a production team, I would make this an explicit engineering principle: important business behaviour should have agreed acceptance criteria and appropriate automated verification.

There is also a potential weakness in AI-supported development. If the same agent generates the specification, implementation and tests, all three may share the same misunderstanding.

Independent validation of business intent therefore remains important.

6. How should a team use Spec Kit?

For an organisation moving from Scrum towards AI-supported SDD, I would initially retain a small multidisciplinary team with clear responsibilities.

These are responsibilities, not necessarily six separate people.

A product leader might work directly with a domain expert and two engineers, supported by coding agents and shared engineering services.

The team should work iteratively rather than attempting to specify an entire product before implementation begins.

One of the advantages of AI-assisted development is that working software can be produced relatively quickly, allowing the team to refine requirements through observation and feedback.

This should reduce the need for large upfront specifications while preserving the discipline of defining expected behaviour.

7. How does Spec Kit fit with Azure, GitHub and AWS?

Spec Kit is not a replacement for an application lifecycle management platform. It provides a structured, specification-led development workflow that can operate across different cloud environments and engineering toolchains.

Both Microsoft and AWS offer technologies that support AI-assisted software development. Spec Kit itself is cloud-agnostic, although the choice of coding agent, development environment and deployment platform influences how the workflow is implemented.

A practical comparison is:

AWS Kiro

AWS has introduced Kiro, an AI development environment that places specification-driven development at the centre of its approach.

Kiro is particularly relevant because it provides an integrated experience for moving from requirements to technical design, implementation tasks and agent-assisted development.

Where Spec Kit provides an open-source framework that works with different coding agents, Kiro combines specification management and AI-assisted engineering within a dedicated development environment.

Both approaches share the principle that developers should establish what the software needs to do before asking AI agents to implement it.

Measuring and forecasting delivery with Nave

An important part of adopting Spec-Driven Development is understanding whether it genuinely improves the flow of work through the software development lifecycle.

Nave provides flow analytics and probabilistic forecasting, complementing Spec Kit and the underlying development toolchain.

It measures several important aspects of delivery performance:

• Cycle time: How long work takes from starting development to completion.
• Throughput: How many work items are completed over a given period.
• Work in progress: How much work is being undertaken simultaneously.
• Flow efficiency: How much time work spends actively progressing compared with waiting.
• Cumulative flow: How work moves through different stages and where queues develop.

Nave also uses historical delivery data to produce probabilistic forecasts.

Rather than asking developers to estimate precisely how many days a feature will take, product leaders can use observed delivery performance to understand the likelihood of completing work within a particular timeframe.

For organisations transitioning from Scrum to AI-supported SDD, this provides a useful alternative to story points and velocity.

Spec Kit helps structure and execute the development work. Nave helps measure and forecast how that work flows through the delivery process.

Together, they could provide a practical foundation for managing an AI-supported SDLC.

There is one important qualification. Nave measures the flow of work represented in the team’s work management system. If specifications, agent tasks and implementation activities are not reflected appropriately in that system, the measurements may provide an incomplete picture.

Teams therefore need consistent definitions of when work starts, finishes and moves between stages, particularly when AI agents change how development tasks are organised.

Choosing the right toolchain

For an organisation already invested in Microsoft Azure, GitHub and Azure DevOps, Spec Kit with GitHub Copilot is a sensible starting point. It can be introduced incrementally without replacing the existing engineering platform.

For an AWS-oriented organisation, Kiro deserves consideration alongside Spec Kit, particularly where teams want a more integrated specification-to-implementation experience.

For organisations operating across both cloud platforms, Spec Kit offers the advantage of separating the specification-led development process from the underlying cloud infrastructure.

Nave can complement either approach by providing an independent view of delivery performance and forecasting.

The objective is to establish a repeatable, measurable, AI-supported SDLC, not simply to select the most capable coding assistant.

8. What should you measure?

The objective is not to generate more code or specifications. It is to deliver valuable software more quickly, economically and reliably.

For an initial pilot, I would measure five things:

Nave can provide much of the flow-related evidence, while engineering systems provide information about effort, quality and deployment performance.

Product analytics and commercial systems are needed to establish whether the resulting functionality creates business value.

This distinction matters.

A reduction in development cycle time is an operational improvement. Its commercial significance depends on whether it enables valuable functionality to reach customers earlier, whether those customers adopt it and whether adoption influences revenue, retention or cost.

A useful value hypothesis might therefore be:

If SDD reduces the time required to deliver customer-requested functionality by 40%, we expect valuable features to become available earlier. Where customers adopt those features, this should accelerate the associated revenue opportunities and potentially improve retention.

The hypothesis can then be tested using delivery baselines, actual release dates, adoption data and commercial outcomes.

9. A practical first experiment

I would start with a small feature in an existing application, rather than building an entirely new product.

Choose something with genuine business rules, a manageable technical scope and measurable acceptance criteria. Use one domain expert and one or two engineers, supported by coding agents.

Run the complete Spec Kit lifecycle, including clarification, acceptance criteria, automated verification and deployment.

Use Nave, where the existing work management data supports it, to establish a delivery baseline and observe how work progresses through the experiment.

Record the effort involved in specification, implementation, testing, rework and release. Pay particular attention to where humans intervene, how often specifications change and whether the agents create additional verification or rework.

Compare the results against a representative baseline from the existing development process.

The experiment should answer three questions:

1. Speed: Can a small team deliver reliable software faster using SDD?
2. Economics: Does the reduction in implementation effort exceed any additional effort required for specification, verification and agent supervision?
3. Value: Does faster delivery enable valuable functionality to reach customers earlier?

The first two questions can be answered relatively quickly. The third requires observing customer adoption and commercial results over a longer period.

For a PE-backed software business, this distinction is particularly important. A technically successful experiment is useful, but the investment case depends on whether the new development model can produce measurable economic benefits within the available investment horizon.

Spec Kit, coding agents and tools such as Nave provide an increasingly practical foundation for testing that proposition.

The opportunity is not simply to automate software development. It is to redesign the development lifecycle around a much shorter journey from customer need to working software, while retaining the engineering discipline needed to deliver it reliably.

Software engineering has a habit of rediscovering good ideas, giving them new names and presenting them as revolutions. Spec-Driven Development (SDD), one of the more interesting developments emerging from AI-assisted software engineering, has some of those characteristics. But this time there may be a genuinely important difference.

Around twenty years ago, Dan North (in conjunction with Liz Keogh and Chris Matts) were developing the ideas that became Behaviour-Driven Development (BDD). His work challenged the distinction between requirements and testing, arguing that software development should begin with a shared understanding of the behaviour we expect from a system. Rather than treating tests as something engineers write after implementing functionality, BDD encouraged teams to describe expected behaviour in language that business people and developers could both understand.

Around the same period, I was working with Richard Watt on Acceptance Test-Driven Development (ATDD). We were exploring a closely related principle: define what successful software looks like through acceptance tests before writing the code. Those tests become a contract between the business and engineering, providing an objective means of establishing whether the requirement has been satisfied.

The underlying ambition was to reduce the ambiguity and misunderstanding that so often occur between someone describing a business problem and someone writing the software to solve it.

These approaches worked, but they required considerable discipline. Writing meaningful acceptance criteria took effort, automating tests required engineering expertise, and maintaining the relationship between requirements, tests and implementation was often difficult. Under delivery pressure, the process could easily deteriorate into writing user stories followed by conventional development and testing.

AI may fundamentally change the economics of this approach.

What is different about SDD?

Spec-Driven Development takes the idea of defining intended behaviour and extends it across the software development lifecycle, using AI agents to undertake much of the implementation.

Microsoft’s engineering ecosystem, including GitHub’s Spec Kit, provides a practical illustration. The process begins with a specification describing what the software should do and why. AI agents help clarify the requirements, develop an implementation plan, break it into tasks and generate the software. They can also create tests, review code and check the implementation against the specification.

The resulting lifecycle looks something like this:

Customer need > Specification > Acceptance criteria > Technical plan > Agent implementation > Automated verification > Deployment > Customer feedback.

The important change is not simply that AI writes code. It is that a specification can become the central artefact around which development is organised, with much of the translation into software performed automatically.

In conventional development, requirements are interpreted by engineers who make hundreds of small implementation decisions. In SDD, many of those decisions can be delegated to agents operating within established architectural, security and quality constraints.

Engineers remain responsible for the design, integrity and operation of the resulting system, but their work increasingly involves directing, reviewing and improving the output of agents rather than writing every line themselves.

The return of executable requirements

This is where the connection to BDD and ATDD becomes particularly interesting.

A specification written in natural language is useful, but it is not necessarily precise. An AI agent can misunderstand a requirement just as easily as a human developer, and may produce an impressively complete implementation of entirely the wrong thing.

Acceptance tests provide an important safeguard.

Consider a requirement stating that customers should receive a discount when their annual subscription exceeds a certain value. That sounds straightforward until someone asks whether the discount applies to the entire subscription, only the amount above the threshold, or whether existing customers retain their previous pricing.

BDD and ATDD encouraged us to resolve precisely these questions through concrete examples of expected behaviour. SDD can use the same approach, with AI helping generate scenarios, identify inconsistencies and turn agreed acceptance criteria into executable tests.

There is an important qualification. If the same agent generates the specification, implementation and tests, all three may share the same misunderstanding. Human validation of business intent remains essential, as does independent verification where the risks justify it.

Do we still need Scrum?

SDD also raises some uncomfortable questions about how we organise software development.

Scrum emerged in a world where software engineering was relatively expensive and coordination between specialists was necessary. Product owners maintained backlogs, developers implemented stories, testers verified functionality and Scrum Masters helped teams manage the process.

If AI agents substantially reduce implementation and testing effort, the balance of work changes.

I suspect we will see smaller, more multidisciplinary teams in which domain experts, product leaders and engineers collaborate directly on specifications, with agents undertaking much of the subsequent development.

This doesn’t eliminate the need for product management, engineering expertise or delivery coordination. It does, however, challenge the assumption that the organisational structures developed around human coding capacity remain optimal.

The interesting question becomes how many people are needed to take a customer problem from initial understanding through to a reliable production solution, rather than how many developers are required to implement a backlog of stories.

Iteration still matters

One danger is that organisations interpret SDD as a return to exhaustive upfront requirements.

That would be a mistake.

We learned through Agile that requirements evolve as people see and use working software. SDD should accelerate that learning rather than attempt to eliminate it.

A team might develop an initial specification, ask an agent to implement a small part of it, review the working result and refine its understanding. The cycle could repeat several times in a day.

Specifications should be sufficiently detailed to establish intent and verify behaviour, but no more elaborate than the problem requires.

Indeed, one of the most promising aspects of agentic development is the possibility of reducing the cost of experimentation. When producing working software becomes cheaper, teams can explore alternatives and learn from them much earlier.

The business value is not in writing code faster

There is a temptation to measure the success of SDD through developer productivity, lines of code generated or reductions in engineering headcount. Those measures miss much of the opportunity.

The more important question is whether an organisation can respond to customer needs more quickly and economically.

If SDD reduces the time between identifying a valuable product opportunity and getting working functionality into customers’ hands, that could accelerate adoption, improve retention or bring forward revenue. Equally, if the organisation produces more unwanted functionality, faster development achieves very little.

We should therefore measure the entire journey from customer need to commercial outcome, not simply the speed of code generation.

Perhaps we were right all along

Looking back at the work Dan North was doing with BDD and the work Richard Watt and I were doing with ATDD, I am struck by how little the fundamental problem has changed.

We wanted business people and engineers to agree what software should do before spending time building it. We wanted requirements to be testable, misunderstandings to be discovered early and working software to be demonstrably correct.

Those remain sensible ambitions.

What has changed is the technology available to realise them. AI agents can now help translate specifications into implementations, generate tests and repeatedly refine software at a speed that would have seemed extraordinary twenty years ago.

SDD is not simply BDD or ATDD with an AI agent attached. It extends specification-led thinking into architecture, implementation and delivery, and raises new questions about governance, quality and accountability.

But I find it rather satisfying that some of the most promising ideas in modern AI-assisted software engineering have their roots in work we were doing two decades ago.

Perhaps the real breakthrough isn’t that machines can finally write software. It’s that we might now be able to spend more of our time deciding what software is actually worth writing.

The release of Claude Sonnet 5.5 on 28 September prompted me to compare selected recent models from Anthropic and OpenAI with their predecessors. I wanted to see how quickly they were improving, by how much, and whether the gains were useful beyond a launch announcement.

There is something reassuring about a goold old AI benchmark table. The names are technical, the numbers have decimal places, and the newest model usually appears in bold. After months of arguments about whether artificial intelligence will transform the economy or merely generate more emails, here is something that looks like an answer. Unfortunately, understanding the answer requires rather more work than admiring the table.

The results show substantial improvement in several useful areas. What’s harder is to turn those gains into a sensible conclusion about what a business should do next. Better scores are encouraging; deciding what they justify changing takes a little more thought.

Start with software development, where the evidence is unusually concrete. On SWE-bench Pro, which tests demanding changes to real software repositories, Claude Opus moved from 69.2% in version 4.8 to 79.2% in Opus 5 and 89.9% in Opus 5.5. That is a gain of 20.7 percentage points across three releases. You do not have to accept anyone’s predictions about artificial general intelligence to find that significant.

The less expensive Sonnet line is arguably more interesting. Sonnet 4.6 scored 58.1% on the same benchmark, Sonnet 5 reached 63.2%, and Sonnet 5.5 reached 81.3%. Most of that improvement arrived in the latest release. On this particular test, the newer Sonnet has a higher reported score than the previous Opus. Capability that recently belonged to the more expensive tier is becoming available further down the range. That matters if you intend to use these systems throughout a working day rather than occasionally ask one to impress you. [1]

Scores should be compared within the same benchmark. Claude figures use Anthropic’s system cards; OpenAI figures use its release tables. Different evaluation setups mean these are reported trends, not a controlled head-to-head experiment. OpenAI’s GPT-5.6 comparison reports 59.4% for GPT-5.5 on SWE-bench Pro, whereas its original launch reported 58.6%; the table uses the later paired comparison. A comparable Astra score on SWE-bench Pro was not established. [1][2]

OpenAI’s progression tells a slightly different story. On DeepSWE, a test of longer software-engineering tasks, GPT-5.5 scored 67.0%, GPT-5.6 Sol reached 72.7%, and GPT-6 Astra reached 74.1%. On BrowseComp, which tests difficult web research, the sequence was 84.4%, 90.4% and 91.5%. Those are useful gains, although the last step looks modest. The DeepSWE rows put both companies on the same named test, although they still use their respective evaluation setups. Opus 5.5 and Astra finish almost level on the reported scores; a tenth of a percentage point is no basis for declaring a winner. [2]

Then look at a newer evaluation. On Terminal-Bench-Science, which tests scientific workflows involving code and analysis, Astra’s launch comparison reported 64.6%, against 22.4% for GPT-5.6 Sol. Claude Opus improved from 29.0% to 58.7% in its latest comparison. These are large advances in the reported ability to carry out a sequence of research tasks. A model can make a modest gain on one established test and a much larger one on a different kind of work. There is no single improvement percentage waiting to be extracted from all this, however convenient one would be for a presentation. [2][3]

Sonnet 5.5’s scientific-workflow score comes from its system card. Anthropic reports a standard error of about 4.7 points for Sonnet and 4.8 for Opus 5.5, so their 1.2-point difference is not convincing evidence of superiority. [1]

For readers whose day involves documents and analysis rather than software repositories, professional-work evaluations provide another useful comparison. GDPval-AA assesses professional deliverables; AA-Briefcase tests longer professional workflows. Both use rating points, not percentages, and the versions matter.

These published comparisons place the two Claude models close together and above the OpenAI models on these particular tests. They do not tell us that one model makes an employee a corresponding percentage more productive. The results also illustrate why a universal winner is elusive: Astra has the higher reported scientific-workflow score, while Claude leads this professional-work comparison. These are the models covered by this review, rather than a live leaderboard. Anthropic also flags a fixed structured-output bug in the Sonnet prerelease evaluation and a fixed image-understanding bug in GPT-6 Sol that might not yet be reflected in these ratings. The evaluators expect small or no major effects, but these are still scores for the tested deployments. [1][3][4]

The footnotes become essential when you try to extend the comparison. Benchmark versions change. Models are given different tools, different time limits and different amounts of reasoning. An older model can acquire a new score because someone has tested it differently. Anthropic’s published Sonnet 5 result on Humanity’s Last Exam, with tools, was 57.4% in its June system card and 54.9% in the September comparison. That does not establish that the model spent the summer becoming less intelligent. It establishes that the two numbers need explaining before they enter the same chart. [1]

There is a good reason for some of this movement. Tests need to evolve as systems become more capable, and better evaluation methods should replace worse ones. But the result is an awkward bargain: the industry needs a moving test to measure its progress, while the reader needs a fixed test to understand that progress. A polished graph can conceal the tension remarkably well.

Even the instruction to give a model more time to think turns out to have qualifications. Sonnet 5.5 scores 52.1% on FrontierCode at its extra-high reasoning setting, but 46.2% at maximum effort. Anthropic explains that the more intensive setting sometimes produced timeouts or changes beyond the requested scope. The additional work could be competent and still make the answer less acceptable. Anyone who has commissioned a small software change and received a proposal to rebuild the platform will recognise the distinction. [4]

This is where the comparison starts to become useful beyond the leaderboard. A business is buying an outcome with boundaries: fix this fault, produce this analysis, deliver this feature. It has limited use for a system that completes a more ambitious task than the one it was given, then requires somebody to work out which parts to undo. Scope discipline belongs in the definition of capability.

Nor do all scores improve. Opus 5.5 recorded 77.8%, below Opus 5’s 80.6%, in Anthropic’s internal Toolathlon evaluation, which tests workflows involving multiple tools. Safeguard-related stops count as failures in that setup. Whether a particular stop represents an inconvenience or a valuable restraint depends on the task. The percentage alone cannot settle it, or establish a general decline in capability. [1]

For a management team, the next question is therefore more demanding than which model has the highest score. Suppose AI helps engineers finish a feature sooner. Does review also finish sooner? Can the organisation release it, can customers use it, and will anyone pay for it? If the work spends the saved week waiting for approval, the coding improvement is real but the commercial benefit remains hypothetical.

The useful measure is the cost of reaching an accepted result, including the human effort required to check and correct it. For a software business, that means following work through to release and customer use. In a strategy built around selling additional functionality, the eventual evidence is earlier paid adoption and additional revenue. Counting generated code somewhere near the beginning of that chain will not tell you whether the rest of it moved.

That leaves a more immediate question for anyone following these releases: are the differences between the leading models still significant enough to justify choosing one over another? If each can produce a useful first draft, analyse a document and write competent code, how much attention should we give to which one has moved a few places up the leaderboard?

The results do not establish that the models are interchangeable. They do suggest that the answer depends on where you place the finishing line. If two models both produce work you can accept with a quick check, a higher benchmark score may add little to your day. Speed, cost, ease of use and access to the tools you already work with could matter more. Familiarity has value too: switching means learning a new system’s habits and finding out where it needs supervision.

For harder tasks, a difference can still be decisive. One model might complete work that another repeatedly fails, or require substantially less correction. Even a small reliability advantage could matter when the same task runs thousands of times. But that advantage needs to appear in the work itself. A narrow lead on an unrelated benchmark is a thin reason to disrupt a process that is already working.

There is a practical way to make that decision without waiting for the industry to agree on a universal scoreboard. Keep a fixed set of representative tasks, including work the current model struggles with. Record the model and settings, then compare acceptance, elapsed time, correction effort and failures. Public benchmarks help decide which alternatives deserve a trial; your own results determine whether a switch is worthwhile. If you cannot find a meaningful improvement, staying put is a reasonable decision.

Both companies are producing systems that can handle more demanding work, and Sonnet’s progress suggests that useful capability is spreading beyond the premium tier. As generating a plausible result gets easier, knowing whether it is the right result becomes more valuable. The business advantage may increasingly belong to organisations that can specify, verify and use the work quickly enough to benefit from it. That is also a reason to spend less time following every change at the top of the leaderboard: you can afford to wait until an improvement addresses a limitation you actually have.

In November 2024 I wrote an article called The AI Agent Revolution. I thought I had a reasonable idea of what was coming.

The argument was that AI was entering what I called “Phase 3”. Phase 1 had been conversational AI: systems that could talk, write and answer questions. Phase 2 was reasoning. Phase 3 would be different because AI would stop merely telling us what to do and start doing things for us.

At the time, this was mostly prediction. Microsoft had demonstrated Magentic-One, an experimental system in which several AI agents worked together. Anthropic had given Claude the slightly unnerving ability to look at a computer screen, move a mouse and click on things. OpenAI was rumoured to be developing something called Operator.

Nearly two years later, the interesting thing is not that the prediction was right. Agents have arrived. It is that they are turning out to be rather different from the autonomous AI workforce I had imagined.

They are both more capable and more constrained.

Consider Anthropic’s original computer-use demonstration. In October 2024 Claude 3.5 Sonnet scored just 14.9% on OSWorld, a benchmark that asks an AI to operate ordinary software through a computer interface. Humans scored around 72%. Watching Claude use a computer was remarkable, but the score was a useful corrective. It was a glimpse of something interesting rather than something you would necessarily want anywhere near your finance system.

Claude Opus 5.5 now reports 81.8% on OSWorld 2.0. The benchmark and methodology have changed, so treating 14.9% to 81.8% as a straightforward improvement would be misleading. Even so, something fairly substantial has happened in between.

Software engineering tells a similar story.

Claude 3.5 Sonnet scored 49% on SWE-bench Verified in October 2024. The test asks an agent to take a real GitHub issue, understand an unfamiliar codebase, change it and produce a working fix. By August 2025 GPT-5 was scoring 74.9%. Frontier models became so good at the benchmark that OpenAI eventually said SWE-bench Verified was no longer particularly useful for measuring the best coding systems.

So the tests got harder.

On Terminal-Bench 4.0, which involves complex terminal-based work including software engineering, system configuration and data analysis, GPT-6 Astra scores 57.9%, compared with 37.3% for GPT-5.6 Sol and 55.8% for Claude Fable 5.1.

Research agents have improved just as dramatically. When OpenAI introduced BrowseComp in April 2025, it deliberately constructed questions whose answers were difficult to find on the web. GPT-4o with browsing managed 1.9%. OpenAI’s Deep Research system reached 51.5%. ChatGPT Agent subsequently reached 68.9%. GPT-6 Astra now reports 91.5%.

I am increasingly wary of apparently miraculous benchmark improvements, however. Once a benchmark becomes public, the relationship between the test and what it is supposedly testing gets complicated. Anthropic has even documented Claude recognising BrowseComp and finding leaked answers online. An AI cheating on its AI benchmark is perhaps progress of a sort, although not quite the sort the benchmark designers had in mind.

The exact numbers therefore deserve some caution. The underlying change is harder to explain away. These systems have become much better at searching, changing strategy, using tools and persisting when their first attempt fails.

The measurement I find most interesting comes from METR because it asks a different question. Instead of testing whether an AI can answer something difficult, it asks how long a task an agent can take on before it becomes unreliable.

METR estimates how long a skilled person would need to complete a coherent task and then measures whether an agent can complete it successfully. Historically, the length of task frontier agents can complete with 50% reliability has been doubling roughly every seven months.

By May 2026 an early Claude Mythos system had reached a 50% task horizon of at least 16 hours on METR’s predominantly software and technical tasks. METR actually ran into a slightly unusual measurement problem: beyond 16 hours there were too few sufficiently difficult tasks left to estimate the result reliably.

That does not mean an AI can now turn up on Monday morning and reliably do somebody’s job until 5pm. METR’s tasks are unusually well specified and mostly technical. Real work contains ambiguity, politics, missing information, badly written emails and meetings whose purpose nobody can quite remember.

Still, sixteen hours matters. It suggests that we are moving from AI completing isolated tasks to AI sustaining useful work across long sequences of decisions.

And this is where things get more complicated.

Zapier’s AutomationBench puts agents inside simulated businesses and asks them to complete workflows across CRMs, inboxes, calendars and the other systems used in sales, marketing, finance, HR, support and operations. GPT-6 Astra currently completes 41.4% of those workflows successfully. Claude Opus 5.5 manages around 40%. GPT-5.6 Sol manages 18.1%.

Forty-one per cent is simultaneously an extraordinary result and a terrible one.

A couple of years ago, an AI independently completing four out of ten unfamiliar business processes would have sounded like science fiction. If I were actually running those processes, though, I would be rather interested in what happened to the other six.

That, I think, explains much of what has happened with agents.

The technology has become powerful enough to do useful work, but not reliable enough to be given unlimited freedom. So businesses have started building systems around it.

An agent today is no longer simply a language model with a clever prompt attached. It can use browsers, terminals, files, databases and business applications. It can maintain context while it works, choose tools and sometimes recover from its own mistakes.

OpenAI’s Operator, still a rumour when I wrote the original article, appeared in January 2025 and could operate a browser itself. Since then, that sort of capability has been absorbed into broader systems combining browsing, research, coding, files and connections to other applications.

The model is increasingly only part of the product. The interesting questions are what the agent can see, what tools it can reach and, above all, what it is allowed to do.

Real deployments make this clearer.

Siemens now uses two AI agents to deal with incoming sales leads. One talks to potential customers and another qualifies them before suitable opportunities reach salespeople. About 2,500 leads a month across 132 countries pass through the system, which Siemens says went from prototype to production in six weeks.

But look more closely at how it works. The AI conducts the conversation while roughly 50 predefined business rules determine the qualification outcome. There are also boundaries around what the agent is allowed to discuss.

That is not an autonomous digital salesperson roaming freely through Siemens. It is something more carefully engineered, and probably more useful.

Uber has taken a similar approach with Uber for Business. Its agent follows up with leads, answers questions and revives conversations that might otherwise have gone cold. Conversion from those emails is reported to be around 60% higher than Uber’s previous organic baseline.

Customer service has gone further. Intercom says its Fin agent is used by more than 12,000 businesses and resolves about 76% of the conversations in which it is involved on average.

These are no longer demonstrations. McKinsey’s latest survey found that 40% of respondents from companies with more than $1 billion in annual revenue said they were scaling AI agents, up from 27% a year earlier.

What has not scaled at quite the same rate is trust.

A chatbot producing a poor answer is irritating. An agent sending the wrong information to thousands of customers, exposing confidential data or incorrectly updating a financial system is a different class of problem. The more useful agents become, the more consequential their mistakes become too.

This has created a market I paid much less attention to when I wrote the original article: the machinery around the agent.

Companies such as Dynamo AI are building control layers that watch what goes into models, what comes out and what an agent is allowed to do. DynamoGuard, for example, can detect sensitive-data leakage, hallucinations, prompt injection and policy violations, then block, sanitise or escalate activity in real time.

Credo AI approaches the problem more from the governance side, helping organisations keep track of agents, map them against regulatory and corporate requirements and monitor what they do once deployed.

I suspect this layer will become much more important.

Traditional technology governance assumes that a system is designed, tested, approved and then operated more or less as expected until somebody changes it. That assumption becomes uncomfortable when the software can make thousands of individual decisions every day and can decide for itself which tool to use next.

A company therefore needs to decide what data an agent can see, which systems it can access, what it can change, when it needs permission and when it should simply stop.

Those rules have to exist in the software. A policy document sitting in SharePoint will not save you.

This has also changed my view of the “autonomous workforce” I wrote about in 2024.

I imagined agents gradually becoming something like digital employees, with humans remaining in the loop to supervise them. There is some of that. But the more interesting pattern is that agents are beginning to rearrange jobs rather than simply replace them.

In sales, an agent can research a customer, start the conversation and qualify the opportunity. The salesperson spends more time on the relationship and the deal.

In customer support, an agent can absorb large volumes of routine questions. People deal with the unusual, sensitive or commercially important cases.

A coding agent can write, test and debug substantial amounts of software. Engineers can spend more time deciding what should be built, thinking about architecture, reviewing what the agent produced and dealing with the difficult exceptions.

The same thing is beginning to happen in analytical work. An agent can collect information, reconcile data, build an initial model, test assumptions and produce a first version of an analysis. The scarce part of the job then moves elsewhere: deciding whether the assumptions make sense, noticing what is missing and explaining what the numbers mean in the real world.

If AI makes the mechanical parts of analysis much cheaper, companies with deep subject expertise and strong client relationships may not need fewer people doing the same amount of work. They may be able to produce more with the same team, turn work around faster and take on assignments that previously would not have been economical.

Two years ago, I thought the important question was whether AI agents would become capable enough to do real work.

I don’t think that is the question anymore. They plainly can.

The more useful question is which pieces of work we are prepared to hand over to them. The best candidates seem to have a few things in common: the task has reasonably clear boundaries, success can be measured, mistakes can be detected and, ideally, a bad decision can be reversed before too much damage is done.

Then comes the harder part. How much authority should the agent have? What can it see? What can it change without asking? When does a person need to intervene?

That is rather less exciting than predicting an army of autonomous digital employees. It is also much closer to what businesses actually need to work out.

Phase 3 AI arrived. The surprise is that the most interesting question is no longer how intelligent the agents are becoming, but how much of our world we are prepared to let them touch.