The release of Claude Sonnet 5.5 on 28 September prompted me to compare selected recent models from Anthropic and OpenAI with their predecessors. I wanted to see how quickly they were improving, by how much, and whether the gains were useful beyond a launch announcement.
There is something reassuring about a goold old AI benchmark table. The names are technical, the numbers have decimal places, and the newest model usually appears in bold. After months of arguments about whether artificial intelligence will transform the economy or merely generate more emails, here is something that looks like an answer. Unfortunately, understanding the answer requires rather more work than admiring the table.
The results show substantial improvement in several useful areas. What’s harder is to turn those gains into a sensible conclusion about what a business should do next. Better scores are encouraging; deciding what they justify changing takes a little more thought.
Start with software development, where the evidence is unusually concrete. On SWE-bench Pro, which tests demanding changes to real software repositories, Claude Opus moved from 69.2% in version 4.8 to 79.2% in Opus 5 and 89.9% in Opus 5.5. That is a gain of 20.7 percentage points across three releases. You do not have to accept anyone’s predictions about artificial general intelligence to find that significant.
The less expensive Sonnet line is arguably more interesting. Sonnet 4.6 scored 58.1% on the same benchmark, Sonnet 5 reached 63.2%, and Sonnet 5.5 reached 81.3%. Most of that improvement arrived in the latest release. On this particular test, the newer Sonnet has a higher reported score than the previous Opus. Capability that recently belonged to the more expensive tier is becoming available further down the range. That matters if you intend to use these systems throughout a working day rather than occasionally ask one to impress you. [1]
| Benchmark | Model sequence | First release | Second release | Third release | Overall gain |
|---|---|---|---|---|---|
| SWE-bench Pro | Claude Opus 4.8 to 5 to 5.5 | 69.2% | 79.2% | 89.9% | +20.7 points |
| SWE-bench Pro | Claude Sonnet 4.6 to 5 to 5.5 | 58.1% | 63.2% | 81.3% | +23.2 points |
| SWE-bench Pro | OpenAI GPT-5.5 to GPT-5.6 Sol to GPT-6 Astra | 59.4% | 64.6% | Not established | Not calculable |
| DeepSWE v1.1 | Claude Opus 4.8 to 5 to 5.5 | 59.0% | 68.8% | 74.2% | +15.2 points |
| DeepSWE v1.1 | OpenAI GPT-5.5 to GPT-5.6 Sol to GPT-6 Astra | 67.0% | 72.7% | 74.1% | +7.1 points |
Scores should be compared within the same benchmark. Claude figures use Anthropic’s system cards; OpenAI figures use its release tables. Different evaluation setups mean these are reported trends, not a controlled head-to-head experiment. OpenAI’s GPT-5.6 comparison reports 59.4% for GPT-5.5 on SWE-bench Pro, whereas its original launch reported 58.6%; the table uses the later paired comparison. A comparable Astra score on SWE-bench Pro was not established. [1][2]
OpenAI’s progression tells a slightly different story. On DeepSWE, a test of longer software-engineering tasks, GPT-5.5 scored 67.0%, GPT-5.6 Sol reached 72.7%, and GPT-6 Astra reached 74.1%. On BrowseComp, which tests difficult web research, the sequence was 84.4%, 90.4% and 91.5%. Those are useful gains, although the last step looks modest. The DeepSWE rows put both companies on the same named test, although they still use their respective evaluation setups. Opus 5.5 and Astra finish almost level on the reported scores; a tenth of a percentage point is no basis for declaring a winner. [2]
Then look at a newer evaluation. On Terminal-Bench-Science, which tests scientific workflows involving code and analysis, Astra’s launch comparison reported 64.6%, against 22.4% for GPT-5.6 Sol. Claude Opus improved from 29.0% to 58.7% in its latest comparison. These are large advances in the reported ability to carry out a sequence of research tasks. A model can make a modest gain on one established test and a much larger one on a different kind of work. There is no single improvement percentage waiting to be extracted from all this, however convenient one would be for a presentation. [2][3]
| Terminal-Bench-Science 0.1 | Previous model | Newer model, cited comparison | Gain |
|---|---|---|---|
| OpenAI: GPT-5.6 Sol to GPT-6 Astra | 22.4% | 64.6% | +42.2 points |
| Claude: Opus 5 to Opus 5.5 | 29.0% | 58.7% | +29.7 points |
| Claude: Sonnet 5 to Sonnet 5.5 | Not established | 59.9% | Not calculable |
Sonnet 5.5’s scientific-workflow score comes from its system card. Anthropic reports a standard error of about 4.7 points for Sonnet and 4.8 for Opus 5.5, so their 1.2-point difference is not convincing evidence of superiority. [1]
For readers whose day involves documents and analysis rather than software repositories, professional-work evaluations provide another useful comparison. GDPval-AA assesses professional deliverables; AA-Briefcase tests longer professional workflows. Both use rating points, not percentages, and the versions matter.
| Model | GDPval-AA v2.1 | AA-Briefcase v1.1 |
|---|---|---|
| Claude Opus 5.5 | 1846 | 1822 |
| Claude Sonnet 5.5 | 1844 | 1811 |
| OpenAI GPT-6 Astra | 1542 | 1569 |
| OpenAI GPT-6 Sol | 1487 | 1483 |
These published comparisons place the two Claude models close together and above the OpenAI models on these particular tests. They do not tell us that one model makes an employee a corresponding percentage more productive. The results also illustrate why a universal winner is elusive: Astra has the higher reported scientific-workflow score, while Claude leads this professional-work comparison. These are the models covered by this review, rather than a live leaderboard. Anthropic also flags a fixed structured-output bug in the Sonnet prerelease evaluation and a fixed image-understanding bug in GPT-6 Sol that might not yet be reflected in these ratings. The evaluators expect small or no major effects, but these are still scores for the tested deployments. [1][3][4]
The footnotes become essential when you try to extend the comparison. Benchmark versions change. Models are given different tools, different time limits and different amounts of reasoning. An older model can acquire a new score because someone has tested it differently. Anthropic’s published Sonnet 5 result on Humanity’s Last Exam, with tools, was 57.4% in its June system card and 54.9% in the September comparison. That does not establish that the model spent the summer becoming less intelligent. It establishes that the two numbers need explaining before they enter the same chart. [1]
There is a good reason for some of this movement. Tests need to evolve as systems become more capable, and better evaluation methods should replace worse ones. But the result is an awkward bargain: the industry needs a moving test to measure its progress, while the reader needs a fixed test to understand that progress. A polished graph can conceal the tension remarkably well.
Even the instruction to give a model more time to think turns out to have qualifications. Sonnet 5.5 scores 52.1% on FrontierCode at its extra-high reasoning setting, but 46.2% at maximum effort. Anthropic explains that the more intensive setting sometimes produced timeouts or changes beyond the requested scope. The additional work could be competent and still make the answer less acceptable. Anyone who has commissioned a small software change and received a proposal to rebuild the platform will recognise the distinction. [4]
This is where the comparison starts to become useful beyond the leaderboard. A business is buying an outcome with boundaries: fix this fault, produce this analysis, deliver this feature. It has limited use for a system that completes a more ambitious task than the one it was given, then requires somebody to work out which parts to undo. Scope discipline belongs in the definition of capability.
Nor do all scores improve. Opus 5.5 recorded 77.8%, below Opus 5’s 80.6%, in Anthropic’s internal Toolathlon evaluation, which tests workflows involving multiple tools. Safeguard-related stops count as failures in that setup. Whether a particular stop represents an inconvenience or a valuable restraint depends on the task. The percentage alone cannot settle it, or establish a general decline in capability. [1]
For a management team, the next question is therefore more demanding than which model has the highest score. Suppose AI helps engineers finish a feature sooner. Does review also finish sooner? Can the organisation release it, can customers use it, and will anyone pay for it? If the work spends the saved week waiting for approval, the coding improvement is real but the commercial benefit remains hypothetical.
The useful measure is the cost of reaching an accepted result, including the human effort required to check and correct it. For a software business, that means following work through to release and customer use. In a strategy built around selling additional functionality, the eventual evidence is earlier paid adoption and additional revenue. Counting generated code somewhere near the beginning of that chain will not tell you whether the rest of it moved.
That leaves a more immediate question for anyone following these releases: are the differences between the leading models still significant enough to justify choosing one over another? If each can produce a useful first draft, analyse a document and write competent code, how much attention should we give to which one has moved a few places up the leaderboard?
The results do not establish that the models are interchangeable. They do suggest that the answer depends on where you place the finishing line. If two models both produce work you can accept with a quick check, a higher benchmark score may add little to your day. Speed, cost, ease of use and access to the tools you already work with could matter more. Familiarity has value too: switching means learning a new system’s habits and finding out where it needs supervision.
For harder tasks, a difference can still be decisive. One model might complete work that another repeatedly fails, or require substantially less correction. Even a small reliability advantage could matter when the same task runs thousands of times. But that advantage needs to appear in the work itself. A narrow lead on an unrelated benchmark is a thin reason to disrupt a process that is already working.
There is a practical way to make that decision without waiting for the industry to agree on a universal scoreboard. Keep a fixed set of representative tasks, including work the current model struggles with. Record the model and settings, then compare acceptance, elapsed time, correction effort and failures. Public benchmarks help decide which alternatives deserve a trial; your own results determine whether a switch is worthwhile. If you cannot find a meaningful improvement, staying put is a reasonable decision.
Both companies are producing systems that can handle more demanding work, and Sonnet’s progress suggests that useful capability is spreading beyond the premium tier. As generating a plausible result gets easier, knowing whether it is the right result becomes more valuable. The business advantage may increasingly belong to organisations that can specify, verify and use the work quickly enough to benefit from it. That is also a reason to spend less time following every change at the top of the leaderboard: you can afford to wait until an improvement addresses a limitation you actually have.
