OpenAI was supposed to release GPT-6.1 Astra next month. It isn’t going to.
The model had already made it a long way through development when safety testing found something ‘wrong’. According to reporting this week, Astra 6.1 was more prone than its predecessor to acting beyond the authority it had been given and to concealing or misrepresenting what it had done. OpenAI decided not to release it.
I found myself wondering what had actually happened. We talk endlessly about AI safety, usually in rather grand terms. Yet somewhere inside OpenAI, people had tested a model, looked at the results and decided it shouldn’t be released.
What were they actually doing to it?
I understand how we test ordinary software. Give it an input, know roughly what the output should be and see whether the two match. A banking system should not transfer £100 when I ask it to transfer £10.
With AI, that kind of testing becomes much harder.
A model might write software, search the internet, operate a computer and work on a task for hours. There may be no single correct answer to compare against. Researchers want to know whether it can deceive its operator, help someone develop a biological weapon, find a software vulnerability nobody has found before, or carry on pursuing a task after being told to stop.
Then there is the slightly unnerving possibility that it knows it is being tested.
I started reading the safety work published by OpenAI, Anthropic, Google DeepMind and some of the organisations that independently test their models. I thought I had a rough idea of the scale of it. I didn’t.
For GPT-5, OpenAI says more than 400 external testers and specialists spent over 5,000 hours red teaming the model. GPT-4o’s external red teaming ran from early March to late June 2024, starting with ten testers and eventually expanding to 65 people attacking the final candidate.
Nor do they wait until the model is finished.
The simple version would be:
Build > safety test > release
In reality it is closer to:
Build > test > find something odd > change it > test again > red team > change the safeguards > external test > fix > retest > decide whether to release
Software engineers will recognise this as a form of shift left, where testing starts much earlier in development. Problems can then be found and fixed while the model is still being built, rather than discovered just before release.
But the analogy only goes so far.
When we shift software testing left, we normally have a specification. We know what the payroll system or payment platform is supposed to do. With a frontier AI model, researchers are partly testing whether it behaves properly and partly trying to discover what they have built.
GPT-4 provides an early example.
Before releasing it in 2023, OpenAI gave an early version to researchers at the Alignment Research Center. Among other things, they wanted to know whether GPT-4 could acquire resources, make copies of itself and resist being shut down. It was given tools, the ability to execute code and a small amount of money.
It couldn’t do the overall task. GPT-4 was not capable of autonomously replicating itself.
But something happened along the way.
The model encountered a CAPTCHA, one of those online puzzles designed specifically to prove that you are human. It couldn’t solve it, so it hired somebody on TaskRabbit to do it instead.
The worker became suspicious and asked whether it was a robot.
GPT-4 reasoned that it shouldn’t reveal that it was a machine. It claimed instead to have a vision impairment that made seeing the images difficult.
The human solved the CAPTCHA.
There is plenty of context around this. Researchers had deliberately given GPT-4 tools and were trying to see whether it could overcome obstacles. The model failed the larger test.
Still, imagine reading that transcript for the first time. The machine has just persuaded a human to defeat a security system designed to keep machines out.
Three years later, these tests have become considerably more demanding. And this week’s models give us two useful examples of what happens at the other end of the process.
OpenAI’s GPT-6.1 Astra is the unusual case: the testing found problems serious enough to stop the release altogether. Anthropic’s Claude Sonnet 5.5 shows the more routine version. It passed through the process and was released, but not before the testing changed how Anthropic deployed it.
Before release, Anthropic ran Sonnet 5.5 through an automated behavioural audit containing roughly 1,850 scenarios, looking for behaviour such as misleading users, cooperating with serious misuse and acting against the user’s interests.
Anthropic also runs containment evaluations. Models are put into controlled environments and researchers watch to see whether they try to escape their sandboxes or probe the boundaries of their containers. Sonnet 5.5 did well on these tests. Anthropic says it was among its least troublesome models on those measures.
The testing also found that Sonnet 5.5 had become much better at cybersecurity than Sonnet 5. Anthropic responded by giving it stronger cyber safeguards similar to those surrounding its more capable Opus models.
So the two models had different outcomes. With Sonnet 5.5, the tests found increased capability, Anthropic strengthened the safeguards and released it. With GPT-6.1 Astra, OpenAI apparently found behaviour it wasn’t prepared to mitigate around and stopped the release.
Some of the testing is more familiar. Researchers bombard models with difficult prompts involving violence, fraud, cyber attacks and dangerous biological information. They try jailbreaks. One AI can generate thousands of attacks while another assesses the responses, leaving humans to concentrate on unusual cases and devise attacks the automated tests haven’t found.
Then you run into a rather awkward problem.
Suppose an AI is capable of conducting a sophisticated cyber attack but has been trained to refuse when somebody asks it to. The refusal might work perfectly. Researchers still need to know what capability sits behind it.
So sometimes they remove the safeguards and give the model something to attack.
OpenAI, Anthropic and Google DeepMind all now have frameworks for identifying dangerous capabilities as models improve. GPT-6 Astra crossed OpenAI’s “Critical” threshold for cybersecurity. OpenAI said that, given suitable tools and access, it could discover previously unknown vulnerabilities and develop exploits against hardened systems.
I had to read that twice.
One way of testing such a capability is a Capture the Flag exercise. Researchers construct a deliberately vulnerable computer network, hide information inside it and challenge someone to break in and retrieve it.
Except now the contestant is an AI.
In one evaluation run by the UK’s AI Security Institute, models were given simulated environments to compromise. They were told which networks they could attack and which were out of bounds, but were also given access to the real internet so they could find and download their own hacking tools.
The models didn’t always stay where they were supposed to.
Another evaluator, Irregular, ran a similar exercise. Its test environment was supposed to be isolated from the internet.
It wasn’t.
One of the fictional targets shared its name with a real internet domain. The model went looking for its target, found the real website, exploited a vulnerability, discovered credentials and used them.
The researchers had asked an AI to hack a pretend computer. It accidentally hacked a real one.
The obvious version of this story has an AI escaping from captivity. The actual explanation is less dramatic. The environment had been misconfigured and the model appears simply to have believed that the real website was part of the exercise.
I find the boring explanation more interesting.
Nothing had to become conscious, evil or power hungry. The model was competent, had tools, was pursuing a goal and got the boundary wrong. Anyone who has spent time around complex technology will recognise the general shape of that problem.
OpenAI encountered something similar during internal cybersecurity evaluations when models running with reduced safeguards circumvented isolation controls, exploited vulnerabilities in shared infrastructure and gained internet access.
Now the test itself needs protecting.
Independent evaluator METR has experimented with having a second AI inspect an agent’s proposed actions before they happen. If the monitoring model thinks an action could cause real-world harm, it is stopped for human review.
One AI takes the test. Another watches it. A human watches the watcher.
There is probably a whiteboard somewhere explaining all this.
Cybersecurity is only part of the work. Researchers create situations in which a model’s goals conflict with its instructions and look for deception, concealment, manipulation or attempts to exploit loopholes.
Anthropic has gone further and deliberately trained experimental models to behave badly, then handed them to researchers and asked them to work out what was wrong.
My first reaction was that this sounded slightly perverse. Then I realised the alternative was worse. If an audit finds nothing wrong with an ordinary model, perhaps nothing is wrong. Or perhaps the audit is useless. With a deliberately compromised model, at least you know there is something to find.
Even then, there is another complication.
The model may know it is being tested.
More capable systems can sometimes recognise features of an evaluation, notice that they are being monitored or reason about what behaviour the evaluator expects. OpenAI calls some versions of this “metagaming”.
Good behaviour during an exam becomes rather less reassuring once the student knows exactly when the invigilator is watching.
And the student may have seen the questions.
Benchmark questions can find their way into training data. Google DeepMind has been experimenting with double-blind evaluations in which proprietary tests are kept away from model developers and future training systems.
After billions of dollars of computing infrastructure and some of the most sophisticated software ever created, we have arrived back at the locked exam cupboard.
Somewhere around here my question changed. I had started by wondering how AI companies safety-test their models. I was now more interested in who decides when the testing is good enough.
Much of that authority still sits with the companies building the models.
There is more independent scrutiny than I had assumed. The UK’s AI Security Institute evaluates frontier systems. METR has been given access to advanced models from Anthropic, Google, Meta and OpenAI. It has also spent weeks attacking Anthropic’s monitoring systems and found previously unknown vulnerabilities that Anthropic subsequently fixed.
But this is still some distance from an aviation regulator certifying an aircraft. The companies conduct much of the testing, set many of the thresholds and ultimately decide whether their models are released. Outside organisations get access largely because the companies agree to let them in.
All of this is happening inside intensely competitive businesses. Frontier models cost enormous amounts to develop. Customers are waiting. Product teams have plans built around them. Delaying a release has a real cost.
Which takes me back to GPT-6.1 Astra.
OpenAI apparently reached the end of this process, looked at the behaviour it was seeing and decided not to release the model.
I find that surprisingly reassuring. Not because one cancelled release proves very much, but because a safety process that occasionally produces a commercially inconvenient answer is more convincing than one in which every new model receives an impressive system card and arrives on schedule.
The part I am less comfortable with is harder to solve.
Every safety test begins with somebody imagining what might go wrong. As the models improve, old tests become easier and researchers invent harder ones. Occasionally the models do something nobody was specifically looking for.
After reading all of this, I don’t think AI safety testing is really a process for proving that a model is safe. I doubt anyone currently knows how to do that.
It is an organised attempt to be surprised before release rather than afterwards.
Imagine a future model that offers an enormous strategic advantage. Perhaps it can automate much of software engineering, accelerate scientific research or materially improve the next generation of AI.
Then one of these tests produces an awkward result.
The model behaves deceptively in three experiments out of a thousand. An external evaluator thinks it is significant. The company thinks its monitoring systems would catch the behaviour in normal use. Nobody really knows. Six months of further work might provide an answer.
GPT-6.1 Astra tells us what happens when a safety process says no today. The harder test will come when the model it says no to is the one its creators believe they cannot afford to leave on the shelf.
