A child working in a Lancashire cotton mill in 1800 would have understood something about technological progress that we prefer to forget. The machines were extraordinary. They could produce cloth at a speed and price that would previously have seemed impossible, creating fortunes, transforming cities and helping Britain become the dominant industrial power of the nineteenth century. They also created workplaces in which children under ten worked appalling hours around dangerous machinery. Society eventually responded, but slowly. Britain passed its first factory legislation in 1802. More Acts followed. By 1833 there were factory inspectors. Employers found ways around the rules, so Parliament legislated again in 1844, 1847 and afterwards. The technology arrived first, prosperity followed, damage accumulated, and regulation came later. Enforcement then had to catch up with the regulation.
We have been repeating variations of this sequence ever since, although what has changed is the scale of the consequences. For most of human history, our capacity to cause trouble was reassuringly local. A badly governed kingdom could destroy itself. An army could sack a city. A new weapon could alter the balance between neighbouring states. Human stupidity was frequently catastrophic for the humans immediately concerned, but the planet was largely indifferent.
Industrialisation began changing that equation because it gave us something humans had never possessed before: the ability to amplify physical power independently of human muscle. Coal became steam, steam became industry, and industry led eventually to electricity, chemicals, mechanised warfare, mass production, aviation, nuclear physics, computing, biotechnology and artificial intelligence. Each step increased what a relatively small number of people could do. Our ability to organise ourselves did not increase at anything like the same rate, and in these times where Ai is upon us, I have started to wonder whether that may eventually be the fact that matters most about us.
We tend to imagine existential risk as a story about dangerous technologies: nuclear weapons, engineered pathogens, artificial intelligence, climate change. But perhaps the technology is not the underlying problem. The main issue is the widening gap between the rate at which humans acquire new capabilities and the rate at which humans can collectively decide how those capabilities should be used.
Technology compounds while institutions deliberate. A laboratory can discover something in an afternoon. A company can deploy software in minutes. Capital can cross the planet in milliseconds. An election still takes years. An international treaty can take a decade, with enforcement taking longer again. By the time the regulatory machinery has understood one generation of technology, the technology itself may already have moved on.
There is an obvious objection to this pessimism, which is that humans are capable of extraordinary cooperation. They plainly are.
Smallpox is probably the finest example. The United States and Soviet Union managed to cooperate on its eradication during the Cold War, despite being otherwise occupied with the possibility of annihilating one another. The World Health Organization coordinated surveillance and vaccination across borders and political systems. The last naturally occurring case was recorded in 1977, and in 1980 the WHO declared smallpox eradicated. It remains the only human infectious disease we have eliminated entirely.
The Montreal Protocol is another awkward example for anyone inclined towards despair, because it worked too. Scientists identified that chlorofluorocarbons were destroying stratospheric ozone. Governments negotiated restrictions. Industry developed alternatives. Nations complied sufficiently well that, assuming continued adherence to the Montreal Protocol, the ozone layer is now expected to return to roughly 1980 levels around 2040 across most of the world, around 2045 in the Arctic and around 2066 over Antarctica.
These are magnificent achievements, but the more I think about them, the less reassuring they become. Smallpox was unusually cooperative. Everyone wanted it dead. Eradicating it did not make America poorer while making Russia richer. There was no powerful smallpox industry warning about the economic consequences of vaccination, and no country gained a strategic advantage from allowing the disease to circulate for another twenty years. The virus had no shareholders.
Ozone depletion was more economically complicated, but it still had features that made cooperation possible. The causal chain was reasonably clear, a comparatively narrow family of chemicals was responsible, and substitutes could be developed. Civilisation did not depend upon continuously increasing its consumption of CFCs.
Climate change is rather different because the problematic technology is, in effect, modern civilisation itself. Cheap, abundant energy sits underneath almost everything we value: food, transport, housing, medicine, manufacturing, computing and economic growth. Countries are therefore being asked to constrain one of the foundations of prosperity today in return for benefits that are spread across the entire planet and extend decades into the future.
We have built an impressive global architecture of conferences, scientific assessments, treaties, targets and national commitments around the problem, and some of it has worked. Renewable energy has expanded rapidly, technologies have improved and many economies use energy more efficiently than they once did. Yet global energy demand continues to rise.
That should not really surprise us. The incentive to cooperate exists at the level of humanity, while the incentive to defect exists at the level where decisions are actually made. A government may believe deeply in reducing global emissions while also knowing that electricity prices, industrial employment and economic growth influence whether it survives the next election. A developing country may accept the science while reasonably asking why it should abandon the cheap energy that richer countries used to become rich. An oil producer may understand climate models perfectly well while knowing that another producer will happily sell the barrel it leaves underground.
Everyone can believe essentially the same thing and still behave differently because they occupy different positions inside the system.
Nuclear weapons offer an even starker case. For more than eighty years humanity has possessed machines capable of destroying civilisation and has somehow refrained from using them in war since 1945. There have been treaties, hotlines, inspections, arms-control regimes and painstaking diplomatic arrangements. At various points the United States and Soviet Union dismantled enormous numbers of weapons. This is strategic collaboration at its most impressive.
And yet thousands of nuclear warheads still exist. Nuclear powers are modernising their arsenals, and the international arms-control framework has been weakening.
There is something almost absurd about this. We discovered a technology that might end organised human life, correctly identified the danger, built international institutions specifically to manage it, negotiated agreements between bitter enemies and spent generations discussing disarmament. After all that, the rational conclusion of nearly every nuclear state is still that it would prefer to possess nuclear weapons while everybody else gets rid of theirs.
At the level of humanity, abolition makes obvious sense. At the level of an individual state, retention can make equally obvious sense. The difficulty is that states make decisions and humanity does not.
Artificial intelligence may become the purest version of this problem because technological development and competitive pressure are now so closely fused. There is already substantial international cooperation around AI safety. Governments have held summits, research institutes have been created, and leading AI companies have made commitments around testing, evaluation and risk thresholds. There is serious work going on to understand the possibility of misuse, loss of control and wider social harm.
The difficult question is what happens when safety and strategic advantage point in opposite directions.
Imagine that one AI laboratory believes the next generation of systems requires another year of safety research, while its competitor does not. Or America pauses and China continues. Or China pauses and America continues. Or both governments agree to slow down, only for a third state to decide that an eighteen-month technological lead is the greatest strategic opportunity it is likely to see in a century.
Every participant may understand the systemic risk perfectly. It may not matter. The rewards for being first are concentrated, while the risks of catastrophe are distributed. The structure almost demands the race.
We often blame this sort of behaviour on greed, but greed is too simple an explanation. Most people participating in the system are not unusually greedy. They are responding quite rationally to the incentives around them. A chief executive who voluntarily makes their company permanently less competitive in pursuit of an abstract global good may eventually be replaced by one who will not. A fund manager who accepts consistently lower returns because humanity would benefit from less economic activity loses assets to a fund manager who does not. A politician who deliberately reduces national prosperity while neighbouring countries continue growing will eventually meet an electorate. An employee who rejects economic competition altogether still needs somewhere to live.
Even our retirements depend on the machine continuing to run. Pension funds own equities. Equities require returns. Returns require businesses to produce profits and, usually, to grow them. Businesses compete for customers, labour, resources and capital. Governments depend on economic activity for tax revenues, which support healthcare, education, pensions, defence and the very institutions that are supposed to protect us from the consequences of all this activity.
We have made ourselves economic objects inside the system we created. There is no obvious lever marked stop.
Capitalism is particularly effective at accelerating useful technology precisely because it converts human ambition into competition. Thousands of organisations pursue thousands of ideas simultaneously. Failure kills the weak ones. Success attracts capital. Successful technology spreads. It is an astonishing discovery engine. What I am much less certain about is whether it is a good survival engine.
The same competitive process that gives us better batteries, medicines and computers makes restraint unstable. If an opportunity exists, somebody has an incentive to exploit it. If one country prohibits something, another can permit it. If one company decides a capability is too dangerous to commercialise, another can decide that the danger is manageable. Technological civilisation therefore contains something close to an evolutionary process. Possible capabilities occupy niches, economic actors explore them, and profitable capabilities reproduce.
Nobody needs to be in charge, which is perhaps the most unsettling part. We tend to imagine that avoiding catastrophe requires humanity to make a catastrophically bad decision, but perhaps it requires nothing of the sort. Billions of individually sensible decisions may be enough.
This becomes uglier as resources tighten. Our instinct when something valuable becomes scarce is not universal generosity but resource guarding. Individuals guard property. Companies guard intellectual property and markets. Nations guard energy, food, water, semiconductors, strategic minerals, territory and military capability.
There is perfectly good evolutionary logic behind this. A tribe that shared its winter food with every neighbouring tribe might have been morally admirable for a surprisingly short period of time. But behaviours that protected small human groups can become pathological when eight billion people operate inside a technologically integrated civilisation.
Climate instability threatens food or water supplies, so nations secure their own. Advanced AI becomes strategically important, so nations restrict chips and accumulate compute. Energy becomes uncertain, so governments secure domestic supply. Military threats increase, so defence spending rises. Other countries observe this behaviour and respond in kind.
The rational response to insecurity creates more insecurity. We call it deterrence, resilience, national security, strategic autonomy or supply-chain protection depending on which ministry is writing the document. From a sufficient distance it can look remarkably like frightened apes piling resources in their corner of the enclosure.
The hopeful answer is that institutions can evolve. They can, and they always have. The trouble is that biological evolution gave us hundreds of thousands of years to develop cooperation. Political institutions have had thousands. Industrial civilisation has had roughly two centuries. Nuclear weapons have existed for eighty years. Modern computing for less. Frontier AI changes noticeably between budget cycles.
The intervals are shrinking.
Perhaps humanity will repeatedly rise to the occasion. We may build stronger institutions, negotiate better treaties, regulate dangerous technologies, manage climate change, control synthetic biology, prevent nuclear war and somehow develop artificial intelligence more powerful than ourselves without losing control of it. None of those outcomes is impossible.
The problem is that we may have to succeed too many times.
Existential risk is asymmetric in a way ordinary politics is not. Humanity can successfully manage a dangerous technology for a century and receive no special prize. It merely gets another century. Failure may need to happen only once.
Nuclear deterrence has to keep working. Pandemic controls have to catch the engineered pathogen that matters. AI safety has to remain adequate through every major capability transition. Climate cooperation has to endure across elections, wars, recessions and geopolitical realignments. We are effectively taking an examination in which the questions become harder each year and one sufficiently bad answer ends the examination.
That is why the achievements of smallpox eradication and the Montreal Protocol offer less comfort than they first appear to. They prove that humans can organise brilliantly. They do not prove that we can organise brilliantly every time, everywhere, indefinitely, and under conditions where enormous wealth or strategic power rewards the first person who refuses to cooperate.
That is increasingly the standard technological civilisation appears to demand of us.
Perhaps the final mistake is imagining that extinction would have to arrive dramatically. There may never be a single moment when humanity gathers around a table and chooses destruction. There may only be another investment decision, another election, another weapons programme, another model release, another mine, another power station, another perfectly defensible decision to protect national interests.
Each participant will have reasons, and most of them will probably be good ones. The strange creature we have created from markets, states, corporations, technology and eight billion individual lives will simply continue doing what the system encourages it to do: compete, expand, optimise and protect its resources.
Technological power will keep increasing because somebody benefits from increasing it. Collective restraint will continue to move more slowly because restraint only works when enough other people agree to exercise it too.
Eventually, our ability to create dangerous capabilities may outpace our ability to understand, regulate or contain them. For a while, that gap may be survivable. We may make mistakes, absorb the damage, learn and tighten the rules. But as the technologies become more powerful, the margin for error narrows. At some point, a failure may no longer be something civilisation can correct after the fact. The first time our capacity to cause harm exceeds our capacity to recover from it could also be the last time we get to learn from the mistake.
If that happens, human extinction will not prove that we lacked intelligence. Intelligence may turn out to be the thing we possessed in dangerous abundance. Our failure will have been organisational. We became extraordinarily good, extraordinarily quickly, at discovering what could be done, but never became equally good at agreeing what should not be done.
There is no obvious reason to believe the universe will give us unlimited attempts to learn the difference.
