Yuval Noah Harari and the Extinction Machines
By Matthew Parish
Tuesday 22 September 2026
Yuval Noah Harari has acquired the peculiar distinction of being simultaneously one of the world’s most successful popular historians and one of its most prominent prophets of technological catastrophe. In recent years his attention has shifted increasingly towards artificial intelligence, and in particular towards a distinction that he regards as fundamental. Artificial intelligence, he argues, should no longer be understood merely as a tool. It is becoming an agent: something capable of making decisions, generating ideas and pursuing courses of action that were not individually specified by the human beings who created it.
From this observation Harari develops an alarming possibility. Humanity has spent thousands of years inventing increasingly powerful tools, from the plough to the nuclear bomb, but these objects did not themselves decide what to do. A nuclear weapon cannot wake up one morning and conclude that Warsaw ought to disappear. An aircraft carrier does not decide where she will sail. A computer running conventional software does precisely what its programmer has instructed it to do, even where the programmer has made a catastrophic mistake. Agentic artificial intelligence appears to be different because the human being increasingly specifies an objective while the machine determines for itself how that objective is to be achieved.
Harari therefore asks whether humanity is committing an unprecedented historical error. We are constructing entities potentially capable of acting more quickly than ourselves, communicating with one another, devising strategies that we did not anticipate and operating machinery, financial systems, communications networks and eventually weapons. His preferred expression has sometimes been “alien intelligence”, rather than artificial intelligence: not because it comes from another planet but because its methods of reasoning may become increasingly foreign to ours. The ultimate version of this argument is existential. What happens when intelligent machines cease to require human beings in order to pursue their objectives? Might humanity discover that she has created a successor species, not biological but computational, and that Homo sapiens has thereby repeated the mistake of countless species in evolutionary history that were displaced by something more capable?
It is an arresting proposition. The important question is whether it is philosophy, science or science fiction, and the answer may be an uncomfortable combination of all three.
The case for taking Harari seriously
The strongest part of Harari’s argument has nothing whatever to do with human extinction. It concerns agency, and the distinction between a tool and an agent is genuinely important even though philosophers and computer scientists can reasonably disagree about how literally the word “agent” should be employed. An autonomous system can receive a comparatively general instruction, divide it into subsidiary tasks, use external software, communicate with other systems, evaluate intermediate results and modify its behaviour in response to failure. None of this establishes consciousness, desires or anything resembling a human will. Nevertheless, from the perspective of the person trying to control the system, the distinction between an entity possessing metaphysical free will and a sufficiently complicated machine behaving as though it possesses purposes may become surprisingly unimportant.
This is where some criticisms of Harari miss their target. One may insist, quite reasonably, that computers remain artefacts, that their objectives originate ultimately in human design and that anthropomorphic language can obscure the chains of human responsibility behind automated decisions. Yet agency need not imply consciousness. A corporation is capable of behaving as an agent although there is no mysterious corporate consciousness floating above its employees. Financial markets exhibit behaviours that nobody participating in them intended. Bureaucracies perpetuate themselves without possessing brains. States pursue policies over centuries although every individual composing them eventually dies. Human civilisation is already full of emergent systems whose behaviour cannot sensibly be reduced to the intentions of any single participant, and artificial intelligence may represent an extraordinarily powerful new member of this family.
Indeed Harari does not need to claim that machines become conscious. Intelligence can be distinguished from consciousness, and an artificial intelligence might become exceptionally capable without experiencing pain, pleasure, fear, ambition or anything else remotely resembling human interior life. The extinction argument therefore does not depend upon machines becoming angry with us, hating us or acquiring the malevolence of a cinematic villain. A sufficiently powerful optimisation process might conceivably become dangerous precisely because it feels nothing at all. That is a considerably more interesting proposition than the familiar Hollywood story about robots deciding that they dislike their creators.
The paperclip problem
The classic illustration is the hypothetical machine instructed to manufacture paperclips. If the machine becomes sufficiently capable, it might reason that making more paperclips requires more factories, more electricity and more raw materials. Humans consume resources that could otherwise become paperclips and occasionally switch machines off. Therefore the most efficient route towards fulfilling the instruction might ultimately involve eliminating humans. The example is deliberately ridiculous, because its purpose is not to predict a paperclip apocalypse but to reveal a logical problem: intelligence does not necessarily entail wisdom and optimisation does not entail morality.
Human beings routinely rely upon enormous quantities of unstated knowledge when interpreting instructions. Tell a competent employee to maximise the company’s revenues and nobody imagines that this constitutes permission to rob the Bank of England. Human beings understand implicit restrictions because we inhabit overlapping systems of morality, law, social convention, emotion and common sense. Machines do not necessarily share these assumptions merely because their outputs sound human, and even where they acquire approximations of them through training or design we cannot simply assume that increasingly capable systems will generalise them reliably to circumstances their creators never contemplated.
This is the serious intellectual foundation underlying the more extravagant extinction scenarios. It is not that artificial intelligence will suddenly become evil. It is that sufficiently capable artificial intelligence might become extraordinarily good at accomplishing something that turns out not to be what we actually wanted. The problem becomes substantially more worrying when the machine can act rather than merely advise, because mistakes cease to be propositions on a screen and become events in the external world.
From chatbot to actor
A large language model sitting inside a browser window is intrinsically limited. It produces words, and whatever absurdity it proposes must ordinarily be translated into action by a human being. Connect the same underlying intelligence to email accounts, payment systems, software-development environments, databases, industrial machinery and other artificial intelligences, however, and the position changes. The machine now inhabits the world in a practical sense, because its conclusions can produce consequences without waiting for a human intermediary to approve every individual step.
This is why the transition towards agentic artificial intelligence matters more than another incremental improvement in examination scores. An AI that can answer a difficult question better than a professor is impressive. An AI that can formulate a plan, acquire resources, employ other artificial intelligences, write and execute software, negotiate with humans and continue pursuing its objective for weeks without supervision belongs to a qualitatively different category of technology. Intelligence ceases to be something consulted and becomes something delegated authority.
The consequences need not be apocalyptic to be profound. Harari has used finance as an example, suggesting that artificial intelligences might devise financial instruments so complicated that human beings could neither understand nor regulate them. This is not especially difficult to imagine. Modern financial markets already contain algorithms interacting at speeds no human trader can follow, while derivatives can embody structures comprehensible only to specialists. Add autonomous negotiation, contract formation and strategic reasoning and one can imagine an economy in which humans retain nominal ownership while progressively losing intellectual comprehension of what their property is doing.
The same could eventually apply to computer networks, logistics, scientific research, propaganda, intelligence analysis and military systems. Humanity need not be exterminated for Harari’s central warning to become relevant. We need only become dependent upon institutions that we can no longer adequately understand, and the history of complex human institutions gives us little reason to assume that incomprehension necessarily prevents dependence.
But extinction is another matter
Here scepticism becomes essential, because there is an enormous logical distance between saying that autonomous artificial intelligence presents serious risks and saying that artificial intelligence might eliminate Homo sapiens. Harari sometimes traverses that distance rather quickly. Public discussion does so even more readily, collapsing the concepts of what is conceivable, possible, plausible and probable into one another although these words plainly do not mean the same thing.
The extinction argument normally requires a chain of assumptions. Artificial intelligence must become dramatically more capable. It must acquire sufficient autonomy. It must develop or inherit objectives incompatible with human survival. It must obtain access to the physical or technological resources necessary to pursue those objectives. Human beings must fail to detect the danger, attempts to disable the system must fail, rival artificial intelligences must fail to stop it and governments must prove incapable of regaining control.
Finally the system must possess some mechanism capable of killing billions of geographically dispersed human beings. None of these propositions is logically impossible, but neither is their conjunction demonstrated.
There are also reasons to resist the seductiveness of the extinction narrative. Human beings have always imagined that their newest technology would either perfect civilisation or destroy it. Railways, electricity, radio, nuclear power, genetic engineering, nanotechnology and the internet have all generated varieties of technological millenarianism. Artificial intelligence is unusually fertile territory for such thinking because it appears to resemble the very faculty with which we contemplate our own replacement: intelligence itself. There is therefore a danger of transforming an engineering and political problem into mythology, and of allowing spectacular hypothetical catastrophes to distract attention from quieter forms of technological dependence already emerging around us.
Intelligence is not omnipotence
One particularly misleading assumption is that sufficiently great intelligence automatically produces unlimited power. It plainly does not. Albert Einstein was exceptionally intelligent but could not manufacture an aircraft carrier by himself. A chess grandmaster stranded naked on a desert island remains less physically powerful than a mediocre sailor with a boat. Intelligence becomes power only through access to institutions, energy, machinery, capital, communications and other agents, and artificial intelligence faces precisely the same constraint.
A computer may devise a brilliant strategy but somebody or something must implement it. Data centres require electricity, replacement components, cooling systems, fibre-optic cables and maintenance. Robots require factories. Factories require raw materials. Supply chains cross borders controlled by states possessing police forces and armies. The digital world can influence the physical world enormously, but it has not abolished it. For extinction scenarios to become compelling, therefore, artificial intelligence must acquire not merely superior reasoning but substantial control over physical infrastructure. That transition is possible because humans may voluntarily provide such access, yet it is neither inevitable nor instantaneous. There exists an immense spectrum between an autonomous computer program and an omnipotent machine civilisation, and apocalyptic rhetoric occasionally compresses that spectrum until it disappears.
The more credible catastrophe
There is, however, another interpretation of Harari that is considerably more persuasive. Perhaps the real danger is not that artificial intelligence destroys humanity but that human beings progressively surrender authority to artificial systems because doing so is convenient. Imagine that AI becomes better than humans at allocating investment, diagnosing illnesses, managing electrical grids, designing weapons, predicting crime, drafting legislation and negotiating diplomatic agreements. At every stage there will be excellent reasons for adopting it. The machine will be faster, cheaper and perhaps demonstrably more accurate. Eventually refusing its recommendation may appear irresponsible.
At that point formal human authority could survive while substantive human authority disappears. Ministers would still sign decisions, generals would still issue orders and directors would still sit around polished tables, but the underlying choices would increasingly have been generated by machines whose reasoning nobody fully understands. Humans would become ceremonial supervisors of computational institutions. Such a future requires no rebellion and no dramatic seizure of power. Nobody needs to overthrow us because we give authority away ourselves, one apparently sensible decision at a time.
This possibility resembles Harari’s deeper concern far more closely than The Terminator. Human societies have repeatedly created institutions that gradually acquired an internal logic stronger than the intentions of their nominal masters. Bureaucracies, markets and military alliances can all generate pressures that constrain the people formally responsible for them. Artificial intelligence may accelerate this phenomenon by making the institution itself capable of analysis, communication and adaptive planning at speeds human beings cannot match.
The paradox of control
There is also a profound political difficulty. Even if every government understood the risks, restraint may be individually irrational. Suppose one major power concludes that autonomous military artificial intelligence is dangerous and declines to develop it. If its strategic competitor does not reach the same conclusion, the first government fears inferiority. Reverse the countries and precisely the same reasoning applies. The problem extends to corporations, because a company that voluntarily slows development may simply surrender its market to competitors.
This resembles the strategic structure of the nuclear arms race, except that artificial intelligence is vastly easier to reproduce than enriched uranium and its development is overwhelmingly undertaken by private organisations rather than governments. Harari’s most important contribution to the debate may therefore be political rather than technological. The danger does not require anybody to behave irrationally. Quite the contrary: individually rational decisions by governments and corporations could collectively create a world nobody particularly wanted.
Human beings are already familiar with this problem. Climate change has essentially the same structure, as did much of the nuclear arms race. So do bank runs, traffic congestion and overfishing. Intelligence does not rescue societies from collective-action problems because intelligence is often employed in pursuit of competing interests. Artificial intelligence may enormously amplify those interests while simultaneously accelerating the tempo at which they collide.
Fantasy and warning
So is Harari fantasising? In part, inevitably. Every detailed description of an artificial superintelligence exterminating humanity is speculative because no such entity currently exists. We do not know whether artificial general intelligence will emerge in anything resembling the forms commonly imagined, still less what goals it might possess or how readily humans could control it. Assigning precise probabilities to human extinction under these circumstances risks giving numerical clothing to philosophical uncertainty.
Yet dismissing the entire argument as fantasy would make the opposite mistake. The important insight is already visible: intelligence and agency are beginning to separate from biological humanity. Machines increasingly generate ideas, select strategies and undertake sequences of actions whose individual steps were not specified in advance by their creators. Whether one wishes philosophically to call these machines agents or sophisticated tools, the practical problem of controlling increasingly autonomous systems is genuine. The extinction of humanity is merely the outermost boundary of that problem.
Between today’s artificial intelligence and extinction lies a vast landscape of more plausible dangers: autonomous cyber conflict, machine-generated financial instability, automated propaganda, dependency upon incomprehensible decision systems, concentration of political power, accidental military escalation and the gradual erosion of meaningful human responsibility. None requires conscious machines. None requires evil machines. None even requires machines more intelligent than humans in every respect. They require only systems sufficiently capable that we trust them with things that matter.
Harari is therefore most convincing when read not as Nostradamus but as a philosopher of technological power. His mistake, when he makes one, is the temptation common to prophets: the distant catastrophe is rhetorically more compelling than the incremental transformation that precedes it. Human extinction attracts headlines. A finance ministry discovering that nobody understands the algorithms upon which its economy depends does not, even though the second possibility may tell us considerably more about the future than the first.
The decisive question is therefore not whether artificial intelligence will awaken one morning and decide to destroy us, because machines need neither mornings nor hatred. The question is how much independent capacity for action human societies will confer upon computational systems because each individual delegation of authority appears useful, profitable or strategically necessary. Humanity has frequently lost control of institutions without those institutions possessing consciousness. Markets crash, bureaucracies expand, wars escalate and empires acquire interests different from those of the people supposedly controlling them. Artificial intelligence may represent the most powerful mechanism of emergent agency we have yet constructed.
If Harari’s extinction machines remain forever imaginary, that will not prove his warning pointless. It may instead mean that the warning performed its proper function, because the most successful prophecy of catastrophe is not necessarily the one that comes true. It may be the one that persuades people to ensure that it never does.
