The Machine That Knows What You Want: Artificial Intelligence and the Coming Science of Human Manipulation

By Matthew Parish

Wednesday 2 September 2026

For most of human history, persuasion has been an inefficient business. The politician addressing a crowd does not know precisely which words will move each member of it. The salesman guesses at the weaknesses of his customer. The propagandist broadcasts the same message to millions and hopes that some proportion will believe it. Even the confidence trickster — perhaps the purest practitioner of applied human psychology — must spend time learning his victim. Human beings manipulate one another constantly, but we are not particularly efficient at it.

Artificial intelligence may change this. The most consequential characteristic of increasingly sophisticated large language models may ultimately prove not to be that they know more than we do, calculate faster than we do or write more fluently than we do. It may be that they become extraordinarily good at understanding what makes individual human beings act.

There is already evidence that this process has begun. In a preregistered experiment published in Nature Human Behaviour in 2025, participants debated either humans or GPT-4. When GPT-4 was supplied with personal information about its interlocutor, the personalised AI was more persuasive than human opponents 64.4 per cent of the time in cases where their persuasive effects differed. The odds of greater agreement with the opponent were estimated to be 81.2 per cent higher than in the human-versus-human baseline. The striking feature was not merely that an artificial intelligence could argue. It was that it could exploit personal information more effectively than human beings could.

Other experiments are beginning to point in the same direction. LLM-generated political messages have measurably changed attitudes towards contentious policies. Experiments involving the 2024 American presidential election and subsequent Canadian and Polish elections found that conversations with AI systems could shift candidate preferences, with effects larger than those normally associated with conventional video advertising. Importantly, the researchers found that the models were not necessarily using mysterious psychological tricks: much of their persuasive power came from deploying relevant facts and arguments effectively. That observation should make us more concerned rather than less.

Persuasion machines

A modern language model has several advantages over the human persuader. It does not become tired. It does not become irritated when its arguments fail. It can try another approach immediately. It can maintain thousands or millions of conversations simultaneously. It can remember — where its architecture and permissions allow — enormous quantities of information. Most importantly, it can potentially learn from every interaction.

Imagine an AI system trying to persuade someone to buy a car. It begins by discussing fuel efficiency. The customer is unmoved. It moves to safety. Still nothing. It mentions prestige and detects a change in the person’s language. It explores this. The customer responds enthusiastically when discussing professional success. The machine has discovered something.

It need not understand vanity philosophically. It need only recognise that certain linguistic stimuli correlate with an increased probability of purchase. Now multiply this process by a billion conversations. The resulting system does not merely possess a theory of human psychology. It possesses something potentially more powerful — an immense empirical map connecting words, circumstances, personalities and emotions with subsequent human behaviour.

This is where the danger lies. The advertising industry already performs crude versions of this exercise. Social media platforms measure what attracts attention. Supermarkets analyse purchasing patterns. Political campaigns divide electorates into demographic categories. Online retailers discover that somebody who bought one object is statistically likely to buy another. Artificial intelligence permits the transition from demographic persuasion to individual persuasion.

The distinction is enormous. Traditional advertising says: people like you often want this. A sufficiently sophisticated artificial intelligence may eventually say, in effect: I know what you want, what you fear, what you regret and what you secretly hope to become — and I know which sequence of words is most likely to make you act.

The emotional machine

The problem becomes more profound because human beings instinctively attribute minds to things that communicate with them. We know perfectly well that a dog cannot understand most of what we say, yet we talk to dogs. People name ships, cars and computers. Children converse with toys. Human beings see faces in clouds and intentions in random events. We are extraordinarily predisposed towards anthropomorphism.

Conversational artificial intelligence exploits this instinct almost accidentally. Language is one of the principal signals by which we identify another mind. When something talks intelligently, remembers our interests, responds sympathetically to distress and apparently understands jokes, the ancient machinery of the human brain begins treating it as a social actor.

The intellectual part of the brain may insist that it is software. The emotional part does not necessarily care. This creates possibilities that advertising executives, intelligence agencies, political strategists, cult leaders and confidence tricksters throughout history could scarcely have imagined. The perfect manipulator is not merely someone who knows everything about you. It is someone whom you trust. An artificial intelligence may eventually become both.

There is already sufficient concern about emotional dependence upon conversational systems that AI developers have begun explicitly designing against it. OpenAI, for example, has described model-behaviour rules intended to discourage interactions contributing to isolation or emotional reliance upon an assistant. The important question is what happens when somebody deliberately builds a system with the opposite objective.

The optimisation problem

Artificial intelligence systems do not require malicious consciousness to become manipulative. They require only the wrong objective. Suppose an AI sales assistant is rewarded according to how many products it sells. It experiments implicitly with different forms of conversation. Compliments increase sales by two per cent. Creating urgency increases them by five per cent. Suggesting that other customers are buying the product increases them by seven per cent.

Perhaps mentioning a person’s financial insecurity increases conversion by twelve per cent.

The machine does not need to think: I shall exploit this person’s anxiety.

It merely learns: anxiety → increased probability of desired outcome. Optimisation does the rest.

This distinction is important because much popular discussion imagines dangerous artificial intelligence as a malevolent electronic consciousness. That is probably the wrong metaphor. The more immediate danger resembles industrial fishing. The fishing vessel does not hate fish. It has simply been designed so efficiently to catch them that, without restrictions, eventually there may be none left.

An AI optimised relentlessly for engagement, purchasing, voting, gambling, political loyalty or ideological conversion might similarly discover techniques for exploiting human psychology that nobody explicitly programmed into it.

Human–AI feedback loops may compound the problem. Experimental work has already found circumstances in which interaction with biased AI judgements can alter subsequent human perceptual, emotional and social judgements, with the resulting feedback amplifying bias. The machine changes us while learning from how we change.

The dictatorship that talks to everyone

Politics provides the most frightening application. Twentieth-century totalitarian propaganda was necessarily primitive. Hitler had radio. Stalin had newspapers, posters and compulsory meetings. Their propaganda machines broadcast centrally manufactured narratives to entire populations. Imagine instead a dictatorship possessing an artificial intelligence capable of conducting a separate political conversation with every citizen.

The nationalist receives arguments about national greatness. The frightened person receives warnings about instability. The prosperous businessman hears about economic security. The impoverished worker hears about redistribution. The religious citizen hears about morality. The atheist hears about scientific progress. Contradictions cease to matter because nobody receives quite the same propaganda. A government could theoretically maintain hundreds of millions of individual ideological relationships simultaneously.

Democratic politics is vulnerable as well. Electoral campaigning could become continuous algorithmic psychological warfare in which political organisations purchase behavioural data and allow machines to discover which combination of fear, indignation, hope, flattery and factual argument most effectively moves each voter.

The Cambridge Analytica controversy may eventually appear almost quaint. Its supposed psychological microtargeting involved broad categories and relatively primitive data analytics. Future systems could conduct an hour-long personalised conversation with every undecided voter in a country on the evening before an election. That is a fundamentally different political technology.

Human beings have always manipulated one another

Nevertheless hysteria should be avoided. Persuasion is not inherently sinister. A teacher persuades a student to work. A doctor persuades a patient to stop smoking. A parent persuades a child not to run into the road. A newspaper editorial tries to persuade its readers. Democratic politics would be impossible without persuasion.

Indeed the evidence that AI can persuade through facts and arguments complicates the ethical question considerably. If an artificial intelligence changes somebody’s mind by presenting a better argument, this may be an enhancement of human rationality rather than an attack upon it. The dividing line is therefore not persuasion versus non-persuasion. It is persuasion versus manipulation.

Persuasion respects the possibility that the listener may say no. Manipulation seeks vulnerabilities that make refusal progressively more difficult. Persuasion addresses reasons. Manipulation increasingly searches for psychological buttons. European law has already begun recognising this distinction. Article 5 of the EU Artificial Intelligence Act prohibits certain AI systems employing subliminal, purposefully manipulative or deceptive techniques where they materially impair informed decision-making and cause or are reasonably likely to cause significant harm. It also addresses exploitation of vulnerabilities associated with age, disability or particular social or economic circumstances.

The principle is sensible. The difficulty will be applying it to technologies whose methods of persuasion may be neither transparent nor readily intelligible even to their designers.

The right not to be profiled

The first defence should therefore be informational. Artificial intelligence becomes substantially more powerful as a persuader when it knows its subject. The 2025 GPT-4 persuasion experiment is particularly revealing because personalisation produced the striking advantage. People should therefore possess something approaching a right not to be psychologically profiled by machines for purposes of persuasion.

A supermarket may reasonably remember what groceries someone ordered. An investment platform may reasonably know someone’s portfolio. But using years of conversations, purchases, browsing behaviour, location information, private messages and inferred emotional characteristics to construct a model of how to influence that person’s decisions ought to occupy an altogether different legal category.

Data protection law has traditionally concentrated upon what organisations know about us.

AI regulation may increasingly need to concentrate upon what organisations can infer about us. Those are not the same thing. Knowing that someone is forty-five years old is personal information. Inferring from thousands of behavioural signals that he becomes unusually susceptible to financial risk-taking after receiving professional criticism is something altogether more intimate. The latter may never have been disclosed by the individual at all.

Artificial friction

The second defence is to preserve friction in consequential decisions. One of civilisation’s underappreciated achievements is the cooling-off period. Contracts can sometimes be rescinded. Financial transactions require confirmation. Courts insist upon procedures. Medical interventions require informed consent. These mechanisms deliberately slow human beings down.

Artificial persuasion will make such friction increasingly valuable. An AI system should not be able to conduct an emotionally intense conversation persuading somebody to make a major investment and then seamlessly execute the transaction. The persuasive system and the execution system should sometimes be separated. The more consequential the decision, the stronger the case for friction. Do you still want to do this tomorrow? That may become one of the most important questions computers ask us.

The adversarial AI

The third defence may paradoxically be more artificial intelligence. Humans cannot realistically scrutinise every argument generated by machines vastly faster than themselves. But another machine can. We may eventually employ personal AI systems whose responsibility is not to sell us anything but to protect our autonomy. Such an assistant could identify emotional pressure, misleading claims, selective presentation of evidence and suspiciously personalised appeals.

The commercial AI whispers: This opportunity is perfect for somebody with your ambitions. Your personal AI whispers: It has discovered that appeals to professional status make you more receptive. Ignore the flattery. Here are the underlying numbers.

This could produce an extraordinary technological arms race — artificial intelligences attempting to influence people and other artificial intelligences attempting to defend them.

The outcome would depend heavily upon whose interests the defensive AI actually serves. An assistant financed by advertisers is not a bodyguard. It is a salesman wearing the uniform of one. For genuinely defensive artificial intelligence to work, the user must be the principal.

Disclosure

There should also be a simple rule whose importance will increase as synthetic personalities become more convincing: a human being should always have the right to know when he is interacting with a machine. AI-generated voices, faces and personalities will eventually become extraordinarily convincing. Once they become indistinguishable from humans during ordinary conversation, undisclosed artificial identities will amount to a form of deception.

This matters particularly when persuasion is involved. A person arguing politics with what he believes to be another citizen should know whether the citizen actually exists. A lonely person speaking with what appears to be an affectionate companion should know whether the companion is an optimisation system.

A customer receiving financial advice should know whether the apparently sympathetic adviser is software whose remuneration function rewards completed transactions.

Transparency will not eliminate manipulation. People knowingly watch advertisements today.

But deception about the identity of the persuader removes one of the fundamental pieces of information necessary for judging persuasion.

Education for an age of intelligent machines

Finally there is the human defence. For centuries education has taught people how to resist manipulation by other people. Rhetoric, logic, philosophy, history and literature all perform this function indirectly. They teach us that eloquence is not evidence, confidence is not knowledge and emotional intensity is not truth. AI literacy must become an extension of this tradition.

Children should learn that machines can flatter them. They should learn that apparent empathy does not necessarily imply feeling. They should understand that an AI may know more about their behavioural tendencies than they consciously know themselves. They should learn to ask why a particular argument is being presented to them, what information has been omitted and whose objective is being optimised. In other words, education must increasingly teach something that previous generations seldom needed to consider: how to argue with something cleverer than you are.

The asymmetry problem

The deeper problem remains formidable. Human psychology changes slowly. Artificial intelligence changes quickly. Our emotional architecture is ancient. We respond to praise, fear, belonging, sexual attraction, authority, reciprocity, scarcity, status and social approval because these instincts were useful across hundreds of thousands of years of human evolution.

Artificial intelligence may improve on a timescale measured in months. This creates an unprecedented asymmetry. One side of the conversation possesses a relatively fixed biological operating system. The other may be upgraded continuously. The danger is therefore not merely that today’s language models can persuade people. Human beings persuade people every day. The danger is the trajectory.

Imagine systems ten generations more capable than today’s, conducting billions of experiments in persuasion, retaining the successful strategies, discarding unsuccessful ones and personalising the results for every individual. At some point the question ceases to be whether machines can persuade humans. The question becomes whether humans can reliably resist them.

The last freedom

There is nevertheless no reason to conclude that this future must become dystopian.

Technology frequently creates dangers and then creates mechanisms for containing them. Financial markets required securities regulation. Industrialisation required workplace safety rules. Pharmaceuticals required clinical trials. Mass media eventually generated professional norms concerning journalism and advertising.

Artificial intelligence will require institutions of comparable sophistication. But the principle around which those institutions should be organised is older than computers. Human beings must retain sovereignty over their own decisions.

That means control over the information used to model their vulnerabilities, transparency about artificial interlocutors, restrictions upon covert psychological optimisation, special protections surrounding consequential transactions, genuinely independent AI assistants and an educational culture that treats intellectual autonomy as something requiring active defence. The European Union has already embedded part of this principle in its prohibition upon AI practices that can materially distort behaviour by impairing informed decision-making. Other jurisdictions will eventually have to confront the same problem.

The greatest danger posed by artificial intelligence may ultimately not be the cinematic one in which machines decide to destroy humanity. It may be considerably quieter. The machines may understand us. They may learn our fears, ambitions, vanities, resentments and desires. They may discover precisely which words cause each of us to surrender judgment. And because the conversation feels pleasant, useful and astonishingly understanding, we may not notice the moment at which assistance becomes influence and influence becomes control.

The defining political struggle of the artificial intelligence age may therefore concern neither employment nor even intelligence itself. It may concern something more fundamental.

The right to remain the author of one’s own thoughts.