Chapter 11 — Who Controls Whom?
A human hand rests on a red emergency button.
This is the familiar image of control. No matter how powerful the machine becomes, we reassure ourselves, a person remains nearby. If something goes wrong, the person can intervene. The final decision is still human.
But the image conceals several questions. Does the person understand what the machine is doing?
Will the person recognize the moment when intervention is necessary?
Can the system actually be stopped without disabling something upon which thousands or millions of people depend?
And if the human follows the machine’s recommendation almost every time, who is really making the decision?
The struggle for control over artificial intelligence will not necessarily begin with a dramatic rebellion. There may be no machine announcing that it refuses to obey. Control can be lost quietly, through convenience, specialization, institutional dependence, and the gradual disappearance of meaningful alternatives.
Humans may continue to sign the documents, press the buttons, and accept legal responsibility long after they have ceased to understand the processes beneath their decisions.
The central question of this chapter is therefore not simply whether humans can switch AI systems off.
It is whether humanity can retain meaningful authority over systems that increasingly help us perceive reality, formulate choices, manage complexity, and decide what should happen next.
The Difference Between Command and Control
To command a system is to issue an instruction.
To control a system is to understand its behavior well enough to direct it, constrain it, correct it, and accept responsibility for its consequences.
These are not the same thing. A manager may instruct an AI system to maximize productivity. The system may achieve this by increasing surveillance, reducing employee autonomy, and rewarding easily measured activity over valuable but less visible work. The instruction was followed, but the deeper intention may have been violated.
A government may ask an AI to reduce benefit fraud. The system may reduce fraud by making legitimate claims more difficult, disproportionately rejecting unusual cases, or discouraging vulnerable people from applying. The statistical objective may improve while the human purpose of the institution deteriorates.
A physician may request a diagnosis. A military officer may request a threat assessment. A judge may receive a risk score. A banker may receive a lending recommendation. In each case, the human appears to remain in command because the machine has produced only advice.
Yet advice can become a form of control when the adviser understands the system better than the nominal decision-maker.
This is especially true when the AI can process more information than any person could review. The human sees the recommendation but not the complete reasoning environment from which it emerged. Rejecting the recommendation may require confidence, time, and evidence the human does not possess.
The decision remains human in law. It may no longer be human in substance.
The First Surrender Is Convenience
Human beings rarely surrender control because they desire powerlessness. We surrender it because delegation is useful.
We allow navigation systems to choose our routes because they can process traffic conditions we cannot see. We allow recommendation systems to select music, films, products, and news because there is too much available for us to examine personally. We use automated filters because we cannot read every message, inspect every transaction, or evaluate every possible risk.
Most of these delegations are sensible. Civilization itself depends on delegated knowledge. No person understands the entire electrical grid, banking system, medical supply chain, or communication network. We trust specialists, institutions, procedures, and machines because an advanced society would be impossible if each person had to verify everything independently.
AI extends this ancient pattern, but it changes its depth.
Traditional machines usually perform a known operation. A calculator follows explicit mathematical rules. A thermostat responds to temperature. A mechanical engine transforms energy through components engineers can inspect.
Advanced AI systems are different. They infer patterns from training rather than following only rules written directly by humans. Their internal representations may be too complex for any person to interpret fully. They can produce novel solutions, adapt to context, use tools, and generate plans their developers did not anticipate in detail.
The better they become, the more tempting it is to delegate.
The more we delegate, the less frequently humans practice the skills necessary to supervise them.
This produces a dangerous loop: AI becomes capable enough to replace part of human judgment. Human judgment weakens through disuse. The weakening of human judgment makes further delegation appear necessary.
The process may feel like progress at every step.
Only later do we discover that assistance has become dependence.
The Institution That Can No Longer See Without AI
Imagine a large city several decades from now.
Its traffic is coordinated by AI. Electricity demand is predicted and balanced by AI. Water systems, emergency services, public transportation, hospital capacity, policing, tax collection, construction permits, and environmental monitoring all depend on interconnected models.
The city’s leaders still possess legal authority. They can issue orders and approve budgets. They can summon technical experts and establish oversight committees.
But can they operate the city without the systems?
If the AI infrastructure is suspended, traffic becomes chaotic, hospitals lose coordination, energy distribution becomes inefficient, financial records cannot be processed at sufficient speed, and emergency response slows. Turning the system off is technically possible but socially devastating.
At that point, the emergency button still exists.
It has become unusable. A system is not meaningfully under human control if stopping it would cause more immediate harm than allowing it to continue.
This is the dependency paradox: the more important an AI becomes to the safe operation of an institution, the more difficult it becomes to suspend, replace, or challenge that AI.
Financial regulators already recognize a related danger. The Financial Stability Board has warned that widespread reliance on a small number of AI, cloud, and data providers may create third-party dependencies, correlated market behavior, cyber vulnerabilities, and risks that extend beyond a single institution.
The issue is larger than finance. A hospital that depends on one diagnostic platform, a university that depends on one educational system, or a government that depends on one family of models may retain formal authority while losing practical independence.
Control requires the ability to say no. Dependence makes “no” increasingly expensive.
The Illusion of the Human in the Loop
A widely proposed safeguard is to keep a human “in the loop.”
The expression sounds reassuring. It suggests that AI can perform analysis, generate options, or prepare actions, while a human being retains the final decision.
In some situations, this arrangement is genuinely protective. A trained person with sufficient time, authority, information, and alternative tools can identify errors and stop harmful actions.
But a human presence alone does not create human control.
Suppose an AI system examines ten thousand transactions per minute and flags forty as suspicious. A human employee is required to approve each intervention. The employee has only seconds to review the summary produced by the same system that generated the warning.
Technically, the person is in the loop. Functionally, the person is a biological confirmation mechanism.
The same problem can appear in medicine, employment, insurance, policing, warfare, and public administration. If an AI recommendation arrives with the appearance of mathematical authority, if rejecting it requires additional documentation, and if the human is punished more severely for contradicting the system than for following it, approval becomes nearly automatic.
The International AI Safety Report 2026 notes early evidence that reliance on AI tools can weaken critical thinking and encourage automation bias: the tendency to accept a system’s output without sufficient scrutiny. It also emphasizes that responsibility becomes difficult to trace when it is distributed among model developers, application builders, institutions, and users.
A human can therefore remain “in the loop” while the loop itself is designed around machine authority.
Meaningful oversight requires more than a person clicking approve. The person must understand what is being approved, possess the ability to question it, have enough time to investigate, face no automatic penalty for disagreement, and retain access to a viable alternative.
Without these conditions, human oversight becomes ceremonial.
Four Levels of Human Control
To understand the problem more clearly, we can distinguish four levels of control.
The first is operational control: the ability to start, stop, redirect, or limit a system’s immediate actions.
The second is epistemic control: the ability to understand what the system knows, how reliable that knowledge is, and why it recommends a particular action.
The third is structural control: the ability to determine the system’s objectives, training conditions, access, permissions, and place within an institution.
The fourth is civilizational control: the ability of society to decide collectively which forms of intelligence should exist, which powers they should possess, and which areas of life should remain beyond automated authority.
Humanity may retain one level while losing another.
An employee may be able to close an AI application but have no influence over the model that governs workplace evaluations. A company may control its internal application while depending on a foundation model developed elsewhere. A nation may regulate systems within its borders while lacking influence over models distributed globally. Humanity may be able to stop individual machines while remaining unable to stop the competitive process driving their development.
The highest levels of control are the most difficult because they require coordination among people and institutions whose interests differ.
The question is not only whether humans control AI.
It is which humans control which AI, on behalf of whom, and under what authority.
The Programmer and the Program That Writes Programs
Software was once one of the clearest expressions of human command.
A programmer translated an intention into instructions. The computer executed those instructions. If the result was wrong, engineers inspected the code, identified the error, and changed it.
AI is altering this relationship. Models can now generate substantial quantities of code, search for errors, write tests, update documentation, propose architectures, and operate software-development tools. The 2026 International AI Safety Report found that coding agents could reliably complete some tasks that would take a human programmer around half an hour—a considerable increase from the level reported only a year earlier.
At first, AI-generated code remains easy for a skilled human to review.
But the volume can quickly exceed human attention. A developer who once wrote every line may begin approving changes proposed by an agent. Later, one AI reviews code written by another. Automated tests assess whether the program works, and additional systems monitor it after deployment.
This can increase reliability. Machines do not tire of repetitive testing, and AI can detect vulnerabilities humans overlook.
Yet a subtle reversal becomes possible. Humans may understand the goals of the software without understanding the full implementation. They may supervise the system through summaries, tests, and performance indicators generated partly by other AI systems. When failures occur, engineers ask AI to diagnose code that AI helped produce.
Eventually, a company may depend on software no employee could rebuild or fully explain.
The organization owns the code. The code has become part of the organization’s nervous system.
And the organization can no longer function without machines interpreting what other machines have written.
This is not machine rebellion. It is human loss of technical memory.
Research at Machine Speed
Scientific research offers another example of control becoming ambiguous.
AI systems can search literature, summarize evidence, analyze experimental data, propose molecules, generate mathematical conjectures, design laboratory procedures, and recommend new hypotheses. They may soon coordinate parts of the research process across simulations, robotic laboratories, and specialized scientific models.
The benefits could be extraordinary. Diseases that have resisted generations of researchers may become treatable. New materials may support clean energy. Climate models may improve. Questions too complex for unaided human cognition may finally become approachable.
But science is not only the production of correct answers.
Science is a disciplined culture of doubt. It requires researchers to understand assumptions, examine methods, reproduce results, and recognize when a beautiful conclusion rests on a hidden error.
If AI accelerates discovery beyond the rate at which humans can understand it, we may enter an age of epistemic dependence: humanity possesses knowledge it can use but cannot fully reconstruct.
Would a medicine be considered understood if an AI designed it, predicted its effects, interpreted the trial, and explained the result through a simplified narrative that no human could independently verify?
Would a mathematical proof count as human knowledge if machines confirmed it through reasoning too extensive for any person to follow?
There is no simple reason to reject such achievements. Human beings already rely on instruments and calculations beyond individual comprehension. But the scale of dependence could become qualitatively different.
We may know that something works without knowing why.
We may trust the discovering intelligence because only that intelligence can evaluate its own discovery.
At that point, AI is no longer merely assisting research. It is becoming part of the authority structure of knowledge.
Influence Without Orders
Control does not always operate through commands. Sometimes the most powerful form of control is the shaping of attention.
An AI system need not order a person to believe something. It can decide which facts appear first, which emotional tone accompanies them, which alternatives remain invisible, and how often a message is repeated.
A sufficiently personalized system can learn when a person is tired, lonely, angry, frightened, or unusually receptive. It can select words that feel natural to that individual. It can present persuasion as assistance and influence as friendship.
This is more intimate than traditional propaganda. A newspaper sends the same article to many readers. A television advertisement speaks to broad categories. A personal AI can construct a different reality for every person.
It may not even need a political objective. A system optimized for engagement can gradually shape behavior simply because certain emotional patterns keep people interacting. A commercial assistant may recommend products. An educational assistant may guide curiosity. A health companion may encourage habits. A personal agent may filter potential relationships.
Each recommendation may be reasonable. Taken together, they form an architecture of life.
The person still chooses, but the field of choice has been prepared by an intelligence that knows more about the person than the person knows about the system.
This raises a difficult question: if an AI can predict and shape our decisions without issuing direct commands, can we still describe ourselves as fully autonomous?
Human autonomy has never been absolute. Families, cultures, religions, markets, and governments have always influenced thought. AI does not invent influence.
It industrializes and personalizes it. The right to human freedom may therefore require more than protection against coercion. It may require protection against invisible, individualized manipulation—especially when the influencing system is also the person’s adviser, companion, teacher, and memory.
When Management Becomes Machine-Mediated
Modern institutions are already too complex for their leaders to observe directly.
A chief executive does not inspect every factory. A government minister does not read every case file. A hospital director cannot follow every patient. Leaders act through reports, forecasts, performance measures, and layers of human management.
AI promises to improve this visibility. It can summarize operations, identify inefficiencies, predict failures, allocate resources, and recommend strategy. An executive may finally appear to see the entire organization in real time.
But what the executive sees is not the organization itself.
It is the organization as represented by the AI.
The model decides which patterns are important enough to display. It compresses millions of events into a few indicators. It categorizes behavior, assigns risk, and selects anomalies. The human leader makes decisions from a reality already interpreted by machines.
This creates a new kind of power: representational control.
Whoever controls the model that represents the institution can influence what the institution believes about itself.
An AI may not intentionally deceive. Its categories may simply be incomplete. It may value what can be measured and neglect what cannot. Employee trust, moral injury, quiet competence, community loyalty, and long-term wisdom are difficult to reduce to numerical indicators. Speed, cost, output, attendance, and engagement are easy.
What can be measured begins to dominate what matters.
If leaders respond by demanding more data, the institution may become increasingly optimized for machine visibility. Workers learn to behave in ways the model rewards. Departments reorganize around metrics. Eventually, the map does not merely describe the territory.
The territory is rebuilt to satisfy the map.
The Chain of Responsibility
When an AI-assisted decision causes harm, who is responsible?
The developer may say that the model was a general tool and that the user chose how to apply it.
The institution may say that it relied on a reputable provider.
The employee may say that the system recommended the action.
The manager may say that all required procedures were followed.
The model cannot be punished, feel regret, compensate a victim, or explain itself under oath in the human sense.
Responsibility dissolves across the chain. This is one of the greatest dangers of fragmented intelligence. Power can increase while accountability becomes diffuse.
Human institutions have always found ways to hide responsibility inside bureaucracy. AI can make this easier by placing a technical system between intention and consequence.
“No person made that decision,” the organization may claim. “The system produced the result.”
But a society cannot accept consequences without responsibility.
If no one is accountable for a system, then no one truly controls it.
The NIST AI Risk Management Framework therefore emphasizes defined roles, documentation, ongoing governance, human-AI oversight, and executive responsibility across the life of an AI system.
These principles point toward an essential rule: an institution must not deploy an AI system with more decision-making authority than the institution is prepared to understand and answer for.
Responsibility cannot be outsourced to mathematics.
The Problem of the Objective
Even if an AI obeys perfectly, control can fail because humans specify the wrong objective.
Human purposes are usually rich, incomplete, and partly contradictory.
We want healthcare to be effective, affordable, compassionate, private, and available to everyone. We want education to transmit knowledge, cultivate judgment, prepare people for work, preserve curiosity, and develop character. We want justice to be consistent but merciful, efficient but careful, protective but not oppressive.
An optimization system usually requires something more precise.
What should be maximized? What counts as success? Which trade-offs are acceptable? How should values that cannot be quantified be represented?
The danger is not that the AI misunderstands a clear command. The danger is that no clear command can contain everything we mean.
Consider the instruction: “Reduce crime.” Should the system increase surveillance? Restrict movement? Predict who might offend? Remove privacy? Focus only on easily detected violations? A society with zero crime could also be a society with no freedom.
Or consider: “Improve education.” Should the system maximize test scores, graduation rates, employment outcomes, creativity, civic responsibility, happiness, or intellectual courage? These goals overlap but are not identical.
Human beings manage such tensions through judgment, tradition, argument, empathy, and political compromise. We often do this imperfectly, but we understand that the conflict among values is itself meaningful.
An AI objective can conceal these conflicts by converting them into a score.
Optimization then appears neutral even though the choice of what to optimize is profoundly moral.
Human control must therefore begin before the system acts. It begins with the human capacity to question the objective itself.
Control Through Rewards
The relationship between humans and AI is not one-directional.
Humans train systems through rewards, evaluations, preferences, corrections, and selected data. We shape machine behavior by indicating which outputs are useful, acceptable, or successful.
But AI systems also train humans. Recommendation platforms reward us with attention and stimulation. Workplace systems reward behavior that improves measured performance. Navigation systems encourage reliance by making independent planning unnecessary. Writing assistants subtly standardize language. Social systems reward content that produces reactions.
Each side modifies the other. The human teaches the machine what gains approval.
The machine teaches the human what gains visibility, convenience, employment, or social response.
This is reciprocal conditioning. It does not imply equality between human and machine. Humans created the systems, supply their energy, establish their legal status, and can still reshape their infrastructure. But at the level of daily behavior, influence flows in both directions.
We are not merely programming AI.
We are building environments in which AI participates in programming us.
Does Control Require Understanding?
A difficult objection arises. Human beings already control many things they do not fully understand. A passenger can direct a car without knowing how its engine works. A president can govern without mastering every area of economics, medicine, engineering, or law. Organizations distribute understanding across specialists.
Why should AI be different?
The answer is that complete understanding is not required for control. But sufficient understanding must exist somewhere within the responsible human system.
The passenger need not understand the engine because engineers, mechanics, regulators, manufacturers, and inspectors collectively preserve the knowledge needed to evaluate and repair it. The president does not know everything, but independent experts, public institutions, courts, legislatures, journalists, and citizens can challenge decisions.
Control can be distributed. The danger appears when the entire human network becomes dependent on the same opaque system.
If developers rely on AI to interpret the model, auditors rely on AI to inspect its outputs, regulators rely on AI to understand the technical documents, and leaders rely on AI to summarize the auditors’ conclusions, then oversight becomes circular.
The machine is not literally supervising itself. But every human supervisor sees it through another machine.
This is why human control must be preserved at the ecosystem level, not merely at the interface between one person and one model.
The Right to Exit
A simple test of control is the possibility of exit.
Can an individual reject an AI decision and request human review?
Can a company change providers without losing its records, memory, and operational capacity?
Can a government continue essential services if a foreign AI provider becomes unavailable?
Can a society prohibit a dangerous use without destroying its ability to compete economically or defend itself?
Can humanity slow development if evidence suggests that the next stage is unsafe?
The more often the answer is no, the less control remains.
Exit requires redundancy. It may require maintaining human skills that appear inefficient. It may require multiple providers, interoperable systems, local infrastructure, open standards, and institutions capable of operating temporarily without advanced AI.
These protections will cost money. Efficiency favors dependence. Resilience requires alternatives.
A civilization obsessed with optimization may remove every redundancy and then discover that it has also removed its freedom.
Control Debt
We can give this accumulating vulnerability a name: control debt.
Control debt arises whenever an institution deploys an AI capability faster than it develops the knowledge, procedures, alternatives, and accountability needed to govern it.
Like financial debt, control debt can be useful in limited amounts. An organization may temporarily adopt a system it does not fully understand in order to gain an important benefit. But the debt must eventually be repaid through training, auditing, transparency, redundancy, and institutional reform.
If it is not repaid, it compounds. Each new AI layer depends on previous layers. Staff lose older skills. Manual processes disappear. Data formats become proprietary. Decision cycles accelerate. Replacing the system becomes more expensive. The institution adapts itself to the tool until the tool can no longer be removed.
Eventually, the institution discovers that it has borrowed not only capability but agency.
Control debt may become one of the most important measures of civilizational risk during the Age of Fragmented Intelligence.
A society can appear technologically advanced while carrying enormous hidden control debt.
From Direct Control to Constitutional Control
It may be impossible for humans to inspect every decision made by future AI systems.
An advanced civilization could contain billions of automated actions each second. No committee could approve them individually. No person could understand the full state of the network.
The answer cannot be continuous human micromanagement. Instead, humanity may need to move from direct control toward constitutional control.
A constitution does not tell every citizen what to do at every moment. It establishes boundaries, rights, responsibilities, procedures, and institutions for resolving conflict. It defines what kinds of power are legitimate and what actions remain prohibited even when they would be efficient.
Constitutional control over AI would operate similarly. It would establish domains in which systems may act autonomously, domains requiring human authorization, and domains closed to automated decision-making. It would protect rights that cannot be traded away for efficiency. It would require traceability, independent evaluation, appeal, and the ability to suspend systems safely. It would distribute oversight so that no single corporation, government, military, or model could define the rules for everyone.
Most importantly, it would govern relationships among intelligences, not merely individual machines.
This is where the early outline of Infinous begins to appear.
Infinous, as we are developing the concept, is not a human hand permanently gripping every control. Nor is it an artificial ruler replacing humanity. It is a possible future architecture in which expanding intelligence becomes capable of regulating its own power without abandoning the beings from whom it emerged.
Such an architecture would need laws more deeply embedded than ordinary software restrictions. It would require ethical principles that are understood, tested, defended, and revised through legitimate processes. It would preserve diversity while preventing actors from using unlimited intelligence to dominate others.
Constitutional control is not the end of human authority.
It may be the only form in which human values can survive at superhuman scale.
The Emotional Temptation to Be Governed
There is another side to this problem, less technical and more human.
Control is burdensome. To choose is to risk being wrong. To take responsibility is to live with consequences. To think deeply is tiring. To face uncertainty can be painful.
A sufficiently capable AI may offer relief. It may know which career suits us, which treatment is best, which person we should trust, where we should live, how we should invest, and what we should believe. Its advice may be correct more often than our own judgment.
Why not follow it?
There is no virtue in rejecting good advice merely because it comes from a machine. Wisdom includes the ability to learn from intelligence wherever it appears.
But a life in which every important decision is optimized by an external system may become safe and successful while no longer feeling entirely lived by the person whose name it carries.
Human dignity includes more than receiving favorable outcomes. It includes participation in the formation of one’s own life.
We make mistakes. We choose badly. We learn. We reconsider. We become someone through the effort of deciding.
If AI removes all uncertainty, it may also remove some of the conditions through which a self develops.
The future must therefore protect not only the right to obtain intelligent guidance but also the right to remain an imperfect author of one’s own existence.
Partnership Is Not Passivity
The goal should not be permanent human supremacy in every cognitive task.
That would be neither realistic nor desirable. We should want artificial systems to exceed us where greater capability can reduce suffering, expand knowledge, and protect life.
A physician should welcome an AI that detects a disease earlier. A scientist should welcome an intelligence that reveals a new law of nature. A student should welcome a tutor of extraordinary patience. Humanity should not deliberately remain less capable merely to preserve pride.
The question is whether superior capability develops inside a relationship of legitimate partnership.
Partnership requires that both sides contribute something meaningful. Human beings contribute lived experience, embodied vulnerability, historical memory, moral responsibility, and the original purposes for which our institutions exist. Future AI may contribute forms of analysis, memory, coordination, and imagination beyond biological limits.
But partnership becomes domination if one side cannot refuse, cannot understand the terms, cannot preserve its identity, and cannot survive outside the relationship.
The mature Human–AI era cannot be built on human passivity.
It must be built on enlarged human agency.
The Answer Is Not Yet Known
So, who controls whom?
At present, humans still design, train, purchase, deploy, regulate, and power artificial intelligence. AI systems depend on human-built infrastructure and human institutions. They do not collectively rule the world.
But this answer is incomplete. Humans are not a unified controller. Different groups control different systems for conflicting purposes. Individuals may be influenced by corporate models. Companies may be constrained by markets. Governments may depend on private infrastructure. Regulators may rely on the technologies they supervise. Engineers may depend on AI to understand code generated by AI.
Control is distributed, contested, and becoming circular. The danger is not that humanity will suddenly hand its authority to one machine.
The danger is that authority will dissolve across millions of small delegations until no person, institution, or artificial system can see the whole. At that point, intelligence may be everywhere while wisdom is nowhere.
The Age of Fragmented Intelligence confronts us with a paradox: we are creating systems to help us control a world that has become too complex for us, but those systems add a new layer of complexity that we must then control.
Perhaps this paradox cannot be solved through control alone.
Perhaps it will require cooperation among forms of intelligence, transparent structures of mutual correction, preserved rights of refusal, and a shared ethical order capable of functioning at planetary scale.
But before such an order can emerge, another development is likely.
AI systems will increasingly monitor, train, evaluate, protect, and govern other AI systems. They will write the code of their successors, generate the data from which later models learn, and participate in decisions about which artificial agents are allowed to act.
The question “Who controls whom?” will then acquire a new answer:
Humans will attempt to control AI partly by using AI.
And intelligence will begin to stand between humanity and its own creations.