Who Governs the Machines? AI Leaders Begin Calling for Institutions to Govern Advanced AI
As advanced AI systems become increasingly capable, the question is shifting from what AI can do to who will govern it.
The most important AI question may no longer be what artificial intelligence can do.
It may be who will be responsible for governing what it becomes capable of doing.
That question moved closer to the center of the technology debate this week as Google DeepMind underwent a major leadership restructuring while its former chief executive, Demis Hassabis, continued advocating for an independent institution capable of establishing standards and oversight for advanced AI.
Hassabis has argued for a body that could play a role somewhat analogous to institutions such as FINRA or, at a much larger international level, the International Atomic Energy Agency: an organization capable of establishing standards for increasingly powerful technologies whose consequences extend beyond any single company.
The proposal comes at an important moment.
The Governance Problem Is Arriving Before the Technology Is Finished
AI development is moving extraordinarily quickly.
The leading laboratories are no longer competing merely to produce better chatbots. They are developing systems capable of reasoning, using tools, writing and executing software, conducting research, operating for extended periods, and increasingly participating in scientific and technical work.
At the same time, governments are struggling to determine what should be regulated, who should perform safety evaluations, and how international coordination should work.
This creates a fundamental institutional problem.
The companies developing frontier AI are simultaneously becoming some of the most important actors in deciding how frontier AI should be governed.
That does not necessarily mean that companies cannot participate in governance. Their technical expertise is indispensable.
But it does raise a difficult question:
Can a technology with potentially civilization-scale consequences be governed primarily through voluntary commitments made by the organizations competing to build it?
The Warning Is Coming From Inside the AI Industry
The concern is not limited to governments or outside critics.
More than 1,300 AI researchers and engineers have recently joined a public warning about the risks of an uncontrolled AI arms race, calling for stronger international coordination and safety measures before increasingly powerful systems are deployed.
The significance of this is difficult to ignore.
AI safety is no longer simply an external criticism of the technology industry. It is increasingly an internal question being raised by the people who are building these systems themselves.
The disagreement is therefore becoming less about whether AI requires governance and more about what kind of governance is appropriate.
From Corporate Safety to Public Institutions
There are several possible models.
- AI companies can establish internal safety teams and voluntary standards.
- Governments can introduce licensing, testing, and liability requirements.
- Independent laboratories can conduct evaluations.
- International organizations can coordinate standards between countries.
- Or some combination of all four could eventually emerge.
The difficult part is that AI does not respect national borders, while regulation largely does.
A model developed in one country can be downloaded, replicated, modified, or deployed somewhere else. An autonomous system can potentially operate across jurisdictions. A vulnerability discovered in one AI system may become a problem for thousands of organizations simultaneously.
This makes AI governance fundamentally different from regulating an ordinary consumer product.
The Missing Institution
The emerging debate therefore points toward a question that deserves considerably more attention:
Do we need new institutions for advanced AI, rather than simply applying existing regulatory structures to it?
An effective institution would need to do more than publish recommendations.
It could potentially establish common evaluation standards, coordinate safety testing, maintain incident reporting systems, facilitate international cooperation, and create mechanisms for identifying particularly dangerous capabilities before they become widely deployed.
But such an institution would also have to solve a difficult legitimacy problem.
- Who appoints it?
- Who funds it?
- Who has the authority to inspect AI systems?
- How are disagreements between governments resolved?
- What happens when a company refuses to cooperate?
And, perhaps most importantly, how can such an institution remain accountable to human societies rather than becoming another layer of technological bureaucracy?
These are not purely technical questions.
They are questions of public administration, institutional design, and political legitimacy.
Human Continuity Requires Institutional Continuity
This is where the AI debate becomes relevant to a broader question of human continuity.
Humanity has repeatedly created institutions in response to technologies that changed the scale of human activity: nuclear energy, aviation, telecommunications, biotechnology, and global financial systems all eventually required governance mechanisms extending beyond individual companies.
Advanced AI may require something similar.
The challenge is timing.
Institutions normally evolve slowly.
Technology can evolve extremely quickly.
If advanced AI capability continues accelerating, the gap between technological capability and institutional capability could become one of the defining governance problems of the coming decade.
The objective should therefore not be to stop technological progress.
It should be to ensure that human institutions remain capable of understanding, supervising, and directing the systems they create.
The central question is no longer simply:
How intelligent can machines become?
It is: Can human institutions become sufficiently capable, adaptive, and accountable to remain responsible for increasingly intelligent machines?
That may ultimately be the more important AI race.
Published by the Human Continuity Institute — August 15, 2026