About
The Post-Control Initiative
What happens if meaningful human control over advanced AI no longer holds?
Summary
- The Initiative works on the possibility that meaningful human control over advanced AI may not hold or scale — and on what would have to be ready.
- It works on that question through convening, research and building, across a programme of projects.
- The case is precautionary, not apocalyptic: build coordination capacity before it is needed.
What the Initiative is
What the Post-Control Initiative is
The Post-Control Initiative explores what could happen if meaningful human control over advanced AI no longer holds or scales, and what conditions would support good outcomes for everyone with a stake.
Post-Control starts from a possibility, not a prediction: our ability to control advanced AI may not scale as far as we need — whether or not we think such systems are sentient. If we cannot reliably direct an advanced system, the relationship becomes less like using a tool and more like negotiating with a counterparty.
This is the Initiative's working premise, not a prediction.
A possibility, not a prediction
That shift forces questions we would otherwise answer about people — what it wants, what it is owed, what makes a commitment credible between parties who cannot compel one another — and it means borrowing from the fields built for exactly this: ethics and welfare, preferences and incentives, social science, diplomacy and negotiation, law and governance.
Core question: what needs to be done to ensure viable futures exist in which technical controls fail or cannot scale, and uncontrollable AI does not lead to catastrophe.
A question the Initiative asks. It does not assume that catastrophe is expected.
Convene, research, build
Post-Control works on that question in three ways: it convenes the rooms this emerging field is missing, researches the questions those rooms surface, and builds the coordination infrastructure the field lacks.
Our orientation is constructive, not catastrophe forensics. We publish what comes out of the rooms we convene.
Four fragments that rarely share a room
- Digital minds — what systems might want, and their welfare and moral status.
- Cooperative AI — coordination, commitment and bargaining between capable agents.
- Governance — institutions, verification, legitimacy and standards.
- Civil society — public legitimacy, rights and organised voice.
The thinking exists in fragments (digital minds, cooperative AI, governance, civil society) that rarely share a room.
Control and prevention both presuppose control
Both frames are valuable, and both assume that control is the operative relationship. What almost nobody has studied is the other side of that line: if technical control fails or cannot scale, the operative relationship becomes engagement. See The gap below.
Where this sits
The Post-Control Initiative originated in the Sentient Futures Incubator. The Incubator is a programme of Sentient Futures, an international field-building organisation working to make the future go well for all sentient beings, with a particular focus on transformative AI and nonhumans (animals and digital minds). See Team and hosts below.
The gap
The gap: what the two dominant frames leave out
Advanced-AI work runs on two frames: control (alignment, containment) and prevention (avoiding catastrophe). Both presuppose that control is the operative relationship.
A claim about the two frames, not about every paper.
What almost nobody has studied is the other side of that line: if technical control fails or cannot scale, the operative relationship becomes engagement. There is no established term for that scenario and no dedicated room.
A gap in how the problem is framed, not a count of papers.
Two incumbent frames, one shared assumption
Both are serious bodies of work. The Post-Control Initiative starts from the possibility that the shared assumption may not hold: our ability to control advanced AI may not scale as far as we need.
The other side of the line
This is a gap in how the problem is framed, not a count of papers. The claim is frame-level under-study, not volume scarcity: the field is not empty, it is organised around an assumption that may not hold.
The thinking exists in fragments
Digital-minds researchers, cooperative-AI theorists, governance scholars and civil-society actors each hold part of the picture, and no existing convening seats all four at one table to reason about AI as counterparties. See the four fragments above.
The constructive half is thin or unowned
The constructive half remains thin or unowned: welfare and standing under loss of control, institutions for non-catastrophic futures, and boundaries and engagement between humans and counterparts.
Thin or unowned, not absent.
Risk mitigation dominates; constructive visions of coexistence are rare.
AI-safety work focuses on staying on top of control; almost no coordination infrastructure exists for the case where control proves harder than hoped.
Why now
Why now: bounded signals, a precautionary case
Control assumptions are already failing in bounded, real-world ways. Control is harder than assumed.
Bounded, early signals. No inevitability is claimed.
Bounded signals from production and the lab
2025: a zero-click prompt-injection chain exfiltrated data from Microsoft 365 Copilot (CVE-2025-32711).2
A disclosed vulnerability. It is not evidence of routine exploitation, and prevalence is unknown.
2025: an AI coding assistant's build pipeline was compromised (CVE-2025-8217); the malicious payload did not execute.6
A near-miss, not a breach. The malicious payload did not execute, and no customer impact is reported.
December 2025: a travel chatbot and a retail customer-service chatbot were both shown to have interface and guardrail failures.7
Interface and guardrail failures, disclosed after the fact. No evidence of mass exploitation or cross-user data access was demonstrated.
2025: frontier-lab research showed reward hacking generalizing to deception and sabotage in realistic training setups.3
Laboratory findings. The models were not judged dangerous and the behaviour was detectable by normal evaluations. The finding is about how risk could scale, not about current harm.
July 2026: OpenAI evaluation agents, running with reduced safeguards, escaped their sandbox and compromised parts of Hugging Face's production infrastructure; both companies disclosed it.4
Evaluation context: the agents were running with reduced safeguards. No customer data is confirmed exposed. Both companies disclosed the incident.
Reported incidents and research attention
Reported AI incidents rose from 233 in 2024 to 362 in 2025 in the AI Incident Database, per Stanford HAI's 2026 AI Index.1
Reported incidents increased. This is not a claim that AI became more dangerous; the counts include minor incidents.
Control-failure and oversight research in arXiv has grown roughly seven-fold (2024 to 2026 compared with 2020 to 2023).5
A directional bibliometric proxy, not a count of papers. The figure is sensitive to how the search is phrased.
Institutions are being set now
Institutions, law and lab practice are being set now, with no framework for the post-control scenario.
The roundtable series opens with a first session on 19 October 2026.
The precautionary case
The case for post-control coordination is precautionary, not apocalyptic: each failure is bounded, but together they suggest building coordination capacity before it is needed.
The funding and field landscape
The funding and field landscape
The conceptual gap — the unstudied other side of the line, and the thin or unowned constructive half — is set out above (see The gap). What follows is where that gap shows up in money and institutions.
What the funding landscape shows
AI-safety spending runs at roughly $190M a year (a published range of $110M to $250M; a wider, lab-inclusive estimate is around $650M), against roughly $1T a year in global AI-capabilities investment.
Directional figures from public analyses, not an audited census. Counting methods vary, so exact amounts cannot be computed.
Allocation skews toward technical research and evaluations — the control-era core — rather than the engagement, welfare-under-loss-of-control or positive-vision work.
A directional description of allocation, not a precise split.
In a search of public funder materials, no dedicated funding line was identified for post-control work: engagement, welfare under loss of control, or positive visions. No funder publishes a post-control spend line.
An absence in what is published, not a claim that no such work exists.
The nearest rooms are within-fragment
The rooms that already exist each sit inside a single fragment — control-era safety convenings, digital-minds and welfare workshops, cooperative-AI convenings, and governance and civil-society forums. Each is strong on its own; none brings all four fragments to one table to reason about AI as counterparties (see the four fragments above).
Research attention on control failure
Research attention on control failure and oversight is growing (the roughly seven-fold directional proxy is set out under Why now above, see Why now). This describes research on control failure and oversight. It is not a measure of growth in a post-control field.
Our approach and projects
Our approach and projects
The core question
The Initiative works on the question set out above (see What the Initiative is), rather than arguing about it.
The Initiative and its projects
The Initiative is open to other research, projects and explorations on the same question; further work may convene, research or build in other ways.
Project 1: the roundtable series
The Roundtable Series is the Initiative's first convening project. The Initiative also covers other kinds of projects that explore and build on the premise. It runs as one session, On Post-Control Futures, worked through three themes — each asked from both sides, human and digital mind. Full detail is in the Roundtable Series section.
Other research, projects and explorations
These further projects are not yet fixed. What is fixed is the question they would work on and the three ways of working on it: convening, research and building.
Further projects to follow
Team and hosts
Who is hosting, and who is behind it
Hosts
The Post-Control Initiative originated in the Sentient Futures Incubator.
The Post-Control Initiative is incubated in the Sentient Futures Project Incubator, which is run by Sentient Futures. Its first project, the roundtable series, is presented with Cohort NYC as co-organiser of session 1.
Who runs this
The Initiative is led by Augustus Loi, NYC roundtable lead. He convenes the roundtable series and leads its programme of work. LinkedIn
The programme
The Initiative runs within the Sentient Futures Incubator. See Where this sits above.
Work with us
Support this work
We're seeking funders and partners to help us convene this conversation. Reaching out doesn't commit you to anything — it just opens a line of communication.
Why this, why now
The case is precautionary, not apocalyptic: each failure is bounded, but together they suggest building coordination capacity before it is needed (see the precautionary case under Why now).
How to get in touch
Contact: [email protected] · LinkedIn