“Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man… [it] could design even better machines; there would then unquestionably be an ‘intelligence explosion”
AI is changing everything at a dizzying pace — sometimes promising, sometimes ominous, sometimes worrisome. Truth be told, we are at the dawn of a new age more profound than any technological change in our history: fire, the wheel, the combustion engine, the internet. Yet we remain woefully unprepared for this new era.
One of the newer concerns — and opportunities — is “recursive self-improvement” (RSI): an AI system that can improve its own capabilities — its algorithms, architecture, training process, or code — without human involvement, where each improvement makes the system better able to make the next.
the core idea
A system capable of RSI would iterate on itself: Version 1 designs a more capable Version 2, which designs Version 3, and so on. Ordinary AI progress, by contrast, has humans designing each new generation. The concept traces back to I.J. Good’s 1965 notion of an “intelligence explosion” — the idea that once a machine surpasses human-level ability at designing intelligent machines, a rapid, self-reinforcing cycle of improvement could follow.
why should we care?
RSI is central to arguments about transformative AI risk. If self-improvement compounds quickly, capabilities could increase much faster than our ability to test, align, or oversee the system — a “fast takeoff” scenario. A system optimizing itself might also drift from the goals it was originally given, since each iteration changes the very thing meant to preserve them. When AI reaches continuous self-improvement without human guidance and guardrails, we are unleashing forces beyond our control — cybersecurity, weaponry, terrorism, and lost autonomy in every sector.
current state of the technology
Full RSI in the strongest sense — an AI autonomously redesigning its own architecture and training with no humans in the loop — hasn’t been demonstrated. But in June 2026, Anthropic published When AI Builds Itself, arguing AI systems “may be on the cusp” of designing and building their own successors with little human input. Its own numbers back that up: over 80% of merged code in Anthropic’s codebase is now written by Claude, and Claude agents have run roughly 800 hours of open-ended AI safety research experiments autonomously. Anthropic’s stance is notably cautious: “We are not there yet, and recursive self-improvement is not inevitable… it could come sooner than most institutions are prepared for.” It has even floated pausing frontier development to let alignment research and governance catch up.
Researchers disagree on takeoff speed (years vs. days), whether physical or data constraints impose a hard ceiling, and whether guardrails can keep pace with self-modifying systems. Given our track record with social media and mobile phones, I’m not optimistic.
open questions
We should be pressing our politicians, AI company leaders, and local civic departments — while studying the European and Chinese plans. How substantive is the threat? What is the actual timeline? What tactics and strategies can put safeguards in place?
so what to do?
As a tech entrepreneur, I am no fan of significant regulation. Yet we need to:
- Develop safeguards through legislation compelling companies to build in controls and stopgaps. Europe’s AI Act (Regulation (EU) 2024/1689) is the first comprehensive legal framework on AI worldwide — a positive step, though its enforcement teeth remain in question.
- Assess and manage risks via an independent body at the federal and state levels — an FDA for AI — to review and approve the actions of AI companies.
- Pass laws to monitor and establish review requirements and guardrails.
- Require AI companies to stand up independent bodies to test, review, monitor, and run simulations before new features ship.
Voluntary efforts are inadequate. Most leading AI companies have adopted non-binding Frontier Safety Frameworks — Google DeepMind’s (v3, April 2026) defines Critical Capability Levels where risk becomes severe enough to require mitigation before deployment. Non-binding is the operative word.
final thoughts
We have the means and know-how to implement these recommendations and others. The question is: Do we have the will and discipline to follow through? Time is fleeting, and RSI is accelerating…