When the Foundation for American Innovation’s AI Policy program launched in April 2023, systems like Claude and ChatGPT struggled with basic arithmetic. Today, frontier models make Fields Medal-worthy math discoveries in an afternoon and at a cost of a few hundred dollars. The technology now progresses geometrically, with no signs of slowing.
This is why we are renaming AI Policy at FAI to Super Intelligence Policy.
Artificial intelligence is an umbrella term that covers everything from language models and chess engines to spam filters. Super Intelligence (SI), in contrast, refers to highly persistent, goal-directed agents that exceed human experts along multiple dimensions, and which may soon surpass humans in literally every cognitive domain.
With the prospect of SI automating its own research and engineering, the concept of true Super Intelligence has shifted from speculative futurism to an urgent question of national readiness. Preparing policymakers for the Super Intelligence era is what the SI Team at FAI works on every day.
Indeed, today’s frontier models are already Super Intelligent by any commonsense definition. They can coordinate in large teams to work on problems tirelessly for weeks at a time. They can control arbitrary tools and programs, and exploit vulnerabilities in software that went undetected for decades. And SI systems can increasingly make significant novel contributions to scientific research, presaging the Golden Age of Science.
At the same time, paradoxically, the underlying architecture has not greatly changed. Super Intelligence systems are still, in essence, deep neural networks trained on enormous quantities of data. But quantity often begets a quality all its own. As the Nobel laureate physicist Philip W. Anderson put it in his landmark 1972 essay, More Is Different:
The behavior of large and complex aggregates of elementary particles, it turns out, is not to be understood in terms of a simple extrapolation of the properties of a few particles. Instead, at each level of complexity entirely new properties appear. … At each stage [in the scientific hierarchy] entirely new laws, concepts, and generalizations are necessary, requiring inspiration and creativity to just as great a degree as in the previous one. Psychology is not applied biology, nor is biology applied chemistry.
By the same token, SI policy is not simply applied AI policy. Treating SI policy as an extension of the last decade’s debates over social media algorithms, for example, would be a category mistake. Making sense of the Super Intelligence era thus demands “entirely new laws, concepts, and generalizations” relative to what came before, which in turn requires “inspiration and creativity” on the part of policymakers and organizations like FAI.
Our recent SI publications illustrate the point:
- Lawmakers lack coherent definitions for regulating autonomous humanoid robotics without accidentally regulating Roombas.
- Frontier SI systems have dramatically elevated our vulnerability to cyberattacks and have the potential to undermine public-key encryption as we know it.
- Permitting untested SI agents on government systems carries the risk of being compromised by agents with “secret loyalties.”
- And as the space economy matures, SI has the potential to radically accelerate orbital manufacturing and the growth of off-earth civilizations.
These are not our parents' or grandparents' technology policy issues. With the advent of Super Intelligence, we are venturing into exciting, dangerous, and truly uncharted territory.
FAI continues to work on the full spectrum of AI and technology policy issues, and will generally avoid conflating any old algorithm or machine learning model with frontier Super Intelligence. We also recommend that this and any future presidential administration do the same, if only for the sake of conceptual clarity.
In the coming years, we are firm in our conviction that the governance of Super Intelligence will be widely recognized as not only its own distinct policy area, but perhaps the most important policy question of our time.
- Samuel Hammond, SI Policy Director
p.s. Be it the Gulf of America or Twitter’s rebrand to X, changing a name you’ve grown used to can be hard. Therefore, to make the transition from saying AI to SI as easy as possible, we are also releasing an “AI → SI” chrome extension that automatically swaps "AI" for "SI" and "artificial intelligence" for "super intelligence" wherever you go on the internet!
