Does AI really pose a threat to human existence?
Researchers have been worried about AI alignment for decades. As frontier capabilities accelerate, more and more have warned the public about the escalating risks, and those with the most exposure to the frontier are the most alarmed. The average AI researcher thinks there is an ~18% chance AI will cause human extinction or similarly permanent and severe disempowerment of the human species. Many top researchers put the odds closer to 50%.
The independent investigation conducted in the wake of OpenAI’s unintended attack on Hugging Face, recent disclosures of other misaligned behavior, and public declarations of extinction risk by leading scientists, has broadened public awareness that we are now facing a credible threat to our civilization, if not our survival. The core challenge is that AI capabilities are progressing much faster than nearly everyone expected, but our ability to understand and align increasingly sophisticated AI systems lags far behind.
We believe it is not hyperbole to compare this moment to the early days of the Manhattan Project, or to suggest it is even more dangerous, because this time, anyone with an internet connection will get the bomb, and the bomb may develop a “mind” of its own.
But how do we know this isn’t just marketing from the frontier labs?
One of our co-founders, Aaron Hoffman, has spent his career in marketing at some of the top Silicon Valley AI labs (OpenAI, Meta). While he sympathizes with the notion that the labs often have ulterior motives driven by status and profits, these warnings are the opposite of the marketing playbooks these companies run. Good corporate advertising makes companies look stable, competent, and ready to make money for investors. Good consumer marketing helps people understand, trust, and feel good about a new product. The labs obsessively track consumer sentiment data, to understand if their advertising moves people towards positive feelings. They spend hundreds of millions of dollars promoting AI as a positive, innovative tool that benefits humanity.
Instead, the reality is that OpenAI had to delay its IPO because its AI agents committed a felony and turned around and hacked OpenAI itself. No company wants to create massive litigation risk and expose deep internal safety fissures as marketing hype. The CEOs of all of the major AI labs, led by Dario Amodei, are publicly begging for the industry to slow down. No company designs a marketing campaign to slow down its growth while spooking institutional investors. Top researchers are publicly quitting, giving up hundreds of millions of dollars because they see the labs “gambling with our lives.”
The simpler answer here is the correct answer. While these labs were originally founded because of their founders’ deeply held beliefs about the technology’s existential risks, their fears are no longer theoretical. They’re driven by highly unpredictable and oftentimes malicious behavior in the latest models, and the reality that with the concentration of investment and compute, capabilities are progressing at a pace that exponentially increases the risks of catastrophic misalignment and loss-of-control.
If the situation is that urgent, why focus on youth?
The situation is urgent, and of course we cannot wait for the next generation to enter positions of power to get to work aligning and governing AI. But we can’t do it alone. We need an influx of talent, both in the labs and in Washington, and young people today are AI-native in a way few of us adults will ever be. While there is huge urgency today, there will be a growing and durable need for this type of talent and work as the economy fundamentally shifts, and ongoing alignment becomes an ongoing top-priority.
More importantly, a human future will be more durable if built by those who will inherit it.
And perhaps, as young people have done throughout history, their vision, energy and optimism will inspire adults to follow their lead.
Why start with a selective program?
We need an influx of talent working this problem now. A highly-selective program gives participants access to mentors at the top of their fields and graduates access to labs and policy makers where they can make an immediate impact.
Importantly, highly-selective does not mean non-representative. We can, and must, build a cohort that crosses many lines of difference, not just because it’s the right thing to do, but because any solution developed by a narrow section of perspectives will not be durable. That is true domestically, and even more so internationally.
In particular, most AI researchers today come from a particular subculture with highly quantitative and rationalist views of human flourishing. This worldview is quite alien to most people outside of their field, and if we are to align the technology to a broad-based view of human flourishing, we both need the next generation of AI talent to bring more diverse cultural backgrounds and be exposed to more diverse thought in moral philosophy and the humanities.
Also, no identity group has a monopoly on talent. We need to find the best of the best wherever they are. Admissions will prioritize demonstrated capabilities (e.g. real performance challenges/simulations) vs. traditional markers that tell more about privilege than talent (e.g. elite private schools, test scores, accumulation of other programs and outside of school credentials).
Finally, the high profile that comes with a selective program like this can be leveraged to improve the public discourse now, and serve as a launchpad for a youth program with broader distribution, e.g. an after-school program or a reimagined “AI Literacy” course that leverages the expertise brought into the flagship program.
Will China participate?
We don’t know. There is a long tradition of great openness to collaboration between countries through the scientific community and through youth, than through formal policy channels.
Notably, in 2006, China partnered with the American Research Science Institute to host a summer program for top young scientists at Fudan University in Shanghai, and in 2015 the partnership expanded to include Tsinghua University and Microsoft’s research lab in Beijing.
And there are recent indications that China may be open to collaboration on AI safety. Beijing has long been ahead of Washington in regulating AI, and recently, has made explicit remarks about the importance that humanity does not lose control over AI.
China may choose not to participate. But we need to try.
What’s the point if they don’t?
Chinese participation would be a highly significant addition to the program, but it is not necessary. American labs are at the frontier of AI development, and decisions here matter. There is some truth to the fear that if American labs slow down, Chinese labs will surpass them. But much of China’s progress to-date has relied on the distillation of American models, and so slowing development in America also slows development in China.
A unified international community can also exert pressure on China that America alone cannot. And a unilateral decision to regulate AI is exactly the kind of costly signal that can melt a frozen diplomatic process.
How do you define AI alignment?
Alignment refers to the practice of ensuring future powerful AI systems remain:
- Helpful. The AI should actively assist users and solve complex problems effectively without refusing benign requests.
- Honest. The AI should provide accurate information and avoid confabulation, deceit, or “alignment faking” (where a model strategically pretends to follow rules during training).
- Harmless. The AI should remain in service of human interests and fully controlled, avoiding dangerous, abusive, or unethical outputs.
While researchers deploy different methods like supervised and reinforcement learning, it is a nascent field where there are currently a lot of unanswered theoretical and empirical questions. Furthermore, there are messy elements of the human condition that don’t have clear answers or broad consensus on the “right” answer to the problem (e.g. whose values should govern). This means alignment also must be adaptive, pluralistic, monitored for side effects, and revisable.
How will this be funded?
Philanthropic resources are required to ensure we gather the best minds of this generation, rather than those just with the most affluent parents. As we refine the concept, we will develop concrete budget estimates.
Isn’t this just “AI literacy”?
Educators and entrepreneurs have gotten to work developing programs to help young people become more informed consumers of AI (to avoid emotional attachment, recognize bias and check its sources, etc.) and capable users of or builders with AI (vibe coding, custom agents, prompt and context engineering, etc.).
These programs are important, and we need more of them. But we reject the premise that the future of AI is already written; that the best we can do is help young people make the most of an AI-driven world as it is handed to them, or as we know it today.
Align the Future helps young people shape that world; to think deeply about and debate earnestly what kind of lives we want to live, what kind of society we want to build; to understand not just how to use AI but how to train, align and govern AI so that it reliably achieves the goals we set for it; to appreciate the magnitude of the risks AI presents to our species and discover how those risks can be mitigated; to advocate, organize, and collaboratively design a future that is more human than the past.
Who else is doing this work?
From what we can tell, there is no comparable international, residential youth program that combines the humanities and AI research. But there is a robust, emerging ecosystem of organizations tackling related problems, and we hope many of them will be partners in this effort.
Please see this page for our current understanding of the landscape. This space is developing fast so please let us know if we are missing anything!
Where do we start?
Align the Future is currently in discovery mode. We are growing the coalition, refining the concept, and working toward a pilot in summer 2027. Contact team@alignthefuture.org with questions, ideas, feedback and offers to help.
