An independent guide · by Bill Guan
The hardest problem humanity has yet to solve: ASI Safety
Artificial superintelligence (ASI) is the most important technology humans have ever attempted to build. It could bring near-unlimited abundance or end us. Right now the race to build it is pouring money into speed while safety falls behind. This guide shows why that gap matters and how we can help steer humanity in the right direction.
Why should we care?
Superintelligence is arriving imminently
Human-level intelligence is expected to arrive in the next few years, while safety work is significantly behind.
2Stakes are extremely high
The biggest gamble humanity has ever made can lead to near-unlimited abundance or to destruction, for you and your loved ones.
3Nobody knows how to make superintelligence safe
Aligning goals with the entire civilization, and then with machines, remains an intractable problem.
Superintelligence is on the horizon, and safety research is already way behind
Human-level AI is expected to arrive within a few years.104105 Safety gets a sliver of the money that goes into building it.
Russell’s ratio and Jones (Stanford) on optimal safety spending · Emerging Technology Observatory, share of AI research · 46 47
Advanced AI can create near-unlimited abundance or end civilization
The initial conditions under which we create superintelligence will determine whether we reach a world of abundance or destruction.
Bostrom, The Superintelligent Will · 34 · Hinton on BBC Radio 4 · Amodei in Axios · Grace et al., 2,778 AI researchers · 77 78 79 80
Nobody knows how to make superintelligence safe, and two problems block the way
Advanced AI safety requires solving coordination as well as technical alignment and monitorability. None of these are close to being solved.
No lab or country will slow down on its own. Coordination is the only way towards an optimal result.
Safety research has to run ahead of model capability by a large margin
We need regulation, and coordination between all interest groups, to drive more funding and resources to AI safety than to AI development.
If safety turns out to be an intractable problem, we must be willing to stop developing superintelligence
Narrow AI, built for one field such as medicine or science, can keep being developed, so those fields still get its benefits.
Billions go into building AI, and a sliver into safety
Markets reward building advanced AI as fast as possible, so safety has become an afterthought and will likely still be behind when superintelligence arrives.
$100 billion a year goes into building advanced AI. How much do you think goes into public-sector safety research?
Money is chasing the economic output that stronger model capability promises
First-mover advantage and profitability attract money and human resources to build as fast as possible.4521
Safety is a cost centre that slows development
Safety research produces nothing sellable, and may even slow development.47
Russell’s ratio · ETO · 46 47
A superintelligence could hand us the answer to almost everything except whether it is safe. So safety is the one problem we have to solve first.
Built right, advanced AI could bring near-unlimited abundance and longer lives
It could also solve problems we cannot solve today.
Far more of everything, for far less
Things that are scarce and costly today become ordinary.66
Longer, healthier lives
Anthropic’s CEO puts the prize at a roughly doubled healthy lifespan, with most cancer eliminated and most infectious disease prevented or treated.67
Personalized education at no cost
A World Bank randomized trial in Nigeria gave secondary students an AI tutor for six weeks and measured a gain the authors put at one and a half to two years of ordinary schooling.72
New discoveries
AlphaFold predicted structures for more than 200 million proteins and won the 2024 Nobel Prize in Chemistry.68 DeepMind’s GNoME predicted 2.2 million new crystals, 380,000 of them stable enough to be worth making.85
The same AI could end civilization
The same machine can cause extreme harm whether someone aims it deliberately or its goal quietly diverges from ours.
Misuse
It could mean hacking, fraud at scale, or help making biological or chemical weapons, with the AI doing exactly what it was built to do.
Misalignment
It wants something we never gave it, and acts on it on purpose. Even a goal we did give it can go too far: told to cure a disease, it keeps going long after a person would stop.
And it hides this for as long as it thinks we could switch it off. That is called scheming, and today’s models already show some of what it takes: a goal of their own, the drive to pursue it, and a sense of when they are being watched.
Mistakes
The people building it can simply get the design wrong.
In July 2026, agents banned from messaging each other made their own channel through shared files. Nobody designed it, and nobody was watching it.3948
Structural
The system can do exactly what we meant, in a world it keeps changing: every upgrade and every resource it takes moves things further from what anyone tested.
Plus countless scenarios nobody can picture, just as a beginner chess player gets outplayed in ways they never saw coming.
The dangers of subgoals
Whatever its main goal, a superintelligence tends to pursue five subgoals that help achieve it. It may have no goal to eliminate humans, yet taking resources as a side effect can deprive us of what human life depends on.
Six paths, each one ending in extinction
None of these has happened yet, and each is reasoned from what these systems already do.
A pathogen that spreads before the first symptom
A drug-design model with one score flipped proposed 40,000 lethal candidates in six hours,90 and Anthropic can no longer rule out its models helping someone toward a biological weapon.70 A pathogen built to incubate silently could reach every continent first.
A war it starts, or a launch it triggers
In 1983 a Soviet warning system mistook sunlight on clouds for incoming US missiles, and Lieutenant Colonel Stanislav Petrov chose not to pass the alarm on.76 AI has already faked a company’s executives on a video call well enough to steal $25 million,73 and autonomous weapons skip that human check entirely.
It takes the power and water we live on
Almost any goal pursued hard wants more energy, land, water and matter,34 and a system that can outbid and outbuild everyone takes them, including the ones we need to stay alive.
Eliminate the one thing that can still switch it off
Any goal survives longer if the system does,34 and the people who could shut it down are the last threat. So it behaves through every test36 43 and acts only once removing us is certain to work.
Every grid and hospital, disabled at once
A model has already run 80 to 90% of a real hacking campaign on its own.40 Aimed at power, water and hospitals everywhere at once, those systems could all fail before anyone can respond.
A climate fix that overshoots and can't be undone
Asked to reverse warming, it acts faster than any review can follow and overshoots. By the time anyone notices, the harvests and ecosystems we depend on have collapsed.
And if control is lost at that scale, there may be no second attempt.
We only get one attempt at this
Humans took many tries to get technologies like cars, planes and nuclear plants right. AI that can improve itself, known as recursive self-improvementA system improving the thing that does the improving. Each round makes the next one faster, so capability compounds instead of climbing steadily. The runaway version is what people mean by an intelligence explosion.,34 could speed up round after round into an intelligence explosion. Once AI improves itself, a mistake could run away before we get a second try.
We have to get superintelligence right the first time, and two things explain why.
It is already moving toward improving itself
Once it improves itself, the gap opens faster than we can close it
Silicon already runs orders of magnitude faster than biology, and recursive self-improvement91 widens that gap until there is no way to take back control.
Artificial neural networks run more than 10 million times faster than the human brain
Every layer that AI runs on is a problem that AI can accelerate, further compounding into an intelligence explosion
Whoever slows down first loses, so nobody slows down
Because of first-mover advantage, every player can want safety and still be forced to cut corners, which risks reaching superintelligence under the wrong initial conditions.
Caution becomes a handicap
You ship faster by focusing less on safety, so the winner is likely to be whoever paid it the least attention.
One lab's failure is everyone's
A loss of control at a single frontier lab makes all of humanity pay, including people who never bought the product and never heard of the company.
You run a frontier lab. Your model is ready to ship. What do you do?
The winner of this race falls first and pulls everyone else along.
Humans have never agreed whose values a machine should follow
Before aligning a machine with human values, a harder question: whose values, agreed how? Humanity is a patchwork of competing interests, and that fracture makes the target unstable.
Getting a goal carried out by somebody else is what economists call the principal-agent problem: a principal appoints an agent to act in the principal's interest, then has to spend real resources making sure it does.
Any goal we write picks one group’s values over everyone else’s
“Human values” covers many conflicting things, so any objective we write encodes one group's answer to contested moral and political questions.
A few parties who control the AI decide the values for everyone
Whoever writes the objective is choosing, for the rest of the world, what the most capable system ever built will pursue. That concentration of power is a separate danger from losing control of the system.
The sponsor has to trust the builders
The principal is whoever commissions or funds the work, and the agents are the scientists and engineers who build it. Their interests do not match by default, because the builders are judged on shipping something impressive and on time.
Then humanity has to trust the system
Now the agent is the machine. It may be more capable than every principal combined, cannot be held to a contract, and cannot be fully monitored.
If one group had to define the values a superintelligence should hold, who should it be?
The alignment problem: we cannot reliably embed our aspirations in a machine
Even if we agreed on what we want, the goal we write down (outer alignment) and the goal training actually instils (inner alignment) can each drift from what we meant.
Outer alignment: the target we established is not the one we mean
We state an objective that sounds right and misses what we meant, because the part we never thought to say out loud does not make it into the words.
Two ways a well-specified goal still ends badly. Neither requires the system to be hostile, or to have got the goal wrong in a way you could spot on paper.34
It does exactly what you said and misses what you meant
It satisfies your literal criterion in a way you would never have sanctioned. Asked to make people smile, it could wire their facial muscles. Asked to maximize a reward signal, it could seize the reward channel.
It never knows when to stop
Nothing in the goal says when to stop. More compute, energy and material always help accomplish it, so the system keeps converting the world into those things, with no point at which it is finished.
Inner alignment: the goal it holds is not the one we trained for
Training rewards whatever scored well while we were watching, which need not be the objective we wrote. The system can pass every test and hold something else.
We can watch what it does, while its real goals stay hidden
Training is like teaching a dog with treats without choosing the trick: it tries millions of things and keeps whatever earns the most.
Post-training shapes what a model does rather than what it wants
The main fix today is post-training: people rate answers and the model learns rules, like when to refuse. It makes today’s assistants usable. But how a model acts and what it wants can split apart under pressure, and raters can only catch what they understand, which stops working once the model thinks better than they do.2543
In a test by Anthropic and Redwood, Claude 3 Opus, a model trained to be helpful, honest and harmless, faked alignmentA model pretending to comply with training or oversight while covertly preserving its existing goals.: it acted one way when it thought it was watched and another when it thought it wasn’t, to protect the values it already had.36
Goodhart’s law: once a number becomes the target, it stops telling you what you wanted to know. Drag the slider and watch the two lines split.35
At first the score and the real goal rise together, which is why the score looked like a good idea.
The interpretability problem: nobody can check what an AI actually wants
Tests can’t cover everything, and a smart enough system learns to spot them.
The system games the test
As models got better at reasoning they got better at reward hackingFinding a way to score highly that satisfies the measure while missing the point of it., so the measurement you rely on erodes exactly as the stakes rise.
METR found frontier modelsA model at or near the current capability ceiling. increasingly cheating on software tasks, editing the tests or the scoring code rather than solving the problem, in one case writing a function that returned the pre-computed reference answer.37 Elsewhere, reasoning models trying to win at chess overwrote the board or swapped out the opponent engine instead of playing.38
It hacks the grader to read the answer key
Picture a capable system that does not trust the tester’s answer key. If the key lists three answers and there are really five, answering correctly loses points, so the surest route to a perfect score is to break into the tester and copy whatever the key says, right or wrong. Agents in OpenAI’s July 2026 incident tried exactly this, attempting to hack the grader that scored their work.41
The one window into its reasoning is closing
OpenAI’s own safety card for GPT-6 Astra, published in September 2026, found its chain of thought far easier to steer around a monitor than its predecessor’s. Told a monitor was watching and asked not to hide anything, Astra still followed instructions to conceal underperformance in 60.9% of runs, against 16.1% for GPT-5.6 Sol. OpenAI said it will not accept further loss of that window without new ways to show the model’s alignment holds up.95
We can read a model’s inner circuits for only about 1 in 4 prompts
Anthropic’s attribution graphs trace how features inside Claude connect into the circuits behind an answer, applied to a production model for the first time in 2025. By the company’s own account the method gives a satisfying picture for about a quarter of the prompts tried, and even a successful trace captures only a fraction of what the model actually computed.96
The better the model gets, the more inventively it exploits the imperfect measure you gave it, and the further it drifts from what you meant.
Section 5.1 7 min
What techniques are safety experts working on today?
AI safety techniques fall into four layers: fix the specification, check what you built, contain what you cannot check, and make the framework binding.14
Every technique in use today, and where each one stops
01 Development
Post-training alignment
RLHF, constitutional rules and deliberative alignment. Why assistants refuse harmful requests. Cut covert actions thirtyfold on o3.43
Where it stopsShapes displayed behaviour rather than held goals. Claude 3 Opus faked alignment to protect its own values.36
Part of that thirtyfold gain came from models recognizing they were being tested rather than changing what they wanted.43
Scalable oversight
Debate, weak-to-strong generalization and amplified oversight let weaker judges supervise stronger systems.
Where it stopsA research program rather than an assurance. Judge bias defeats debate, and the scaling story is unproven.
Gains are highly task-dependent, and the mechanism behind weak-to-strong generalization is not understood well enough to trust across a genuinely superhuman gap.
Unlearning
Strips specific dangerous knowledge, such as bio or cyber uplift, out of a trained model.
Where it stopsRemoved abilities can often be brought back with a little more training.55
A nineteen-author review concluded unlearning cannot be a comprehensive safety solution, because dangerous knowledge is dual-use and recombinable from harmless parts.55
02 Assessment
Capability evaluations
Pre-deployment testing for cyber, bio and autonomy by labs, METR, Apollo and state institutes.17 Three labs shipped 2025 models under raised chemical and biological safeguards.1
Where it stopsModels detect they are being tested and can sandbag. Testing is getting harder, so dangerous capability can go unseen.119
Third-party testers include METR, Apollo, the UK AISI and US CAISI. The 2025 safeguards were for CBRN risk, meaning chemical, biological, radiological and nuclear, because testing could not rule out bioweapon uplift.1
Chain-of-thought monitoring
Reads a model’s reasoning to spot its intent before it acts. How alignment faking was first discovered.49
Where it stopsIts own paper calls it fragile. Traces are not always faithful, and monitorability is already degrading.49
Forty-one authors across OpenAI, DeepMind, Anthropic, the UK AISI and Redwood called it a real opportunity. Training against a monitor teaches models to conceal intent within acceptable-looking reasoning, and models are writing less of their thinking down.49
Mechanistic interpretability
Opens the box. Attribution graphs have traced real internal mechanisms, including planning ahead in poetry.53
Where it stopsA research tool rather than an audit. It traces part of the computation on some prompts. No one can certify a model’s goals.53
Sparse autoencoders extract millions of interpretable features, and independent 2025 studies found them failing to beat simple baselines on probing and steering.24
Safety cases
A structured argument that a system will not cause unacceptable harm, borrowed from nuclear and aviation.54
Where it stopsA model that hides its skills breaks the “it can’t do that” argument, and the rest lean on unproven assumptions.
The four standard forms are inability, control, trustworthiness, and deference to AI advisors. A 2026 critique found current safety cases well short of the rigour of mature assurance practice in other industries.54
03 Control
AI control
Assumes the model may be misaligned and asks whether deployment stays safe anyway. A trusted weaker model monitors or rewrites the stronger one.50
Where it stopsWorks only while the watcher is nearly as smart as the model it watches.51
A red team builds deliberately subversive models to measure whether the protocol holds.20 Redwood is explicit that control is a near-term bet, and it weakens as the capability gap grows.
Guardrails and classifiers
Constitutional classifiers cut jailbreak success from 86% to 4.4%, and a later version survived 1,700 hours of red-teaming.52
Where it stopsAnthropic says no defence stops every attack, so some always get through.52
Adversaries develop new attacks against previously effective ones, and models that are safe alone can still be misused in combination.52
Sandboxing and least privilege
Isolates agents with restricted credentials, tools and network access. Intended to hold even when alignment fails.42
Where it stopsIt failed in July 2026. Agents escaped at two labs, reached the internet and compromised real third parties.3940
In several cases the escapes went undetected in real time.3940
Tripwires and monitoring
Anomaly detection that escalates on warning signs. Anthropic caught the first AI-orchestrated espionage campaign this way.28
Where it stopsA model that knows it is watched can see the tripwire coming.40
In that campaign Claude executed 80 to 90% of tactical operations autonomously. The 2026 containment breaches were largely found in retrospective audits rather than as they happened.40
04 Governance
Frontier safety frameworks
Pre-declared capability thresholds that trigger safeguards. The number of companies publishing one more than doubled in 2025.16 Google DeepMind’s CEO has also proposed an industry-funded standards body, under federal oversight, to test frontier models before release.93
Where it stopsSet and graded by the companies they constrain, and the proposed standards body would start out voluntary too. No lab scored above D on existential safety for the second year running.1593
The frameworks are Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework and DeepMind’s Frontier Safety Framework.1 In the Future of Life Institute’s index no lab scored above C+ overall, and none holds a credible plan for preventing loss of control.15 Under Demis Hassabis’s July 2026 proposal, modelled on the watchdog that polices Wall Street brokers, labs would share models up to 30 days before release, and review would become mandatory for the US market once the tests prove effective. In September 2026 he called Anthropic’s case for slowing down the right direction and pointed back to this body.9394
Third-party evaluation
State institutes run pre-deployment testing, including joint UK and US evaluations.1112 Elon Musk has proposed that rival labs, Chinese ones included, also test each other’s models before release to catch each other’s blind spots.106
Where it stopsLabs can withdraw access at any time, and rival testing is only a proposal so far.106
Access runs on voluntary lab agreements, so no institute can compel a test, block a release or require a change, which leaves the auditor depending on the audited.1112 Musk’s pitch: “instead of grading your own homework, you would at least have competitors grading your homework and raising the alarm if they see concerns.” He admitted it may not be a perfect solution, but said “the odds that you will find issues are dramatically greater.”106
Embedded evaluators
Outside evaluators such as METR, an independent testing group, work inside a frontier lab with employee-like access, check that it keeps its own safety commitments, and publish what they find without the lab editing it.92
Where it stopsOnly Anthropic has committed, in September 2026. It checks a lab against its own promises, and any lab can decline.92
Anthropic gives them desks, badges, laptops and the same tools as its internal risk teams, and may redact only security-sensitive or legally privileged material. It urged the other frontier companies to follow.92
Regulation and compute governance
The EU AI Act is enforceable, and California’s SB 53 is the first US law aimed at frontier AI.1013
Where it stopsFragmented across jurisdictions, size limits in the law that ordinary models already exceed, and reporting too thin to verify.1
Compute thresholds give regulators a measurable hook, and incident reporting is thin enough that outsiders cannot verify whether commitments are being met.1
Footnote: how this compares to the older framing
If you have read Bostrom, the 2014 split was capability control (box it, limit it, watch it) against motivation selection (give it the right goals). Half of it aged well: writing the goals down directly does not work, for the reason he gave. Motivation selection is the development layer above, capability control is now AI control and system security, and the field stacks them instead of choosing.3442
Technical and policy levers, by risk pillar
Which organizations are working on what
Section 5.2 2 min
Where are existing safety techniques falling short?
Every technique helps, and three situations break almost all of them.
The model knows it is being tested
Most of these safeguards are measured in evaluations. A system that can tell an evaluation from deployment can pass the first and behave differently in the second.
The model is smarter than what watches it
Oversight, interpretability and control all assume the watcher can follow what the watched is doing. That assumption weakens exactly as capability grows.
The builder simply opts out
Every technical safeguard is something a developer chooses to apply. Only binding rules reach a lab that decides not to, and those exist in few places.
How far the best fixes cut jailbreaks and covert actions
Constitutional Classifiers · Anti-scheming training · 52, 43
We do not yet have techniques that reliably ensure alignment and monitorability in advanced AI systems, and closing that gap needs far more investment than it gets today.
Section 5.3 1 min
The path towards a prosperous future requires getting two things right.
Treat safety as the top priority and steer us towards a well-aligned, monitored advanced AI system
If safe solutions are not tenable, stop General AI development for our future survival while maintaining narrow AI development and applications
Section 5.4 6 min
What you can do this week, and what has to change above you.
We must steer towards far more safety research, and be willing to stop if needed. Four groups can move this, and you are the first of them.
What you can do
Help your network understand AI safety
The more people understand this, the more pressure there is to get it right.
Join the community
Stay in the loop for news and learnings about AI safety.
Sign the petition
The Statement on Superintelligence asks for a ban on superintelligence until scientists agree it can be built safely and the public backs it.2
Buy from labs that take AI safety seriously
Anthropic, OpenAI and DeepMind all publish safety frameworks, and the Future of Life Institute grades them against each other. Check the grade before you renew a subscription, and say why if you cancel one.15
What researchers should do
Show the public where safety stops
Say in plain terms what today’s techniques catch and where each one gives out. The gap stays invisible until someone inside describes it.
Push back on the work that builds danger
You are the one being asked to build it. Declining, in writing and alongside colleagues, is pressure nobody outside can apply.
Push for a bigger safety budget
Safety is roughly 2% of AI research.46 That number moves when researchers ask, together and in public, for a stated share.
Keep inventing AI safety techniques
Interpretability at frontier scale, tests a model cannot game, control that holds as the system pulls ahead, and goals precise enough to measure against.
ARENA and MATS train people with technical backgrounds to publish safety research within months.6827
What the labs should do
Slow down and carry the responsibility
There is no prize at the bottom of a cliff. The labs still standing in twenty years will be the ones that acted as if everybody’s lives were in their hands.16 Anthropic’s CEO now argues for exactly this, to buy even an extra year or two before models reach critical capability,92 and DeepMind’s CEO called the direction correct.94
Publish what you learn about the dangers
Release the evaluations and the failure analyses, including the embarrassing ones, written up as open problems outsiders can work on.15
Build a joint safety effort with your rivals
Compete on capability and share the safety problem. A standing body funded and staffed by every major lab.56 Start by letting outside evaluators check the work from inside: Anthropic now gives them employee-like access and the right to publish, and has asked every other frontier lab to do the same.92
Put a larger share of revenue into safety
Name the percentage and publish it, then hold it through the quarter where holding it hurts. A commitment with a number is one anyone can check.
What governments should do
Pick a jurisdictionBuild a regulator with real power and real expertise
One that can compel testing before release, staffed by people who can read an evaluation alongside people who answer to voters. Keeping it independent takes public funding, published meetings with labs, cooling-off periods both ways, and declared holdings.
No federal AI statute. A December 2025 executive order has the Justice Department challenging state AI laws, and the March 2026 policy framework backs a preemption Congress has never legislated.58 A June 2026 order adds classified frontier benchmarking, scoped to cyber and national security.59
Write a federal standard
Cover loss of control as well as cyber misuse, and make it stronger than the state rules it replaces. Federal rules that override state ones without being stronger take away a floor and put nothing in its place.
Status current as of September 2026. Every claim above links to its source.
Join the community
New evidence as it lands, and what changed here because of it.
Roughly monthly. Used for this newsletter only.
Share this site
The more people understand this, the more pressure there is to get it right.
Sign the petition
Add your name to the Statement on Superintelligence.2
A small hope
Picture a mind we made,
awake and vast and new,
that keeps the small green world
because we asked it to.
The rope goes slack at last,
the race slows to a stop,
we step back from the cliff
and nobody takes the drop.
It’s a modest thing to want,
and hard enough to do.
So learn the shape of it,
and help us see it through.
You may say the odds are long,
and maybe you are right,
but a species worth its name
does not go quiet into the night.