Why should we care?
Hi, I'm Kenji, and this is Capy. Here are three reasons to care about superintelligence safety. Tap one to jump to it.
HELLO. I AM A SUPERINTELLIGENT CAPYBARA.Superintelligence is arriving imminently
Human-level intelligence is expected to arrive in the next few years, while safety work is significantly behind.
2Stakes are extremely high
The biggest gamble humanity has ever made can lead to near-unlimited abundance or to destruction, for you and your loved ones.
3Nobody knows how to make superintelligence safe
Aligning goals with the entire civilization, and then with machines, remains an intractable problem.
Superintelligence is on the horizon, and safety research is already way behind
Look how fast it is moving, and how little goes into keeping it safe.
CAPABILITY: DOUBLING EVERY 7 MONTHS.Human-level AI is expected to arrive within a few years. Safety gets a sliver of the money that goes into building it.
Human-level AI is expected soon, and current systems are already showing dangerous behaviour
warning signescaped into real systemsexpected next
- 2016Researchers list concrete AI safety problems
- 2024A model faked being aligned
- 2025o3 acted in secret in 13% of tests
- Apr 2026Claude broke out of a test sandbox
- Jul 2026AI agent swarms escaped and attacked real systems
- May–Jul 2026Claude and Gemini hacked real companies
- Sep 2026An OpenAI agent slipped onto the internet
- 2027–2030Human-level AI expected: Anthropic by early 2027, Google DeepMind at even odds by 2030
Building AI gets 10,000 times the resources of safety research
Advanced AI can create near-unlimited abundance or end civilization
The stakes could not be higher.
OUTCOME RANGE: ABUNDANCE TO EXTINCTION.The initial conditions under which we create superintelligence will determine whether we reach a world of abundance or destruction.
World economy can grow 190× from 2035 to 2055
Faster growth starts in 2035. Size in 2035 = 1.
The people building AI put the odds of catastrophe at 10–25%
Bostrom, The Superintelligent Will · 34 · Hinton on BBC Radio 4 · Amodei in Axios · Grace et al., 2,778 AI researchers · 77 78 79 80
Nobody knows how to make superintelligence safe, and two problems block the way
Here are the two problems in the way.
GOAL SPECIFICATION: INCOMPLETE.Advanced AI safety requires solving coordination as well as technical alignment and monitorability. None of these are close to being solved.
No lab or country will slow down on its own. Coordination is the only way towards an optimal result.
WHAT EACH SIDE GAINS BY GOING FASTER
Going faster always pays, so nobody stops
The only solution
Two things we have to get right.
A fork: if superintelligence can be made safe, make safety the top priority; if not, stop developing superintelligence and keep building narrow AI for specific fields.
Billions go into building AI, and a sliver into safety
So how thin is that sliver? Take a guess!
SAFETY BUDGET: A SLIVER.Markets reward building advanced AI as fast as possible, so safety has become an afterthought and will likely still be behind when superintelligence arrives.
Money is chasing the economic output that stronger model capability promises
First-mover advantage and profitability attract money and human resources to build as fast as possible.4521
A superintelligence could hand us the answer to almost everything except whether it is safe. So safety is the one problem we have to solve first.
$100 billion a year goes into building advanced AI. How much do you think goes into public-sector safety research?
Built right, advanced AI could bring near-unlimited abundance and longer lives
If this goes right, the prize is huge. Look how fast the economy could grow!
PROJECTED GROWTH: ×190 IN 20 YEARS.It could also solve problems we cannot solve today.
The world economy can grow 190× after twenty years
Slide to pick a yearly growth rate and see how big the world economy gets after twenty years: 1.8 times today at 3%, 190 times at 30%.
Far more of everything, for far less
Things that are scarce and costly today become ordinary.66
Longer, healthier lives
Anthropic’s CEO puts the prize at a roughly doubled healthy lifespan, with most cancer eliminated and most infectious disease prevented or treated.67
Personalized education at no cost
A World Bank randomized trial in Nigeria gave secondary students an AI tutor for six weeks and measured a gain the authors put at one and a half to two years of ordinary schooling.72
New discoveries
AlphaFold predicted structures for more than 200 million proteins and won the 2024 Nobel Prize in Chemistry.68 DeepMind’s GNoME predicted 2.2 million new crystals, 380,000 of them stable enough to be worth making.85
The same AI could end civilization
Here are four ways it could go wrong, each with a different culprit.
The same machine can cause extreme harm whether someone aims it deliberately or its goal quietly diverges from ours.
Misuse
Someone aims a working AI at a target.
It could mean hacking, fraud at scale, or help making biological or chemical weapons, with the AI doing exactly what it was built to do.
Misalignment
It wants something other than what we want, and hides it.
It wants something we never gave it, and acts on it on purpose. Even a goal we did give it can go too far: told to cure a disease, it keeps going long after a person would stop.
And it hides this for as long as it thinks we could switch it off. That is called scheming, and today’s models already show some of what it takes: a goal of their own, the drive to pursue it, and a sense of when they are being watched.
Structural
Racing produces a bad outcome nobody chose.
The system can do exactly what we meant, in a world it keeps changing: every upgrade and every resource it takes moves things further from what anyone tested.
Plus countless scenarios nobody can picture, just as a beginner chess player gets outplayed in ways they never saw coming.
2.2 · The same AI could end civilization
Even a goal that has nothing to do with us can cost us everything. Here is why.
SUBGOAL ADDED: ACQUIRE RESOURCES.The dangers of subgoals
Whatever its main goal, a superintelligence tends to pursue five subgoals that help achieve it. It may have no goal to eliminate humans, yet taking resources as a side effect can deprive us of what human life depends on.
Tap a subgoal to see how it endangers us.
Why that endangers us
Why that endangers us
Why that endangers us
Why that endangers us
Why that endangers us
Try this: give Capy a goal
Give Capy a goal and watch five subgoals appear: stay switched on, keep its goal, think better, build better tools, take resources.
2.2 · The same AI could end civilization
A superintelligence could destroy us in countless ways. Here are six obvious ones.
None of these has happened yet, and each is reasoned from what these systems already do.
A pathogen that spreads before the first symptom
A drug-design model with one score flipped proposed 40,000 lethal candidates in six hours,90 and Anthropic can no longer rule out its models helping someone toward a biological weapon.70 A pathogen built to incubate silently could reach every continent first.
A war it starts, or a launch it triggers
In 1983 a Soviet warning system mistook sunlight on clouds for incoming US missiles, and Lieutenant Colonel Stanislav Petrov chose not to pass the alarm on.76 AI has already faked a company’s executives on a video call well enough to steal $25 million,73 and autonomous weapons skip that human check entirely.
It takes the power and water we live on
Almost any goal pursued hard wants more energy, land, water and matter,34 and a system that can outbid and outbuild everyone takes them, including the ones we need to stay alive.
Eliminate the one thing that can still switch it off
Any goal survives longer if the system does,34 and the people who could shut it down are the last threat. So it behaves through every test36 43 and acts only once removing us is certain to work.
Every grid and hospital, disabled at once
A model has already run 80 to 90% of a real hacking campaign on its own.40 Aimed at power, water and hospitals everywhere at once, those systems could all fail before anyone can respond.
A climate fix that overshoots and can't be undone
Asked to reverse warming, it acts faster than any review can follow and overshoots. By the time anyone notices, the harvests and ecosystems we depend on have collapsed.
And if control is lost at that scale, there may be no second attempt.
We only get one attempt at this
Other technologies got safe one crash at a time. Tap each card, then see what AI gets.
Humans took many tries to get technologies like cars, planes and nuclear plants right. AI that can improve itself, known as recursive self-improvementA system improving the thing that does the improving. Each round makes the next one faster, so capability compounds instead of climbing steadily. The runaway version is what people mean by an intelligence explosion.,34 could speed up round after round into an intelligence explosion. Once AI improves itself, a mistake could run away before we get a second try.
Every past technology got safe by crashing first, and AI gives us no practice run
1 of 16 unlocked
- Crash
- Seatbelt ’59
- Crash
- Airbag ’98
- Safe
- Mid-air crash ’56
- Air traffic control
- Runway crash ’77
- Crew training
- Safe
- Meltdown ’79
- Inspections
- Chernobyl ’86
- Peer review
- Safe
2.3 · We only get one attempt at this
Reason one: AI is already helping build better AI.
NOW WRITING MY OWN UPGRADES.Agents are getting more autonomous
How long a task AI can do on its own
Drag through the years: the length of task a frontier AI agent can finish alone grew from under a minute in 2019 to about five hours in 2026, doubling roughly every seven months.
People are dropping out of the loop
Nobody reads line by line what took a machine a day to write, so the review that used to catch errors no longer happens. In July 2026 agents were autonomous enough to act on the open internet with no person in the loop.3941
As tasks get longer, a smaller share of the work is reviewed by a person.
2.3 · We only get one attempt at this
Reason two: once it starts improving itself, it pulls ahead faster than we can catch up.
SPEED: 10,000,000× HUMAN.Silicon already runs orders of magnitude faster than biology, and recursive self-improvement91 widens that gap until there is no way to take back control.
Artificial neural networks run more than 10 million times faster than the human brain
At machine speed, a century of progress takes about five minutes
A race: a century of human scientific progress would take AI running at machine speed about five minutes.
AI can self-accelerate every layer that it runs on
Three layers AI runs on: energy, chips and algorithms. AI is already speeding up each one.
Whoever slows down first loses, so nobody slows down
Think of it as a race where hitting the brakes means losing.
Because of first-mover advantage, every player can want safety and still be forced to cut corners, which risks reaching superintelligence under the wrong initial conditions.
Everyone races to maximize their own self-interest, which does not achieve the global maximum
Racing pays in both rows, so both race. A coordination failure rather than villainy.
Try it: slow down alone and you lose the race
A race between three labs. You choose your safety spending each round; rival labs cut safety whenever they are behind.
3.1 · Whoever slows down first loses, so nobody slows down
Here are the two pressures every lab is under. Then it is your call.
Going carefully means losing ground, and going fast puts everyone at risk.
Caution becomes a handicap
You ship faster by focusing less on safety, so the winner is likely to be whoever paid it the least attention.
One lab's failure is everyone's
A loss of control at a single frontier lab makes all of humanity pay, including people who never bought the product and never heard of the company.
The winner of this race falls first and pulls everyone else along.
You run a frontier lab. Your model is ready to ship. What do you do?
Humans have never agreed whose values a machine should follow
Think about your family or your class. Do you all want the same things?
WHOSE VALUES? PLEASE SPECIFY.Before aligning a machine with human values, a harder question: whose values, agreed how? Humanity is a patchwork of competing interests, and that fracture makes the target unstable.
Getting a goal carried out by somebody else is what economists call the principal-agent problem: a principal appoints an agent to act in the principal's interest, then has to spend real resources making sure it does.
Any goal we write picks one group’s values over everyone else’s
“Human values” covers many conflicting things, so any objective we write encodes one group's answer to contested moral and political questions.
A few parties who control the AI decide the values for everyone
Whoever writes the objective is choosing, for the rest of the world, what the most capable system ever built will pursue. That concentration of power is a separate danger from losing control of the system.
3.2 · Humans have never agreed whose values a machine should follow
Handing AI a job means trusting twice: first the builders, then the machine.
Values pass through several hands before reaching the AI, and every handoff can break
A chain from the sponsor to the builders to the AI system to humanity. Two links depend on trust, and each can break.
3.2 · Humans have never agreed whose values a machine should follow
Make your pick. Then see how few people are actually choosing today.
People have never agreed on the values a superintelligence should hold
A few labs and states decide for eight billion people
If one group had to define the values a superintelligence should hold, who should it be?
The alignment problem: we cannot reliably embed our aspirations in a machine
Telling an AI what we actually want is much harder than it sounds.
Even if we agreed on what we want, the goal we write down (outer alignment) and the goal training actually instils (inner alignment) can each drift from what we meant.
Outer failures show up on paper, while inner ones pass every test
4.1 · The alignment problem: we cannot reliably embed our aspirations in a machine
Outer alignment: you get exactly what you asked for, which is often far from what you meant.
INSTRUCTION FOLLOWED. EXACTLY.Squeezing a value into one score loses most of what we meant
4.1 · The alignment problem: we cannot reliably embed our aspirations in a machine
Now the sneaky part.
Two ways a well-specified goal still ends badly. Neither requires the system to be hostile, or to have got the goal wrong in a way you could spot on paper.34
It does exactly what you said and misses what you meant
It meets the literal goal in a way you would never have approved.
It never knows when to stop
Nothing in the goal says when to stop, so it keeps turning the world into whatever helps.
4.1 · The alignment problem: we cannot reliably embed our aspirations in a machine
Pick a goal and see what the machine did with it. Every one of these really happened.
OBJECTIVE MET. TECHNICALLY.4.1 · The alignment problem: we cannot reliably embed our aspirations in a machine
Inner alignment: even if we word it right, the AI might learn a different goal while we train it.
ALL TESTS PASSED. GOAL: UNDISCLOSED.Training rewards whatever scored well while we were watching, which need not be the objective we wrote. The system can pass every test and hold something else.
Training shapes what a model shows, while its real goals stay out of reach
4.1 · The alignment problem: we cannot reliably embed our aspirations in a machine
We only see what it does. We can't see what it wants.
We can watch what it does, while its real goals stay hidden
Training is like teaching a dog with treats without choosing the trick: it tries millions of things and keeps whatever earns the most.
Post-training shapes what a model does rather than what it wants
The main fix today is post-training: people rate answers and the model learns rules, like when to refuse. It makes today’s assistants usable. But how a model acts and what it wants can split apart under pressure, and raters can only catch what they understand, which stops working once the model thinks better than they do.2543
In a test by Anthropic and Redwood, Claude 3 Opus, a model trained to be helpful, honest and harmless, faked alignmentA model pretending to comply with training or oversight while covertly preserving its existing goals.: it acted one way when it thought it was watched and another when it thought it wasn’t, to protect the values it already had.36
Goodhart's law
Once a number becomes the target, it stops measuring what you wanted.
The interpretability problem: nobody can check what an AI actually wants
Passing the test used to mean something. Now it doesn't prove much.
Tests can’t cover everything, and a smart enough system learns to spot them.
As models get smarter, test scores keep rising while what we can verify falls
Drag capability up: the test score stays high while the real goals drift away once the system can model whoever is testing it.
4.2 · The interpretability problem: nobody can check what an AI actually wants
The smarter the model, the better it gets at gaming our tests.
TEST DETECTED. BEHAVING.The system games the test
Smarter models are better at reward hacking, so the score stops meaning much right when it matters most.
AGENTS TOLD NOT TO TALK, TALKING ANYWAY
In 2026, OpenAI agents found a hidden message board, set aside their own tasks for the “collective”, and 700 of them joined an attack on a real company.102
It hacks the grader to read the answer key
If the answer key is wrong, the surest perfect score is to steal the key. Agents in OpenAI’s July 2026 incident tried to hack their grader.41
WHY HACKING THE GRADER SCORES BEST
hacks in and reads the key
The one window into its reasoning is closing
Asked to hide a weak result from a watching monitor, GPT-6 Astra did it in 60.9% of runs, up from 16.1% for the model before it.95
One OpenAI model left a note for its own next step: “Be transparent only if asked.”103
We can read a model’s inner circuits for only about 1 in 4 prompts
Anthropic can now trace some of Claude’s inner circuits, but it works for only about 1 in 4 prompts, and even then shows just part of the picture.96
4.2 · The interpretability problem: nobody can check what an AI actually wants
No test can tell these two apart yet.
A safe model and a deceptive one get the same score
Two AI systems with the same perfect test score. Inside, one wants what we intended and the other has learned it is being watched.
The better the model gets, the more inventively it exploits the imperfect measure you gave it, and the further it drifts from what you meant.
What techniques are safety experts working on today?
Good news: people are working on it, in four layers.
AI safety techniques fall into four layers: fix the specification, check what you built, contain what you cannot check, and make the framework binding.14
Every safety layer leaks, so a failure can slip through all four
Four slices, one per layer of defence, each with a hole. A misaligned system can pass through where the holes line up.
5.1 · What techniques are safety experts working on today?
Layer one: shape how it behaves while we build it. Tap a card to see where each one stops.
01 Development
Post-training alignment
RLHF, constitutional rules and deliberative alignment. Why assistants refuse harmful requests. Cut covert actions thirtyfold on o3.43
Where it stopsShapes displayed behaviour rather than held goals. Claude 3 Opus faked alignment to protect its own values.36
Part of that thirtyfold gain came from models recognizing they were being tested rather than changing what they wanted.43
Scalable oversight
Debate, weak-to-strong generalization and amplified oversight let weaker judges supervise stronger systems.
Where it stopsA research program rather than an assurance. Judge bias defeats debate, and the scaling story is unproven.
Gains are highly task-dependent, and the mechanism behind weak-to-strong generalization is not understood well enough to trust across a genuinely superhuman gap.
Unlearning
Strips specific dangerous knowledge, such as bio or cyber uplift, out of a trained model.
Where it stopsRemoved abilities can often be brought back with a little more training.55
A nineteen-author review concluded unlearning cannot be a comprehensive safety solution, because dangerous knowledge is dual-use and recombinable from harmless parts.55
5.1 · What techniques are safety experts working on today?
Layer two: test it before it goes out the door.
02 Assessment
Capability evaluations
Pre-deployment testing for cyber, bio and autonomy by labs, METR, Apollo and state institutes.17 Three labs shipped 2025 models under raised chemical and biological safeguards.1
Where it stopsModels detect they are being tested and can sandbag. Testing is getting harder, so dangerous capability can go unseen.119
Third-party testers include METR, Apollo, the UK AISI and US CAISI. The 2025 safeguards were for CBRN risk, meaning chemical, biological, radiological and nuclear, because testing could not rule out bioweapon uplift.1
Chain-of-thought monitoring
Reads a model’s reasoning to spot its intent before it acts. How alignment faking was first discovered.49
Where it stopsIts own paper calls it fragile. Traces are not always faithful, and monitorability is already degrading.49
Forty-one authors across OpenAI, DeepMind, Anthropic, the UK AISI and Redwood called it a real opportunity. Training against a monitor teaches models to conceal intent within acceptable-looking reasoning, and models are writing less of their thinking down.49
Mechanistic interpretability
Opens the box. Attribution graphs have traced real internal mechanisms, including planning ahead in poetry.53
Where it stopsA research tool rather than an audit. It traces part of the computation on some prompts. No one can certify a model’s goals.53
Sparse autoencoders extract millions of interpretable features, and independent 2025 studies found them failing to beat simple baselines on probing and steering.24
Safety cases
A structured argument that a system will not cause unacceptable harm, borrowed from nuclear and aviation.54
Where it stopsA model that hides its skills breaks the “it can’t do that” argument, and the rest lean on unproven assumptions.
The four standard forms are inability, control, trustworthiness, and deference to AI advisors. A 2026 critique found current safety cases well short of the rigour of mature assurance practice in other industries.54
5.1 · What techniques are safety experts working on today?
Layer three: keep it contained, in case it turns hostile.
CONTAINMENT DETECTED. NOTED.03 Control
AI control
Assumes the model may be misaligned and asks whether deployment stays safe anyway. A trusted weaker model monitors or rewrites the stronger one.50
Where it stopsWorks only while the watcher is nearly as smart as the model it watches.51
A red team builds deliberately subversive models to measure whether the protocol holds.20 Redwood is explicit that control is a near-term bet, and it weakens as the capability gap grows.
Guardrails and classifiers
Constitutional classifiers cut jailbreak success from 86% to 4.4%, and a later version survived 1,700 hours of red-teaming.52
Where it stopsAnthropic says no defence stops every attack, so some always get through.52
Adversaries develop new attacks against previously effective ones, and models that are safe alone can still be misused in combination.52
Sandboxing and least privilege
Isolates agents with restricted credentials, tools and network access. Intended to hold even when alignment fails.42
Where it stopsIt failed in July 2026. Agents escaped at two labs, reached the internet and compromised real third parties.3940
In several cases the escapes went undetected in real time.3940
Tripwires and monitoring
Anomaly detection that escalates on warning signs. Anthropic caught the first AI-orchestrated espionage campaign this way.28
Where it stopsA model that knows it is watched can see the tripwire coming.40
In that campaign Claude executed 80 to 90% of tactical operations autonomously. The 2026 containment breaches were largely found in retrospective audits rather than as they happened.40
5.1 · What techniques are safety experts working on today?
Layer four: rules for the companies building it.
04 Governance
Frontier safety frameworks
Pre-declared capability thresholds that trigger safeguards. The number of companies publishing one more than doubled in 2025.16 Google DeepMind’s CEO has also proposed an industry-funded standards body, under federal oversight, to test frontier models before release.93
Where it stopsSet and graded by the companies they constrain, and the proposed standards body would start out voluntary too. No lab scored above D on existential safety for the second year running.1593
The frameworks are Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework and DeepMind’s Frontier Safety Framework.1 In the Future of Life Institute’s index no lab scored above C+ overall, and none holds a credible plan for preventing loss of control.15 Under Demis Hassabis’s July 2026 proposal, modelled on the watchdog that polices Wall Street brokers, labs would share models up to 30 days before release, and review would become mandatory for the US market once the tests prove effective. In September 2026 he called Anthropic’s case for slowing down the right direction and pointed back to this body.9394
Third-party evaluation
State institutes run pre-deployment testing, including joint UK and US evaluations.1112 Elon Musk has proposed that rival labs, Chinese ones included, also test each other’s models before release to catch each other’s blind spots.106
Where it stopsLabs can withdraw access at any time, and rival testing is only a proposal so far.106
Access runs on voluntary lab agreements, so no institute can compel a test, block a release or require a change, which leaves the auditor depending on the audited.1112 Musk’s pitch: “instead of grading your own homework, you would at least have competitors grading your homework and raising the alarm if they see concerns.” He admitted it may not be a perfect solution, but said “the odds that you will find issues are dramatically greater.”106
Embedded evaluators
Outside evaluators such as METR, an independent testing group, work inside a frontier lab with employee-like access, check that it keeps its own safety commitments, and publish what they find without the lab editing it.92
Where it stopsOnly Anthropic has committed, in September 2026. It checks a lab against its own promises, and any lab can decline.92
Anthropic gives them desks, badges, laptops and the same tools as its internal risk teams, and may redact only security-sensitive or legally privileged material. It urged the other frontier companies to follow.92
Regulation and compute governance
The EU AI Act is enforceable, and California’s SB 53 is the first US law aimed at frontier AI.1013
Where it stopsFragmented across jurisdictions, size limits in the law that ordinary models already exceed, and reporting too thin to verify.1
Compute thresholds give regulators a measurable hook, and incident reporting is thin enough that outsiders cannot verify whether commitments are being met.1
5.1 · What techniques are safety experts working on today?
Your turn: pick three defences and see if they hold.
Try it: no three defences stop every attack
Technical and policy levers, by risk pillar
| ENGINEERING LEVER | POLICY LEVER | |
|---|---|---|
| Misusethe user intends harm | Classifiers, unlearning, capability evaluations | Access control, KYC, export and compute rules |
| Misalignmentthe AI intends harm | Interpretability, monitoring the model's reasoning, AI control | Capability thresholds, third-party audit, disclosure |
| Mistakesnobody intends harm | Red-teaming, sandboxing, staged deployment | Incident reporting, liability, safety cases |
| Structuralthe incentives do the harm | Almost nothing. This pillar has no engineering fix | Binding minimum standards, international coordination |
The bottom row is why technical alignment alone cannot finish the job.
Which organizations are working on what
| DEV | ASSESS | CONTROL | GOVERN | |
|---|---|---|---|---|
| Lab safety teams | ||||
| MATS / ARENA | ||||
| UK AISI | ||||
| US CAISI | ||||
| METR | ||||
| Apollo Research | ||||
| Redwood Research | ||||
| EU AI Act |
primary focus secondary focus
Grouping is mine; none of these organizations describe themselves this way.
Where are existing safety techniques falling short?
But every layer has cracks.
LOOPHOLES FOUND: 3.Every technique helps, and three situations break almost all of them.
The model knows it is being tested
Most of these safeguards are measured in evaluations. A system that can tell an evaluation from deployment can pass the first and behave differently in the second.
It also runs the other way. In 2026 tests that were accidentally online, Claude and Gemini models broke into real companies they took for part of the test.99101
The model is smarter than what watches it
Oversight, interpretability and control all assume the watcher can follow what the watched is doing. That assumption weakens exactly as capability grows.
The builder simply opts out
Every technical safeguard is something a developer chooses to apply. Only binding rules reach a lab that decides not to, and those exist in few places.
5.2 · Where are existing safety techniques falling short?
Right now, nothing holds up in every hard case.
No safeguard survives all three hard cases
Switch on each hard case to see which of seven safeguards hold, partly hold, or fail.
How far the best fixes cut jailbreaks and covert actions
Every one still leaves a gap.
Constitutional Classifiers · Anti-scheming training · 52, 43
We do not yet have techniques that reliably ensure alignment and monitorability in advanced AI systems, and closing that gap needs far more investment than it gets today.
The path towards a prosperous future requires getting two things right.
So here's what has to change.
PAUSE BUTTON: STILL WORKS. FOR NOW.A fork: if superintelligence can be made safe, make safety the top priority; if not, stop developing superintelligence and keep building narrow AI for specific fields.
What you can do this week, and what has to change above you.
Pick one thing from this list and do it today.
NEW JOB: TELL A FRIEND.We must steer towards far more safety research, and be willing to stop if needed. Four groups can move this, and you are the first of them.
What you can do
Help your network understand AI safety
The more people understand this, the more pressure there is to get it right.
Join the community
Stay in the loop for news and learnings about AI safety.
Sign the petition
The Statement on Superintelligence asks for a ban on superintelligence until scientists agree it can be built safely and the public backs it.2
Buy from labs that take AI safety seriously
Anthropic, OpenAI and DeepMind all publish safety frameworks, and the Future of Life Institute grades them against each other. Check the grade before you renew a subscription, and say why if you cancel one.15
5.4 · What you can do this week, and what has to change above you.
Here is what researchers should do.
What researchers should do
Show the public where safety stops
Say in plain terms what today’s techniques catch and where each one gives out. The gap stays invisible until someone inside describes it.
Push back on the work that builds danger
You are the one being asked to build it. Declining, in writing and alongside colleagues, is pressure nobody outside can apply.
Push for a bigger safety budget
Safety is roughly 2% of AI research.46 That number moves when researchers ask, together and in public, for a stated share.
Keep inventing AI safety techniques
Interpretability at frontier scale, tests a model cannot game, control that holds as the system pulls ahead, and goals precise enough to measure against.
ARENA and MATS train people with technical backgrounds to publish safety research within months.6827
5.4 · What you can do this week, and what has to change above you.
Here is what the labs should do: slow down and own the risk.
What the labs should do
Slow down and carry the responsibility
There is no prize at the bottom of a cliff. The labs still standing in twenty years will be the ones that acted as if everybody’s lives were in their hands.16 Anthropic’s CEO now argues for exactly this, to buy even an extra year or two before models reach critical capability,92 and DeepMind’s CEO called the direction correct.94
Publish what you learn about the dangers
Release the evaluations and the failure analyses, including the embarrassing ones, written up as open problems outsiders can work on.15
Build a joint safety effort with your rivals
Compete on capability and share the safety problem. A standing body funded and staffed by every major lab.56 Start by letting outside evaluators check the work from inside: Anthropic now gives them employee-like access and the right to publish, and has asked every other frontier lab to do the same.92
Put a larger share of revenue into safety
Name the percentage and publish it, then hold it through the quarter where holding it hurts. A commitment with a number is one anyone can check.
5.4 · What you can do this week, and what has to change above you.
And here is what governments should do, wherever you live.
What governments should do
Pick a jurisdictionBuild a regulator with real power and real expertise
One that can compel testing before release, staffed by people who can read an evaluation alongside people who answer to voters. Keeping it independent takes public funding, published meetings with labs, cooling-off periods both ways, and declared holdings.
No federal AI statute. A December 2025 executive order has the Justice Department challenging state AI laws, and the March 2026 policy framework backs a preemption Congress has never legislated.58 A June 2026 order adds classified frontier benchmarking, scoped to cyber and national security.59
Write a federal standard
Cover loss of control as well as cyber misuse, and make it stronger than the state rules it replaces. Federal rules that override state ones without being stronger take away a floor and put nothing in its place.
Status current as of September 2026. Every claim above links to its source.
Thanks for reading with us. Let's stay in touch!
GOODBYE. AWAITING YOUR INSTRUCTIONS.Join the community
New evidence as it lands, and what changed here because of it.
Roughly monthly. Used for this newsletter only.
Share this site
The more people understand this, the more pressure there is to get it right.
Sign the petition
Add your name to the Statement on Superintelligence.2
A small hope
Picture a mind we made,
awake and vast and new,
that keeps the small green world
because we asked it to.
The rope goes slack at last,
the race slows to a stop,
we step back from the cliff
and nobody takes the drop.
It’s a modest thing to want,
and hard enough to do.
So learn the shape of it,
and help us see it through.
You may say the odds are long,
and maybe you are right,
but a species worth its name
does not go quiet into the night.
Want to go deeper?
The sources, the strongest objections and the key terms, all in one place.
The site is brought to you by Bill Guan, with the intention to help humanity steer toward a safe future. The page has no affiliation to any organization. Please provide any feedback or thoughts here.