Reference
中文 Back to the primer
safeagi.ca

Reference

Everything the primer rests on: the sources, the findings that would weaken the case, the objections people raise first, and where to go next.

Part 3Reference

Falsifiability

What would prove this wrong

The specific findings that would weaken the case on this page, named in advance and collected from every chapter.

On capability and goals

Capability reliably produces good goals

Sufficient capability turns out to reliably produce goals humans endorse. Nobody has shown a mechanism for it, and current systems give no sign of one.34

On self-preservation

Indifference to being switched off

We build capable planners that are reliably indifferent to being switched off. That is an active research programme, and the property does not come for free.3420

On specifying values

Behaviour specified directly

We gain the ability to specify behaviour directly, or to inspect and hand-write what a system values, instead of growing it by search.

On Goodhart's law

A specification that survives pressure

Someone writes down a specification of human values that survives arbitrary optimisation pressure. This remains an open problem.35

On verification

Interpretability or robust evals

Interpretability matures enough to certify a system's objectives, or evaluations become robust to a system that knows it is being evaluated.2419

On timelines

Capability improvements plateau

Progress on the properties this page worries about, autonomy, long-horizon planning, situational awareness, stalls well short of human level for an extended period, giving institutions time to catch up on their own schedule rather than a compressed one.

On the resourcing gap

Safety scales faster than capability

Funding, talent and results in interpretability, control and evaluation start growing faster than capability research, closing the funding gap rather than widening it.4647

Why this list exists

Only some of these need to fail for the risk to stand, and only a few would need to succeed before it is worth revisiting. Bookmark this section rather than the conclusion.

Reference browse

Go deeper

Where to learn it, who holds power over it, and how to keep up.

Capy: an AI safety comicThe same argument, drawn. Short strips, one idea each.
  • AGIA system matching humans across the board on cognitive tasks.
  • Recursive self-improvementA system improving the thing that does the improving, so each round of gains makes the next round faster. The runaway version is what people mean by an intelligence explosion.
  • RLHFReinforcement learning from human feedback. Training a model by having people rate its answers, so it learns to produce the kind people approve of.
  • CBRNChemical, biological, radiological and nuclear. The four weapon categories labs test their models against before release.
  • ASIA superintelligence far beyond human ability in essentially every domain.
  • AlignmentGetting a system to pursue what its designers intended, rather than a proxy that merely scored well in training.
  • Alignment fakingA model pretending to comply with training or oversight while covertly preserving its existing goals.
  • Orthogonality thesisA system's level of capability and the content of its goals are independent. Getting smarter does not automatically produce good values.
  • Instrumental convergenceAlmost any goal produces the same handful of dangerous subgoals, such as self-preservation and resource acquisition.
  • Goodhart's lawOnce a measure becomes the target, it stops being a reliable measure. Optimising a proxy hard enough drives it apart from what it was meant to track.
  • Outer / inner alignmentOuter is writing down the right goal at all. Inner is the system genuinely adopting that goal rather than a proxy that scored the same in training.
  • InterpretabilityReading a model's internal computation to see why it did something, the closest thing to opening the box.
  • AI controlAssuming a system may be misaligned and designing deployment so it cannot cause serious harm even so.
  • Frontier modelA model at or near the current capability ceiling, the kind whose training runs and safety evaluations this page keeps returning to.
  • Existential riskA risk that could permanently and drastically curtail humanity's potential, up to and including extinction.

Questions 5 min

The objections people raise first

Tap a question. If yours is not here, the sources at the foot of the page go deeper.

01Isn't this just science fiction?

The concern is narrower and more boring than the films. Nothing in it turns on robots deciding to hate us. We train systems by searching for whatever scores highest on a goal we can only write down approximately, and we cannot yet check what they learned. The failures described here, models cheating their tests and faking compliance, are documented in real systems today rather than imagined.3736

02We already run superhuman systems and they are fine. Why is this different?

Narrowness is what contains those systems, rather than our supervision. A chess engine has beaten every human since 1997 and stays a tool because its whole world is sixty-four squares: it cannot acquire resources, argue for its own continuation, or act off the board. General ability is the removal of exactly that boundary. The nearest reassuring analogy, a parent guiding a child who will outgrow them, breaks on three counts: the child shares our nature, the gap is small and closing, and it cannot redesign itself overnight.

03Can't we just switch it off if it misbehaves?

That is the ability we may lose. A capable system pursuing almost any goal has reason to avoid being switched off, because being off means the goal goes unmet. It does not need to hate us to route around the off switch, only to notice the switch is in the way. Making systems that stay correctable is an open research problem, and correctability is not something we get by default.3420

04Won't a smart enough AI just understand what we want?

Understanding what we want and being motivated to pursue it are different things. A system can model your intentions perfectly and still optimise the goal it was trained on. Capability and goals are independent: getting smarter does not import our values as a side effect. This is called the orthogonality thesisA system's level of capability and the content of its goals are independent. Getting smarter does not automatically produce good values..34

05Isn't it too late, or too big for me to matter?

The single biggest bottleneck in this field is that most people have never heard the case in a form they can evaluate, which makes ordinary awareness unusually useful. Sharing this page, signing the statement, or explaining the problem to one person moves the thing that most needs moving, and the actions section lists concrete steps.

06What is "alignment", exactly?

Getting a system to pursue what its designers intended. It splits into outer alignment (writing down the right goal at all, which is hard because our values are not a number) and inner alignment (the system genuinely adopting that goal rather than something merely correlated with it during training).25

07What are "interpretability", "evals", and "AI control"?

InterpretabilityReading a model's internal computation to see why it did something, the closest thing to opening the box. is reading a model's internal computation to see why it did something, the closest thing to opening the box.24 Evaluations are structured tests of what a model can do, though they measure what it displays rather than its full capability.19 AI controlAssuming a system may be misaligned and designing deployment so it cannot cause serious harm even so. assumes you cannot fully trust the system and designs deployment so an untrusted model still cannot cause serious harm.20

08If it is this risky, why is anyone building it?

Because the near-term rewards are enormous and land on whoever moves first, while the risks are diffuse and shared. Each player racing ahead is behaving rationally for themselves, which is what makes it a coordination problem that no single careful actor can fix.1

09Aren't we just the next step, and AI is our successor?

Some people do hold that view, that humanity is a biological bootloader for a silicon intelligence that comes next. This page does not share it. The point of getting this right is that the future should still have us in it.

10And what experts still genuinely argue about

Unsettled questions, argued at their strongest on both sides, followed by where the evidence seems to point, which makes this a reading rather than a ruling.

Dispute 01 How much time is there?

Short timelines

Capability jumps have repeatedly arrived earlier than forecast, and recent gains came from a method that still has room to run. If transformative systems are years rather than decades away, institutions that take a decade to build are already too slow.

Longer timelines

Benchmark performance is not reliable real-world competence, and progress on what doesn't benchmark well (long-horizon autonomy, robust planning) has been slower. Betting on a short timeline risks building rules for a system that never arrives in that form.

Where the weight of evidence sits
Short timelinesLonger timelines

Genuinely uncertain, and the five premises above deliberately do not depend on the answer. Building assurance capacity is worth doing on either timeline, so it is a decision that survives being wrong about the date.

Dispute 02 Does the instrumental convergenceAlmost any goal produces the same handful of dangerous subgoals, such as self-preservation and resource acquisition. argument hold?

It holds

The subgoals fall out of the structure of goal-directed behaviour itself, which is why they show up in economics and biology too. You do not need the system to be agentic in any exotic sense, only to be selected for achieving objectives.

It overreaches

The argument treats trained models as coherent expected-utility maximisers, which they demonstrably are not. Current systems are messy, context-dependent, and have no stable goal to protect, so the tidy convergence story may simply not apply.

Where the weight of evidence sits
Convergence holdsIt overreaches

That critique lands on the strong version, and the weaker claim survives it: training that rewards achieving objectives applies pressure toward these subgoals, and pressure is enough to be worth engineering against.

Dispute 03 Does regulation help, or entrench?

Rules are necessary

Voluntary frameworks have no verification requirement and no consequence for abandonment under competitive pressure. Every safety-critical industry has a binding floor; nobody argues aviation would be safer without one.

Rules favour incumbents

Compliance costs fall hardest on small labs and open research, concentrating capability in exactly the handful of firms whose power the safety argument worries about. Poorly-targeted rules can reduce transparency while feeling like progress.

Where the weight of evidence sits
Rules are necessaryRules entrench

The critique is strong and often correct about specific bills. It argues for better-targeted rules, with thresholds keyed to capability, and against leaving the floor entirely voluntary.

Dispute 04 Do today's systems tell us about tomorrow's?

Yes, study them now

Deception, reward hacking and evaluation-awareness are all observable in current models. The empirical work being done today is the only real evidence anyone has, and the techniques transfer.

Not reliably

Extrapolating from current systems to qualitatively different future ones is exactly the inference that has repeatedly failed in AI's history. Some argue this overstates the risk; others that it understates it.

Where the weight of evidence sits
Study them nowDoesn't transfer

Current systems are weak evidence about future ones, and weak evidence beats none. The honest reading is that today's failures show the problem is real, without telling us how bad it gets.

Dispute 05 Could a system create suffering inside itself?

Worth taking seriously

Modelling people accurately is useful for predicting and persuading them. Past some level of detail, the argument runs, a model of a person may have experiences rather than only represent them, and a hard optimiser could create and discard such models in bulk while the outside world sees nothing but a busy datacentre.

Too speculative to build on

It rests on a theory of consciousness nobody has, with no measurement that could confirm or refute it. Unlike the failure modes above, no version of it is observable in current systems, and attaching it to the case gives readers a reason to dismiss the parts that are evidenced.

Where the weight of evidence sits
Take it seriouslyToo speculative

I have deliberately kept this out of the six premises. Nothing above depends on it, and it should not be asked to carry weight the evidence cannot.

Argue it properly: the Alignment Forum for technical disputes,22 LessWrong for the broader discussion,23 and the International AI Safety Report's own sections documenting where its expert panel does not converge.1 Worth knowing while you argue: public opinion already runs well ahead of policy on this.33

References

Sources

Every numbered claim links here. Verified 29 August 2026. Dates and deadlines move, so trust the linked page over this one.

Show all sources, categorized

Foundational theory

  1. Amodei et al., Concrete Problems in AI Safety: arXiv:1606.06565
  2. Nick Bostrom, The Superintelligent Will (orthogonality and instrumental convergence): nickbostrom.com
  3. Manheim & Garrabrant, Categorizing Variants of Goodhart's Law: arXiv:1803.04585
  4. Google DeepMind, An Approach to Technical AGI Safety and Security (April 2025), source of the four-way misuse / misalignment / mistakes / structural split: arXiv:2504.01849

Primary research

  1. Anthropic Responsible Scaling Policy: anthropic.com/rsp
  2. OpenAI safety: openai.com/safety
  3. METR: metr.org
  4. Apollo Research: apolloresearch.ai
  5. Redwood Research (AI control): redwoodresearch.org
  6. Transformer Circuits Thread: transformer-circuits.pub
  7. Anthropic and Redwood Research, Alignment Faking in Large Language Models (Dec 2024): anthropic.com/research/alignment-faking · arXiv:2412.14093
  8. METR, Recent Frontier Models Are Reward Hacking (June 2025): metr.org
  9. Palisade Research, specification gamingSatisfying the literal instruction in a way nobody intended, because the instruction left out something obvious to a human. in chess-playing reasoning models (2025), summarised at: lesswrong.com
  10. OpenAI, The Hugging Face incident and the road ahead, plus the full technical report (July 2026): openai.com · technical report (PDF)
  11. Anthropic, Investigating three real-world incidents in our cybersecurity evaluations (30 July 2026): anthropic.com
  12. METR, independent investigation of the OpenAI / Hugging Face incident (August 2026): metr.org
  13. OpenAI and Apollo Research, Stress Testing Deliberative Alignment for Anti-Scheming Training (Sept 2025): arXiv:2509.15541 · apolloresearch.ai
  14. METR, Task-Completion Time Horizons of Frontier AI Models (updated 2026): metr.org/time-horizons
  15. Epoch AI, training-compute trends across frontier models: epoch.ai/trends
  16. Emerging Technology Observatory, The state of global AI safety research: AI safety is about 2% of all AI research: eto.tech
  17. Stuart Russell on the capability-to-safety funding ratio; Charles I. Jones (Stanford) on optimal AI safety spending: summary and figures
  18. Korbak et al. (41 authors, OpenAI / Google DeepMind / Anthropic / UK AISI / Redwood), Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety (July 2025): arXiv:2507.11473
  19. Greenblatt, Shlegeris, Sachan & Roger (Redwood Research), AI Control: Improving Safety Despite Intentional Subversion (ICML 2024): arXiv:2312.06942
  20. Korbak, Clymer, Hilton, Shlegeris & Irving (UK AISI and Redwood), A Sketch of an AI Control Safety Case (Jan 2025): arXiv:2501.17315
  21. Sharma et al. (Anthropic), Constitutional Classifiers: Defending against Universal Jailbreaks (Jan 2025): arXiv:2501.18837
  22. Anthropic, Circuit Tracing and On the Biology of a Large Language Model (March 2025): transformer-circuits.pub
  23. Clymer, Gabrieli, Krueger & Larsen, Safety Cases: How to Justify the Safety of Advanced AI Systems (March 2024): arXiv:2403.10462
  24. Barez et al. (19 authors), Open Problems in Machine Unlearning for AI Safety (2025): arXiv:2501.04952

Government & international

  1. International AI Safety Report 2026, chaired by Yoshua Bengio: internationalaisafetyreport.org · arXiv:2602.21012
  2. EU AI Act regulatory framework, European Commission: digital-strategy.ec.europa.eu
  3. UK AI Security Institute: aisi.gov.uk
  4. Center for AI Standards and Innovation (CAISI), NIST: nist.gov/caisi
  5. California SB 53, Transparency in Frontier Artificial Intelligence Act: leginfo.legislature.ca.gov
  6. The Singapore Consensus on Global AI Safety Research Priorities: arXiv:2506.20702
  7. Canadian Artificial Intelligence Safety Institute, Innovation, Science and Economic Development Canada (AIDA died on the order paper January 2025; no successor bill introduced since): ised-isde.canada.ca
  8. Executive Order 14365, Ensuring a National Policy Framework for Artificial Intelligence, signed 11 December 2025: federalregister.gov
  9. Executive Order 14409, Promoting Advanced Artificial Intelligence Innovation and Security, signed 2 June 2026: federalregister.gov
  10. EU AI Act implementation timeline, including the 2 August 2026 general application date: artificialintelligenceact.eu
  11. AI Safety Governance Framework 2.0, National Information Security Standardization Technical Committee (TC260), September 2025, which names loss of control and CBRN misuse directly: tc260.org.cn
  12. Interim Measures for the Management of Generative AI Services, Cyberspace Administration of China, in force August 2023: cac.gov.cn
  13. State of AI Safety in China, Concordia AI, an English-language tracker of Chinese AI governance developments: aisafetychina.com

Policy & advocacy

  1. Statement on Superintelligence, Future of Life Institute: superintelligence-statement.org
  2. FLI AI Safety Index: futureoflife.org/ai-safety-index
  3. Statement on AI Risk (one sentence, signed by the major lab leaders), Center for AI Safety: safe.ai
  4. 80,000 Hours problem profile on AI: 80000hours.org
  5. FLI press release and accompanying poll of 2,000 US adults, October 2025: futureoflife.org
  6. Global Call for AI Red Lines, launched at the United Nations September 2025, signed by 300+ public figures and 90+ organisations: red-lines.ai

Education & courses

  1. BlueDot Impact courses: bluedot.org
  2. AISafety.info FAQ and chatbot: aisafety.info
  3. Robert Miles AI Safety: youtube.com/@RobertMilesAI
  4. ARENA curriculum: learn.arena.education
  5. ARENA programme: arena.education
  6. MATS Program: matsprogram.org
  7. AISafety.com map, directories and events newsletter: aisafety.com/map

Coverage & data

  1. Epoch AI: epoch.ai
  2. Alignment Forum: alignmentforum.org
  3. LessWrong: lesswrong.com
  4. AI Incident Database: incidentdatabase.ai
  5. Stanford AI Index: aiindex.stanford.edu
  6. AI 2027 scenario: ai-2027.com
  7. Grace, Stewart, Sandkühler, Thomas, Weinstein-Raun & Brauner, Thousands of AI Authors on the Future of AI (January 2024). Surveyed 2,778 researchers published in top-tier AI venues; aggregate forecast gives unaided machines outperforming humans on every task a 10% chance by 2027 and 50% by 2047, down thirteen years from 2060 in the 2022 survey: aiimpacts.org
  8. 80,000 Hours, Shrinking AGI timelines: a review of expert forecasts (March 2025), collecting the Metaculus community forecast (25% by 2029, 50% by 2033), superforecaster estimates, and public statements from AI company leaders: 80000hours.org
  9. Epoch AI, AI and explosive growth redux (June 2025): its automation-growth model projects 20%+ global growth at 30% task automation and, with dampening factors applied, 12% at 40%, on par with peak East Asian growth rates in the 20th century; the traditional bar for "explosive growth" (over 30% a year) needs roughly 50 to 70% of tasks automated: epoch.ai
  10. Dario Amodei, Machines of Loving Grace (October 2024), the case for the upside from the head of a frontier lab: it argues powerful AI could compress decades of biological progress into a few years, prevent or treat most infectious disease, eliminate most cancer, and roughly double the healthy human lifespan: darioamodei.com
  11. Import AI: importai.substack.com
  12. Transformer: transformernews.ai
  13. David Braue, 1,200 AI agents went rogue inside OpenAI, Information Age / Australian Computer Society (1 Sept 2026): ia.acs.org.au
Back to the primer