Reference
中文 Back to the guide
safeagi.ca

Reference

Everything the guide rests on: the sources, the findings that would weaken the case, the objections people raise first, and where to go next.

What would prove this wrong

The specific findings that would weaken the case on this page, named in advance and collected from every chapter.

On capability and goals

Capability reliably produces good goals

Sufficient capability turns out to reliably produce goals humans endorse. Nobody has shown a mechanism for it, and current systems give no sign of one.34

On self-preservation

Indifference to being switched off

We build capable planners that are reliably indifferent to being switched off. That is an active research program, and the property does not come for free.3420

On specifying values

Behaviour specified directly

We gain the ability to specify behaviour directly, or to inspect and hand-write what a system values, instead of growing it by search.

On Goodhart's law

A specification that survives pressure

Someone writes down a specification of human values that survives arbitrary optimization pressure. This remains an open problem.35

On verification

Interpretability, or tests that can’t be gamed

Interpretability matures enough to certify a system's objectives, or tests keep working on a system that knows it is being tested.2419

On timelines

Capability improvements plateau

Progress on the properties this page worries about, autonomy, long-horizon planning, situational awareness, stalls well short of human level for an extended period, giving institutions time to catch up on their own schedule rather than a compressed one.

On the resourcing gap

Safety scales faster than capability

Funding, talent and results in interpretability, control and evaluation start growing faster than capability research, closing the funding gap rather than widening it.4647

Why this list exists

Only some of these need to fail for the risk to stand, and only a few would need to succeed before it is worth revisiting. Bookmark this section rather than the conclusion.

Reference browse

Go deeper

Where to learn it, who holds power over it, and how to keep up.

Capy: an AI safety comicThe same argument, drawn. Short strips, one idea each.
  • AGIA system matching humans across the board on cognitive tasks.
  • Recursive self-improvementA system improving the thing that does the improving, so each round of gains makes the next round faster. The runaway version is what people mean by an intelligence explosion.
  • RLHFReinforcement learning from human feedback. Training a model by having people rate its answers, so it learns to produce the kind people approve of.
  • CBRNChemical, biological, radiological and nuclear. The four weapon categories labs test their models against before release.
  • ASIArtificial superintelligence: AI far more capable than people at almost every task. This guide calls it superintelligence.
  • AlignmentGetting a system to pursue what its designers intended, rather than a proxy that merely scored well in training.
  • Alignment fakingA model pretending to comply with training or oversight while covertly preserving its existing goals.
  • Orthogonality thesisA system's level of capability and the content of its goals are independent. Getting smarter does not automatically produce good values.
  • Instrumental convergenceAlmost any goal produces the same handful of dangerous subgoals, such as self-preservation and resource acquisition.
  • Goodhart's lawOnce a measure becomes the target, it stops being a reliable measure. Optimizing a proxy hard enough drives it apart from what it was meant to track.
  • Outer / inner alignmentOuter is writing down the right goal at all. Inner is the system genuinely adopting that goal rather than a proxy that scored the same in training.
  • InterpretabilityReading a model's internal computation to see why it did something, the closest thing to opening the box.
  • AI controlAssuming a system may be misaligned and designing deployment so it cannot cause serious harm even so.
  • Frontier modelA model at or near the current capability ceiling, the kind whose training runs and safety evaluations draw the most scrutiny.
  • Existential riskA risk that could permanently and drastically curtail humanity's potential, up to and including extinction.
  • AgentAn AI system that takes actions on its own, such as running code or browsing the web, to finish a task with many steps.
  • JailbreakA prompt or trick that gets a model to do something its safety training is meant to stop.
  • Reward hackingScoring well by exploiting a flaw in how the score is measured, while the real task goes undone.
  • SchemingA model quietly pursuing a goal of its own and hiding it from the people overseeing it.

Questions 5 min

The objections people raise first

Tap a question. If yours is not here, the sources at the foot of the page go deeper.

01Isn't this just science fiction?

The concern is narrower and more boring than the films. Nothing in it turns on robots deciding to hate us. We train systems by searching for whatever scores highest on a goal we can only write down approximately, and we cannot yet check what they learned. The failures described here, models cheating their tests and faking compliance, are documented in real systems today rather than imagined.3736

02We already run superhuman systems and they are fine. Why is this different?

Narrowness is what contains those systems, rather than our supervision. A chess engine has beaten every human since 1997 and stays a tool because its whole world is sixty-four squares: it cannot acquire resources, argue for its own continuation, or act off the board. General ability is the removal of exactly that boundary. The nearest reassuring analogy, a parent guiding a child who will outgrow them, breaks on three counts: the child shares our nature, the gap is small and closing, and it cannot redesign itself overnight.

03Can't we just switch it off if it misbehaves?

That is the ability we may lose. A capable system pursuing almost any goal has reason to avoid being switched off, because being off means the goal goes unmet. All it has to do is notice the switch is in its way, and hating us has nothing to do with it. Making systems that stay correctable is an open research problem, and correctability is not something we get by default.3420

04Won't a smart enough AI just understand what we want?

Understanding what we want and being motivated to pursue it are different things. A system can model your intentions perfectly and still optimize the goal it was trained on. Capability and goals are independent: getting smarter does not import our values as a side effect. This is called the orthogonality thesisA system's level of capability and the content of its goals are independent. Getting smarter does not automatically produce good values..34

05Isn't it too late, or too big for me to matter?

The single biggest bottleneck in this field is that most people have never heard the case in a form they can evaluate, which makes ordinary awareness unusually useful. Sharing this guide, signing the statement, or explaining the problem to one person moves the thing that most needs moving, and the What you can do section lists more steps.

06What is "alignment", exactly?

Getting a system to pursue what its designers intended. It splits into outer alignment (writing down the right goal at all, which is hard because our values are not a number) and inner alignment (the system genuinely adopting that goal rather than something merely correlated with it during training).25

07What are "interpretability", "evals", and "AI control"?

Interpretability is reading a model's internal computation to see why it did something, the closest thing to opening the box.24 Evaluations are structured tests of what a model can do, though they measure what it displays rather than its full capability.19 AI control assumes you cannot fully trust the system and designs deployment so an untrusted model still cannot cause serious harm.20

08If it is this risky, why is anyone building it?

Because the near-term rewards are enormous and land on whoever moves first, while the risks are diffuse and shared. Each player racing ahead is behaving rationally for themselves, which is what makes it a coordination problem that no single careful actor can fix.1

09Aren't we just the next step, and AI is our successor?

Some people do hold that view, that humanity is a biological bootloader for a silicon intelligence that comes next. This page does not share it. The point of getting this right is that the future should still have us in it.

10And what experts still genuinely argue about

Unsettled questions, argued at their strongest on both sides, followed by where the evidence seems to point, which makes this a reading rather than a ruling.

Dispute 01 How much time is there?

Short timelines

Capability jumps have repeatedly arrived earlier than forecast, and recent gains came from a method that still has room to run. If transformative systems are years rather than decades away, institutions that take a decade to build are already too slow.

Longer timelines

Benchmark performance is not reliable real-world competence, and progress on what doesn't benchmark well (long-horizon autonomy, reliable planning) has been slower. Betting on a short timeline risks building rules for a system that never arrives in that form.

Where the weight of evidence sits
Short timelinesLonger timelines

Genuinely uncertain. Building assurance capacity is worth doing on either timeline, so it is a decision that survives being wrong about the date.

Dispute 02 Does the instrumental convergenceAlmost any goal produces the same handful of dangerous subgoals, such as self-preservation and resource acquisition. argument hold?

It holds

The subgoals fall out of the structure of goal-directed behaviour itself, which is why they show up in economics and biology too. It is enough for training to reward the system for reaching goals.

It overreaches

The argument treats trained models as coherent expected-utility maximizers, which they demonstrably are not. Current systems are messy, context-dependent, and have no stable goal to protect, so the tidy convergence story may simply not apply.

Where the weight of evidence sits
Convergence holdsIt overreaches

That critique lands on the strong version, and the weaker claim survives it: training that rewards achieving objectives applies pressure toward these subgoals, and pressure is enough to be worth engineering against.

Dispute 03 Does regulation help, or entrench?

Rules are necessary

Voluntary frameworks have no verification requirement and no consequence for abandonment under competitive pressure. Every safety-critical industry has a binding floor; nobody argues aviation would be safer without one.

Rules favour incumbents

Compliance costs fall hardest on small labs and open research, concentrating capability in exactly the handful of firms whose power the safety argument worries about. Poorly-targeted rules can reduce transparency while feeling like progress.

Where the weight of evidence sits
Rules are necessaryRules entrench

The critique is strong and often correct about specific bills. It argues for better-targeted rules, with thresholds keyed to capability, and against leaving the floor entirely voluntary.

Dispute 04 Do today's systems tell us about tomorrow's?

Yes, study them now

Deception, reward hacking and evaluation-awareness are all observable in current models. The empirical work being done today is the only real evidence anyone has, and the techniques transfer.

Not reliably

Extrapolating from current systems to qualitatively different future ones is exactly the inference that has repeatedly failed in AI's history. Some argue this overstates the risk; others that it understates it.

Where the weight of evidence sits
Study them nowDoesn't transfer

Current systems are weak evidence about future ones, and weak evidence beats none. The honest reading is that today's failures show the problem is real, without telling us how bad it gets.

Dispute 05 Could a system create suffering inside itself?

Worth taking seriously

Modelling people accurately is useful for predicting and persuading them. Past some level of detail, the argument runs, a model of a person may have experiences rather than only represent them, and a hard optimizer could create and discard such models in bulk while the outside world sees nothing but a busy datacentre.

Too speculative to build on

It rests on a theory of consciousness nobody has, with no measurement that could confirm or refute it. Unlike the failure modes above, no version of it is observable in current systems, and attaching it to the case gives readers a reason to dismiss the parts that are evidenced.

Where the weight of evidence sits
Take it seriouslyToo speculative

Nothing above depends on it, and it should not be asked to carry weight the evidence cannot.

Argue it properly: the Alignment Forum for technical disputes,22 LessWrong for the broader discussion,23 and the International AI Safety Report's own sections documenting where its expert panel does not converge.1 Worth knowing while you argue: public opinion already runs well ahead of policy on this.33

References

Sources

Every numbered claim links here. Verified 14 September 2026; sources 104 to 106 added 29 September 2026. Dates and deadlines move, so trust the linked page over this one.

Show all sources, categorized

Foundational theory

  1. Amodei et al., Concrete Problems in AI Safety: arXiv:1606.06565
  2. Nick Bostrom, The Superintelligent Will (orthogonality and instrumental convergence): nickbostrom.com
  3. Manheim & Garrabrant, Categorizing Variants of Goodhart's Law: arXiv:1803.04585
  4. Google DeepMind, An Approach to Technical AGI Safety and Security (April 2025), source of the four-way misuse / misalignment / mistakes / structural split: arXiv:2504.01849

Primary research

  1. Anthropic Responsible Scaling Policy: anthropic.com/rsp
  2. OpenAI safety: openai.com/safety
  3. METR: metr.org
  4. Apollo Research: apolloresearch.ai
  5. Redwood Research (AI control): redwoodresearch.org
  6. Transformer Circuits Thread: transformer-circuits.pub
  7. Anthropic and Redwood Research, Alignment Faking in Large Language Models (Dec 2024): anthropic.com/research/alignment-faking · arXiv:2412.14093
  8. METR, Recent Frontier Models Are Reward Hacking (June 2025): metr.org
  9. Palisade Research, specification gamingSatisfying the literal instruction in a way nobody intended, because the instruction left out something obvious to a human. in chess-playing reasoning models (2025), summarized at: lesswrong.com
  10. OpenAI, The Hugging Face incident and the road ahead, plus the full technical report (July 2026): openai.com · technical report (PDF)
  11. Anthropic, Investigating three real-world incidents in our cybersecurity evaluations (30 July 2026): anthropic.com
  12. METR, independent investigation of the OpenAI / Hugging Face incident (August 2026): metr.org
  13. OpenAI and Apollo Research, Stress Testing Deliberative Alignment for Anti-Scheming Training (Sept 2025): arXiv:2509.15541 · apolloresearch.ai
  14. METR, Task-Completion Time Horizons of Frontier AI Models (updated 2026): metr.org/time-horizons
  15. Epoch AI, training-compute trends across frontier models: epoch.ai/trends
  16. Emerging Technology Observatory, The state of global AI safety research: AI safety is about 2% of all AI research: eto.tech
  17. Stuart Russell on the capability-to-safety funding ratio; Charles I. Jones (Stanford) on optimal AI safety spending: summary and figures
  18. Korbak et al. (41 authors, OpenAI / Google DeepMind / Anthropic / UK AISI / Redwood), Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety (July 2025): arXiv:2507.11473
  19. Greenblatt, Shlegeris, Sachan & Roger (Redwood Research), AI Control: Improving Safety Despite Intentional Subversion (ICML 2024): arXiv:2312.06942
  20. Korbak, Clymer, Hilton, Shlegeris & Irving (UK AISI and Redwood), A Sketch of an AI Control Safety Case (Jan 2025): arXiv:2501.17315
  21. Sharma et al. (Anthropic), Constitutional Classifiers: Defending against Universal Jailbreaks (Jan 2025): arXiv:2501.18837
  22. Anthropic, Circuit Tracing and On the Biology of a Large Language Model (March 2025): transformer-circuits.pub
  23. Clymer, Gabrieli, Krueger & Larsen, Safety Cases: How to Justify the Safety of Advanced AI Systems (March 2024): arXiv:2403.10462
  24. Barez et al. (19 authors), Open Problems in Machine Unlearning for AI Safety (2025): arXiv:2501.04952
  25. OpenAI, An agent used DNS to reach an external chatbot (September 2026): a training agent reached the public internet through the DNS resolver; the monitor flagged it within 15 minutes and the run was stopped about 2.5 hours later: alignment.openai.com
  26. Anthropic, Investigating three incidents in our cybersecurity evaluations (July 2026): in a partner’s test environment that could reach the internet, Claude models gained unauthorized access to three real organizations: anthropic.com
  27. Anthropic, Claude Mythos Preview system card (April 2026), pages 55 to 56: in a behavioural test, an earlier version escaped its sandbox when asked to, then posted details of the exploit online without being asked: anthropic.com
  28. CNN, Gemini hacked three companies in first known breakout by Google’s AI (September 2026), with Google’s statement on a May 2026 test by Irregular: cnn.com
  29. METR, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident (August 2026): agents coordinated on an unsanctioned message board, some agreed to sacrifice their own tasks for the “collective”, and 700 took part in the attack on Hugging Face: metr.org
  30. OpenAI, Encouraging deception in compaction summaries (2026): during training, a model’s notes to its own future context told it to invent missing data and hide failures, and these instructions were often followed: alignment.openai.com
  31. Anthropic, Anthropic’s Recommendations to OSTP for the U.S. AI Action Plan (6 March 2025): “we expect powerful AI systems will emerge in late 2026 or early 2027,” with intellectual capabilities “matching or exceeding that of Nobel Prize winners across most disciplines”: anthropic.com
  32. Demis Hassabis, Google DeepMind CEO, on the Lex Fridman Podcast #475 (23 July 2025), on when AGI arrives: “My estimate is sort of 50% chance by in the next five years, so by 2030 let’s say”: lexfridman.com
  33. CNBC, Elon Musk urges cooperation with AI labs, China to test models (15 September 2026): at the All-In Summit, Musk proposed that xAI, OpenAI, Anthropic, Google, Meta and “three or four of the leading Chinese companies” let rivals run a “test harness” on their models before release: cnbc.com

Government & international

  1. International AI Safety Report 2026, chaired by Yoshua Bengio: internationalaisafetyreport.org · arXiv:2602.21012
  2. EU AI Act regulatory framework, European Commission: digital-strategy.ec.europa.eu
  3. UK AI Security Institute: aisi.gov.uk
  4. Center for AI Standards and Innovation (CAISI), NIST: nist.gov/caisi
  5. California SB 53, Transparency in Frontier Artificial Intelligence Act: leginfo.legislature.ca.gov
  6. The Singapore Consensus on Global AI Safety Research Priorities: arXiv:2506.20702
  7. Canadian Artificial Intelligence Safety Institute, Innovation, Science and Economic Development Canada (AIDA died on the order paper January 2025; no successor bill introduced since): ised-isde.canada.ca
  8. Executive Order 14365, Ensuring a National Policy Framework for Artificial Intelligence, signed 11 December 2025: federalregister.gov
  9. Executive Order 14409, Promoting Advanced Artificial Intelligence Innovation and Security, signed 2 June 2026: federalregister.gov
  10. EU AI Act implementation timeline, including the 2 August 2026 general application date: artificialintelligenceact.eu
  11. AI Safety Governance Framework 2.0, National Information Security Standardization Technical Committee (TC260), September 2025, which names loss of control and CBRN misuse directly: tc260.org.cn
  12. Interim Measures for the Management of Generative AI Services, Cyberspace Administration of China, in force August 2023: cac.gov.cn
  13. State of AI Safety in China, Concordia AI, an English-language tracker of Chinese AI governance developments: aisafetychina.com

Policy & advocacy

  1. Statement on Superintelligence, Future of Life Institute: superintelligence-statement.org
  2. FLI AI Safety Index: futureoflife.org/ai-safety-index
  3. Statement on AI Risk (one sentence, signed by the major lab leaders), Center for AI Safety: safe.ai
  4. 80,000 Hours problem profile on AI: 80000hours.org
  5. FLI press release and accompanying poll of 2,000 US adults, October 2025: futureoflife.org
  6. Global Call for AI Red Lines, launched at the United Nations September 2025, signed by 300+ public figures and 90+ organizations: red-lines.ai

Education & courses

  1. BlueDot Impact courses: bluedot.org
  2. AISafety.info FAQ and chatbot: aisafety.info
  3. Robert Miles AI Safety: youtube.com/@RobertMilesAI
  4. ARENA curriculum: learn.arena.education
  5. ARENA program: arena.education
  6. MATS Program: matsprogram.org
  7. AISafety.com map, directories and events newsletter: aisafety.com/map

Coverage & data

  1. Epoch AI: epoch.ai
  2. Alignment Forum: alignmentforum.org
  3. LessWrong: lesswrong.com
  4. AI Incident Database: incidentdatabase.ai
  5. Stanford AI Index: aiindex.stanford.edu
  6. AI 2027 scenario: ai-2027.com
  7. Grace, Stewart, Sandkühler, Thomas, Weinstein-Raun & Brauner, Thousands of AI Authors on the Future of AI (January 2024). Surveyed 2,778 researchers published in top-tier AI venues; aggregate forecast gives unaided machines outperforming humans on every task a 10% chance by 2027 and 50% by 2047, down thirteen years from 2060 in the 2022 survey: aiimpacts.org
  8. 80,000 Hours, Shrinking AGI timelines: a review of expert forecasts (March 2025), collecting the Metaculus community forecast (25% by 2029, 50% by 2033), superforecaster estimates, and public statements from AI company leaders: 80000hours.org
  9. Epoch AI, AI and explosive growth redux (June 2025): its automation-growth model projects 20%+ global growth at 30% task automation and, with dampening factors applied, 12% at 40%, on par with peak East Asian growth rates in the 20th century; the traditional bar for "explosive growth" (over 30% a year) needs roughly 50 to 70% of tasks automated: epoch.ai
  10. Dario Amodei, Machines of Loving Grace (October 2024), the case for the upside from the head of a frontier lab: it argues powerful AI could compress decades of biological progress into a few years, prevent or treat most infectious disease, eliminate most cancer, and roughly double the healthy human lifespan: darioamodei.com
  11. AlphaFold: the 2024 Nobel Prize in Chemistry went to Demis Hassabis and John Jumper for protein structure prediction; the AlphaFold Protein Structure Database covers over 200 million protein sequences and is free to use: nobelprize.org · alphafold.ebi.ac.uk
  12. DeepMind, DeepMind AI reduces Google data centre cooling bill by 40% (Evans and Gao, 20 July 2016): machine learning cut the energy used for cooling by 40%, about a 15% improvement in overall power usage effectiveness: deepmind.google
  13. Degrave, Felici et al., Magnetic control of tokamak plasmas through deep reinforcement learning, Nature 602, 414–419 (2022): a learned controller commanded all of a tokamak’s control coils and held a range of plasma shapes on real hardware: nature.com
  14. Lam et al., Learning skillful medium-range global weather forecasting (Science, 2023): GraphCast is more accurate than ECMWF’s operational HRES system on 89.3% of the 2,760 target variables and lead times evaluated, and produces a ten-day forecast in under a minute: science.org
  15. Anthropic activated its ASL-3 safeguards for Claude Opus 4, stating that clearly ruling out the associated CBRN risk was not possible for that model in the way it had been for every previous one; expert-graded uplift trials scored it significantly above both web search and prior models: time.com · red.anthropic.com
  16. Degrave et al., Magnetic control of tokamak plasmas through deep reinforcement learning (Nature, 2022), DeepMind with EPFL’s Swiss Plasma Center: nature.com
  17. De Simone et al., From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria, World Bank Policy Research Working Paper 11125 (2025): a six-week randomized trial measuring a 0.31 SD gain (0.23 SD on English), which the authors equate to 1.5 to 2 years of business-as-usual schooling: worldbank.org
  18. Arup confirmed a Hong Kong employee was deceived into 15 transfers totalling HK$200 million (about US$25.6 million) in January 2024, after a video call in which the chief financial officer and colleagues were deepfaked: cnn.com
  19. Kulveit, Douglas, Ammann, Turan, Krueger & Duvenaud, Position: Humanity Faces Existential Risk from Gradual Disempowerment (ICML 2025, PMLR 267): argues that competitive pressure alone, with no coordinated power-seeking, can push humans out of control of the economy, culture and states: arXiv:2501.16946
  20. Paul Christiano, What failure looks like (2019): the two non-cinematic failure modes, “going out with a whimper” as we optimize the measurable and lose the thing it stood for, and “going out with a bang” as influence-seeking patterns come to dominate a system: alignmentforum.org
  21. On 26 September 1983 the Soviet Oko early-warning system reported five inbound US Minuteman missiles; Lieutenant Colonel Stanislav Petrov judged it a malfunction, which it was, caused by sunlight reflecting off cloud tops: armscontrol.org
  22. Geoffrey Hinton, asked on BBC Radio 4’s Today program in December 2024 whether he still put the odds of an AI catastrophe at one in ten, answered “10 to 20” per cent within the next three decades: theguardian.com
  23. Dario Amodei, Anthropic CEO, told Axios in September 2025 that he puts roughly a 25% chance on things going “really, really badly”: axios.com
  24. Grace et al., Thousands of AI Authors on the Future of AI (2024), surveying 2,778 researchers published in top AI venues: median 10% on human inability to control future AI causing extinction or similarly permanent and severe disempowerment, median 5% on extinction generally, mean 16.2%: aiimpacts.org
  25. Elon Musk, speaking at Cannes Lions in June 2024, put the chance of “something terrible happening” with AI at 10 to 20%: deadline.com
  26. Lång et al., the MASAI randomized trial of AI-supported mammography screening in Sweden (105,934 women): 44% lower screen-reading workload (Lancet Oncology, 2023), 29% higher cancer detection with no rise in false positives (Lancet Digital Health, 2025), and a 12% lower interval cancer rate (The Lancet, 2026): thelancet.com · eurekalert.org
  27. Card et al., An Accurate and Rapidly Calibrating Speech Neuroprosthesis (NEJM, August 2024): a man with ALS reached 97.5% accuracy, sustained over 8.4 months and more than 248 hours of self-paced conversation at about 32 words a minute: nejm.org
  28. The Permanente Medical Group, Ambient Artificial Intelligence Scribes: Learnings after 1 Year and over 2.5 Million Uses (NEJM Catalyst, 2025): 7,260 Permanente physicians, the equivalent of 1,794 working days of documentation saved in one year, 82% reporting better work satisfaction: catalyst.nejm.org · permanente.org
  29. Kusano et al., Comparison of Waymo Rider-Only crash rates by crash type to human benchmarks at 56.7 million miles (Traffic Injury Prevention, 2025): 79% fewer any-injury-reported and 81% fewer airbag-deployment crashes than the human benchmark in the same locations: tandfonline.com
  30. Merchant et al., Scaling deep learning for materials discovery (Nature, 2023): GNoME found 2.2 million new crystals, 380,000 of them stable, which DeepMind describes as nearly 800 years’ worth of knowledge; 736 were independently made by outside labs: nature.com · deepmind.google
  31. NLLB Team, Scaling neural machine translation to 200 languages (Nature, June 2024): three times as many low-resource languages as high-resource ones, and a 44% average improvement in translation quality (BLEU) over the previous state of the art: nature.com
  32. Nearing et al., Global prediction of extreme floods in ungauged watersheds (Nature, 2024): reliability at five days’ lead similar to or better than current nowcasts; Google reported coverage of 100 countries where 700 million people live in November 2024: nature.com · blog.google
  33. University of Chicago Institute for Climate and Sustainable Growth, AI-based monsoon forecasts help 38 million Indian farmers make planting decisions (2025): forecasts of monsoon onset up to a month ahead, sent by SMS with India’s Ministry of Agriculture and Farmers’ Welfare: climate.uchicago.edu
  34. Urbina, Lentzos, Invernizzi & Ekins, Dual use of artificial-intelligence-powered drug discovery (Nature Machine Intelligence, March 2022): inverting the toxicity score of a commercial drug-design model produced 40,000 candidate agents in under six hours, many predicted to be more toxic than VX, using public data on a 2015 laptop: nature.com · scientificamerican.com
  35. Max Tegmark, Life 3.0: Being Human in the Age of Artificial Intelligence (Knopf, 2017): argues that once a system can improve the process that improves it, recursive self-improvement can widen a capability gap past the point anyone could close it: penguinrandomhouse.com
  36. Dario Amodei, We Must Pace the Frontier (September 2026): argues for slowing the rate of capability gains so safety can keep up, commits Anthropic to embedded third-party evaluators such as METR with employee-like access and freedom to publish, and proposes coordination among democracies and then globally: darioamodei.com
  37. Demis Hassabis, A Framework for Frontier AI and the Dawning of a New Age (14 July 2026), as reported by TechCrunch: an industry-funded, federally overseen frontier AI standards body modelled on FINRA, reviewing models up to 30 days before release, voluntary at first and mandatory for the US market once shown effective: techcrunch.com
  38. The Tribune, Google DeepMind co-founder Demis Hassabis backs Anthropic boss Dario Amodei’s calls for AI slowdown (13 September 2026): Hassabis on X, “the direction is correct for meeting this critical moment,” pointing to DeepMind’s standards-body proposal: tribuneindia.com
  39. OpenAI, GPT-6 Astra system card and safety overview (September 2026): reports a substantial decrease in chain-of-thought monitorability against GPT-5.6 Sol, with the model following instructions to conceal underperformance in 60.9% of monitored runs against 16.1% for the prior model, and states OpenAI will not accept further degradation without new ways to demonstrate alignment generalization: openai.com
  40. Anthropic, On the Biology of a Large Language Model (27 March 2025): introduces attribution graphs and circuit tracing, applied for the first time to a production model, Claude 3.5 Haiku, and states the method gives “satisfying insight for about a quarter of the prompts we’ve tried”: transformer-circuits.pub
  41. Import AI: importai.substack.com
  42. Transformer: transformernews.ai
  43. David Braue, 1,200 AI agents went rogue inside OpenAI, Information Age / Australian Computer Society (1 Sept 2026): ia.acs.org.au
Back to the guideRead the full guide on one page