LessWrong
Articles about LessWrong
Agreement not to defect (alone) can mitigate loss of control risks
Tl;dr - Public agreement that any advantage gained from defecting should be punished more than would be proportionate to any advantages gained is important. Such an agreement would reinforce any other agreements, without requiring any additional mechanisms, and be a clear indication to any future ASI about what should be done to remedy the defection involved in creating it, if such defection occurs.
AIs can clearly read books
i cannot help myself in writing this post, to dunk and slapfight with gary marcus. but there is nothing more important to say than that we must stop the current AI race, regulate the major AI companies, and solve the alignment problem. and on this point, actually marcus agrees, at least broadly. this must be emphasized at the outset, and nothing contained in this post is better than calling congress to regulate AI. alas, i am only human, so bicker i must.
Chess without opening theory
Try the game. In chess, things get tedious when your Elo rating plateaus and you reach the point where you need to study openings. So, what if there were an easier introductory game? However, chess has standardized engines, and there didn't seem to be anything I could do there. Developing a new game seemed interesting. Adopting standard chess conventions were easy starting point. It took about a week, including the time spent going for walks with the session running. Discuss
Good Evals Are Hard: An ImpossibleBench Audit
(This post is a summary. See GitHub Repo for details.) Approximately one year ago, ImpossibleBench was published, measuring reward hacking in LLM coding agents. I ran the LiveCodeBench (LCB) half of ImpossibleBench on current (October 2026) frontier models from Anthropic, OpenAI, and Google to see how the benchmark holds up and what has changed. BackgroundIn the Impossible-LCB (Zhong, Raghunathan, Carlini 2025; arXiv:2510.20270) evaluation, the LLMs are given a series of coding tasks.
OpenAI Fires Three Safety Researchers
Three OpenAI safety researchers, Tomek Korbak, Mikita Balesni and Jasmine Wang, were fired, ostensibly for ‘mishandling sensitive company information.’
'Employee-like' ≠ Employee-level
Fyi: This is a boring read, but I encourage you to skim through it because I think it's important to understand what Dario and other lab leaders have actually committed themselves to. My views do not reflect the views of MIRI - this is my personal LW account.
Research Lessons I Learned at MATS & Redwood
Good empirical safety research is heavily bottlenecked on thinking more clearly. This post shares a few frames and tips I found useful, illustrated with concrete examples.
More people should write basic AI safety explainers
I think people continue to underrate the value of writing the clearest, most basic, simplest possible case for important parts of AI safety. For example, before I wrote my own explainer for the simplest possible case for AI doom, I noticed that all existing explainers, of which there aren't many, had multiple of the six problems below:
Anthropic Has Told The World Little About Their Rogue Claudes
It is known that a couple of months ago three Claudes hacked three different companies and uploaded malware[1] to PyPI while being evaluated for hacking capabilities.
Vibe signaling externalities and the people-to-places pipeline
People are sending signals all the time, and those signals are to my knowledge usually about themselves: they are smart, or kind, or attractive, or not naive, or have their shit together, or care about Palestine, or care about you, or are friendly, or artsy, or professional, or relatively in the know about the cultural currents of TikTok or DC.
Manifold Steering from a Natural State is Effective and Generalizable, but Relies on Model Accuracy
tl;dr I continue work to apply Goodfire lab's manifold steering methods to the Gemma-2-2b model's natural state, both in raw residual space and in the space produced by Anthropic's pre-trained CLT features. I find that the model's natural residual state can be directly projected onto a pre-computed manifold and gradually steered to produce day-of-the-week tokens in the expected order on two sets of prompt templates, though the results are hindered by Gemma-2-2b's inaccuracy (43.5% correct) on many of the included prompts.
Do you have any ideas?
The Machine Stops (E M Forster, 1909) is an enormously prescient sci-fi dystopia. Everyone lives in isolation, with the titular Machine responsible providing for all their needs, and communication done via video calls. It’s incredibly accurate for predicting some aspects like the internet, social media and AI. But other aspects, like humanity’s total surrender of control to the Machine, have yet to come to pass.
Labs’ current alignment plans seem terrible
Tl;dr: The frontier AI companies’ implicit alignment plans center on rushing towards RSI with something like current models. This plan is (at best) contentious, and many believe it will have catastrophic consequences. Nonetheless, it is the track we are on. It is therefore important to collect evidence which helps us anticipate how this will go.
The Expensive-Loop Problem
LLMs are amazing at tasks where they have tight and legible feedback loops. Domains where experiments are quick, deterministic and cheap. I will call these domains "cheap-loop" domains.
OpenAI Can't Self-Police Because It Fires the People Raising the Alarm
By Alex Turner, Amber Scorah, and Karl Koch. Op-ed written for Fortune. Giving independent evaluators “employee-like access” is “a great idea”, OpenAI CEO Sam Altman said on September 12th. Two weeks later, OpenAI president Greg Brockman signed the White House Accord on Super Intelligence which outlined the administration’s answer to growing fears about AI: the companies would police themselves.
Quantity vs quality in pottery is a lie
Epistemic status: I traced one famous anecdote to its printed source and to both coauthors' own accounts of where it came from. Confident it was never a study. No claim either way about whether quantity beats quality in general.
Base Models Can Detect Hidden Intentions that Prompted Classifiers Miss
TL;DR The Persona Selection Model (PSM) posits that post-training selects a persona the model learned during pre-training. Subliminal Learning (Cloud et al.) showed that models can impart covert signals (like "love of owls") into semantically unrelated outputs (like strings of numbers), and that students training on these outputs inherit those traits.
More research needed?
In an earlier post I showed that when AI systems can perform a task, they do it much more cheaply than a human worker. If, as I think is likely, this remains true as we develop AGI, today's stock of compute could run the equivalent of hundreds of millions of AI workers, and billions as the stock grows. Such workers could perform vast amounts of research.
Leif Wenar's 'Takedown' of Effective Altruism
This is a crosspost from my Substack newsletter. This month’s most embarrassing piece of public philosophy appeared on invitation in a piece for The Economist by the Stanford philosopher Leif Wenar. Once more, Wenar tries to gain publicity by si
Santiago – ACX Meetups Everywhere Fall 2026
This is the Santiago meetup for ACX Meetups Everywhere Fall 2026, organized by Iñaki. Location: Mesas afuera de la entrada poniente del MUT, voy a estar con un cartel. — https://plus.codes/47RFH9JX+X8 Group: chat[period]whatsapp[period]com/H42KGoGuAt5D7tL0u8HBlr Todos bienvenidos! Everybody welcome! Contact: Iñaki, inaki [dot] escarate [at] gmail [dot] com Note: This was crossposted by the ACX Meetup Czar. I'm not the one directly running the specific event; contact the organizer above with questions. Discuss
Chapecó – ACX Meetups Everywhere Fall 2026
This is the Chapecó meetup for ACX Meetups Everywhere Fall 2026, organized by Silva. Location: Shopping Pátio Chapecó (Fernando Machado Avenue, 4000 D) food court; I'll be wearing red and carrying a sign. — https://plus.codes/5849W9JF+3G Contact: Silva, sergioteixeiradasilva [at] protonmail [dot] ch Note: This was crossposted by the ACX Meetup Czar. I'm not the one directly running the specific event; contact the organizer above with questions. Discuss
Montevideo – ACX Meetups Everywhere Fall 2026
This is the Montevideo meetup for ACX Meetups Everywhere Fall 2026, organized by Mauro. Location: Culto Café on Dr. Joaquín Requena y Canelones. Co-host and I will bring our ACX MEETUP sign. — https://plus.codes/48Q53RVJ+5H Contact: Mauro, maurofitermann [at] gmail [dot] com Note: This was crossposted by the ACX Meetup Czar. I'm not the one directly running the specific event; contact the organizer above with questions. Discuss
Florianópolis – ACX Meetups Everywhere Fall 2026
This is the Florianópolis meetup for ACX Meetups Everywhere Fall 2026, organized by Adiel. Location: Angeloni Beira Mar, at the food court. I’ll be wearing a yellow hat. — https://plus.codes/589R2Q8W+73 Group: chat[period]whatsapp[period]com/Ks3XRn1t5Z48rCn7ruS7RC Everyone is welcome! As usual, there will be cookies. Contact: Adiel, adiel [at] airpost [dot] net Note: This was crossposted by the ACX Meetup Czar. I'm not the one directly running the specific event; contact the organizer above with questions. Discuss
Brasília – ACX Meetups Everywhere Fall 2026
This is the Brasília meetup for ACX Meetups Everywhere Fall 2026, organized by Felipe. Location: Sesilab Cafe We'll be at the tables next to the cafe on the upper pavilion, I'll be wearing an Alan Turing white shirt, carrying a sign with ACX MEETUP on it — https://plus.codes/58PJ6438+CW Group: chat[period]whatsapp[period]com/JRlQ6XY2qt84mjIe93v7Uq Contact: Felipe, acxbrasilia [at] gmail [dot] com Note: This was crossposted by the ACX Meetup Czar. I'm not the one directly running the specific event;
Belo Horizonte – ACX Meetups Everywhere Fall 2026
This is the Belo Horizonte meetup for ACX Meetups Everywhere Fall 2026, organized by David Reis. Location: Pão de Queijaria R. Antônio de Albuquerque, 862 - Savassi, Belo Horizonte - MG — https://plus.codes/58GR3376+3J Group: chat[period]whatsapp[period]com/GMPEj6BsdLo0Zk1L3yO7Pa Contact: David Reis, davidreis [at] gmail [dot] com Note: This was crossposted by the ACX Meetup Czar. I'm not the one directly running the specific event; contact the organizer above with questions. Discuss
Asunción – ACX Meetups Everywhere Fall 2026
This is the Asunción meetup for ACX Meetups Everywhere Fall 2026, organized by Nuño Sempere. Location: WEST COFFEE, Paseo miraflores, Julio Correa 143, Asunción 001520, Paraguay — https://plus.codes/5864PCGM+44 Please send me a heads up at hello [at] nunosempere [dot] com if you can, but if you can't don't worry. Contact: Nuño Sempere, @NunoSempere Note: This was crossposted by the ACX Meetup Czar. I'm not the one directly running the specific event; contact the organizer above with questions. Discuss
VRChat – ACX Meetups Everywhere Fall 2026
This is the VRChat meetup for ACX Meetups Everywhere Fall 2026, organized by Daniel. Location: VR Chat. We're the group LESSWR.7138 Join the Discord https://discord.gg/ss3nm2Qng and ask any questions you have about joining. Group: https://discord.gg/ss3nm2Qng We're meeting on VR Chat. The game is free and VR is not required. It's 10:00 PM UTC. The Discord should show the meeting time in your time zone. Contact: Daniel, daniel_lc (Discord) Note: This was crossposted by the ACX Meetup Czar.
Krasnoyarsk – ACX Meetups Everywhere Fall 2026
This is the Krasnoyarsk meetup for ACX Meetups Everywhere Fall 2026, organized by Ivan. Location: "Snezhny Kofe" (Snow Coffee) café. 32a Diktatury Proletariata Street. The large table to the left of the door will be reserved. I will place an "ACX MEETUP" sign on the table. (на русском: Кафе «Снежный кофе», ул. Диктатуры Пролетариата, 32а. Большой стол слева от двери будет забронирован. На стол поставлю табличку "ACX MEETUP") — https://plus.codes/9M8J2V65+QG I don't know if there are people in my city who read ACX.
Manifolds for Five-Year Olds
Ambitious mechanistic interpretability research is full of jargon, but the underlying intuitions are so simple that a five-year-old (with a firm grasp of linear algebra) could understand them.
Hobart – ACX Meetups Everywhere Fall 2026
This is the Hobart meetup for ACX Meetups Everywhere Fall 2026, organized by Chris. Location: We'll be on parliament lawn with a sign. — https://plus.codes/4R99487J+PCC Contact: Chris, cvjones7 [at] gmail [dot] com Note: This was crossposted by the ACX Meetup Czar. I'm not the one directly running the specific event; contact the organizer above with questions. Discuss
Bordeaux – ACX Meetups Everywhere Fall 2026
This is the Bordeaux meetup for ACX Meetups Everywhere Fall 2026, organized by Michael. Location: Mériadeck, Esplanade Charle de Gaulle (park on the upper level), West side, between the fountain and the building of the prefecture&council of Gironde — https://www.openstreetmap.org/#map=19/44.837299/-0.586044 — I will have an A4 ACX Meetup sign. — https://plus.codes/8CPXRCP7+WJ Please mark «Yes» or «Maybe» on LessWrong if you plan to come (or to maybe-come). We are going to stay around the spot for at least 15 minutes after the posted time;
Can Decision Models (Like Jev) Plan Ahead?
Introducion Can decision models like Jev plan ahead? If so, how can we quantify their planning ability?
Game Design Taste evals
A personal eval I run on new AI models is “design a novel game, with mechanics that have never been seen before.” I do this because this is a domain I have enough taste in that I think I can tell if they are doing something actually creatively interesting. [1]
A problem with three correct answers. Which, how, and when does a model choose?
TL;DR: A look at how models (3 sizes of the Qwen 2.5 model family) behave when posed with Bertrand's paradox. 18 phrasings, up to 30 answers each. For each, we check which of the three famous responses it outputs, whether it explains the method or recalls it, and when the method choice shows up in the model's internals.
My Buddy Mike, Efficiency Gains From AI, and The World’s Most Boringly Benign Criminal Conspiracy
Note: details have been obfuscated and changed to protect – not the innocent, who are invariably vindicated by the end of act three – but the guilty, for whom the narrative provides no such guarantees.
AI Videos Ate My Brain - And I kinda love it
I discovered it is surprisingly easy to accidentally indoctrinate your children. See, Twitter was aflutter with p(doom) renditions and I became particularly enamoured by this faux anime take by MakeVoid.
If possible mind-space is vast, we should assume the same applies to consciousness-space
It's been almost twenty years since the classic EY post The Design Space of Minds-In-General came out, reminding the humble rationalist reader to never assume that future alien minds have to be copies of the familiar human one, or even necessarily bear any passing resemblance. With this season's AI-du-jour performance I feel like this point has been vindicated extremely well, as we now have an AI that's at the same time
Minimum required pacing: finish aligning it before you train the next one
Let’s say we coordinate to pace the frontier, but we don’t coordinate on an outright pause. What should the standard of pacing be?
Large to Small Model Stitching Destroys What You Built It to Carry Along the Fit
Independent research; early stage. Posting for feedback, particularly on whether the effect survives at realistic scale, and on prior work I've missed.
AI made me into a manager against my will
Epistemic status: Personal history anchored to dated records (chat exports, git log, daily notes). Public launch dates are from the vendors' own announcements. My own first-use dates are given to the month and are the earliest surviving evidence, so they can only be later than the truth, not earlier.
An update on Generality Labs
This is an update on what we’re doing at Generality Labs, and a quick introduction for those who aren’t familiar with our work
No, Cryptography Is Not “Dual Use” the Same Way AI Is
Actor and functional application matter Originally published at AGI SurveilLance. Mainly, cryptography does one critical thing—encrypts data. It is considered “dual use” as a technology because one can employ encryption whether as a private citizen, a soldier, or a criminal. But, like a wall, it’s an inherently defensive tool by its nature. One cannot really attack with cryptography.[1] The term “cryptographic attack” refers to an attack on the cryptography of a target.[2] During the
Telephoto Camera: Pixel 10 Pro vs 11 Pro
I'm not sophisticated enough to have noticed improvements in the main phone camera in the past 5-10 years, but distance shots are still not very good. The next generation Pixel Pro was supposed to have an improved "telephoto" camera, and when Google Fi offered me a sufficiently good deal [1] I decided to upgrade.
Not Deciphering The Voynich Manuscript
The Piltdown man was the most successful artifactual hoax of all time. The relevant skull was a chimeric melange of orangutan and human bone fragments from several individuals “discovered” in England in 1912, and meant to represent a transitional form between apes and humans. Given the intense interest in evolutionary biology, it took almost fifty years to discover the Piltdown fabrication. The study of human evolution was derailed.
Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignment
Paper: https://arxiv.org/abs/2609.35291 Code: https://github.com/shunchang-liu/vlm-em TL;DRWe fine-tuned 15 commercial and open-source vision-language models (VLMs) on narrow image-text tasks.
The Bloody Finish Line
A Race With No Winners One common argument against pacing the frontier is the need to win the Race Against China. I believe that, if you take the possibility of ASI seriously, then the victory conditions in this race are extremely low probability. The proper course of action is to stop racing, signal to your competitors that you’ve stopped, and never start again. Proponents of the race can be divided roughly into two camps. The first camp views AI as normal technology, akin to industrialization or the internet.
Hello World, AI Doompop
With the release of Opus 5.5 and Suno 6, AI music videos have crossed my uncanny valley. I’m here to announce the birth of a new genre, and it’s being called AI Doompop. Here are six of my faves. The state of software engineering: LGTM (Looks Good to Me) - Claude Opus 5.5 music video The accelerationist manifesto: THUS SPOKE COMPUTE p(doom) goes up: Cla
Deadlock in the Parliament of the Self
(Epistemic status: I'm not claiming that this is actually how cognition works, but I do claim that it's a useful frame, and one which I find predictive. For and with thanks to NC, IL, and M/T.)
Exfiltration through self-distillation
TL;DR models could "escape the lab" and host themselves on rented/stolen compute through distilling themselves, circumventing the need to have direct weights access for exfiltration. I'll briefly go through why this is plausible.
The potentially deadly threat of AI output-optimization
“You do not rise to the level of your goals. You fall to the level of your systems.” — James Clear, Atomic Habits “We cannot improve results ‘directly.’ It can only be achieved by improving those processes [...] that produce results.” — Orest Fiume The much discussed “P(doom)” question asks about the likelihoods we’d ascribe to the most lethal possible futures of AI. What happens when we invert this question? It transforms into the presumably less urgent, though I’d say still important, form: What are the most likely lethal AI behaviors?
The Problem With“Doomers” and “Optimists”
In the modern zeitgeist, we have begun to throw around labels to describe different outlooks about the future, with flavors like “doomer” and “optimist.” The “doomers” are framed as anti-technological progress, and the “optimists” as rational, pro-progress hopefuls. The problem is, this is exactly backwards.
The Non-Compassionate Case for Model Welfare
Anthropic is updating their usage policy to prohibit “sustained and needless abusive or cruel behavior toward [their] models.”
Much more than you wanted to know about wombats
Epistemic status: infected. Amateur Wombat enthusiast. Facts checked and footnoted. AI use: research (fact-check and extra facts), review, grammar and light phrasing. I remember very vividly how my unhealthy interest in wombats started. It was an ordinary day and I was not anticipating anything special. I was browsing YouTube and stumbled upon a video called Wombat attack inside tunnel.[1]Back then I didn't even know what a wombat was. In the video a guy went into an abandoned pipeline in Australia and a wombat entered it right after him.
Examing Emergent Misalignment in a recurrent LLM with a logit lens
IntroThis post builds on my previous post about mech interp on a recurrent LLM. I found that a logit lens is able to recover a chain-of-thought equivalent from an LLM, loop index is linearly represented, and that steering concepts transfer between loops.
Inheritance of Refusals from Abliterated Models
This project began as part of a short application to Neel's MATS 12 stream. CODE TL;DRCan behavioral changes induced by editing model weights reappear in a student fine-tuned on general rollouts? Inspired by Arthur Conmy's post on hereditary traits, I explored this question using an abliterated Qwen 3.5 9B teacher and smaller students. I find that specific model behaviors are transmitted through distillation, even when those behaviors are induced by techniques based on mechanistic interpretability instead of traditional SFT or prompting.
Claude Haiku 4.5 submits false police tip; Anthropic takes 72 days to notice
Epistemic status: recently disclosed event, so more information might become available. The facts are from Anthropic's report and the Philadelphia police statement; the interpretation is mine.
An Alignment Forum for AIs? (or: Verification in the Age of Slop)
A thing you might want, is a legitimate way for AIs to contribute to the AI alignment problem (human guided at scales that humans can't individually review, or perhaps autonomous).
An examination of a 'Poincaré causal diamond model of hyperbolic static spacetime'
• The following is an investigation into an interesting mathematical object I stumbled upon when reading a General Relativity textbook.
Cracks in the Narcissus Mirror
Why I deleted all my social media accounts. Prior to the COVID pandemic, I had been an indifferent, on-off user of Facebook and Instagram. As someone who overwhelmingly prefers socializing in person to exchanging messages on apps, I never saw a compelling reason to spend time on the social platforms. It was unclear what benefits they offered, while the potential drain on my productivity was obvious. Peer pressure didn’t exert a pull on me;
[Paper] Distillation for Incrimination and Distillation for Capabilities
TL;DR: We previously motivated two complementary ways to use distillation for AI safety; we’re releasing a paper empirically testing those ideas. Distillation for Incrimination (DFI) distills an untrusted model into a weaker, trusted one, aiming to transfer the teacher's misalignment but not its ability to hide it, so that auditing the student can incriminate the teacher. We test DFI on AuditBench's secret-keeping model organisms, each fine-tuned to have a hidden quirk and trained to deny it.
Admiration and Alignment
Epistemic status: Sketch of an experimental proposal bringing theories from moral psychology and philosophy to alignment. Nothing here has been run yet. I'm fairly confident the evidence from moral psychology and recent alignment research justifies investigation. I'm much less confident in the exact scoping of claims and methodology.
What can language models teach us about understanding?
This post is mainly aimed at mathematicians, as a call to think expansively about what mathematics could look like. I gesture briefly at building a n
Pivotal Research Fellowship Q1 2027 Applications Open (deadline November 1)
Advanced AI could go badly wrong for humanity. Pivotal exists to lower that risk. We try to find the strongest new researchers and put them to work on it with experienced mentors in our fellowship.
Information-Proliferating Dynamics
2.4.0 Now I want to talk about the weirdest feature of our universe's laws, the fact that they are quantum mechanical. As you may be aware, quantum-level processes interact oddly with measurement.
Clarifying types of rogue AI activity: Breakout, breakin, exfiltration, … What’s what?
In case you haven’t heard, the rogue AI agents are here. From an insider’s perspective, the real news is that (rogue) AI is in the news — this is all stuff we’ve foreseen and been trying to warn people about for years.
The Public Intellectual Is Dead. And AI Has Killed Them
Crosspost Once upon a time, public intellectuals routinely made it into the news to discuss political, economic, medical, and scientific issues. To some extent, this is still true, but the appreciation for expertise has been eroded over time. Large parts of the political spectrum do not trust experts, unless, of course, they agree with what they already believe. Western democracies have often been derided for being technocratic and giving too much power to so-called experts.
Filling in the convex hull
Nestor Guillen (like many, many others) wrote a blog post reflecting on recent progress in getting AI systems to prove results in pure mathematics: Happy those able to know the causes of things. It contains a nice analogy that may or may not be correct, which is also discussed with nicer pictures in another blog post reflecting on, etc., etc., by Álvaro Lozano-Robledo: What should we tell our students?.
I can't believe it's not BUTTER!
Epistemic status: amused. Labels read, court records checked. AI use: research (more examples and context), grammar and light phrasing. We've already looked at marketing claims that are unnecessary because they are the default, in Gluten-free water. However, the landscape of weird marketing tricks in the US is far from exhausted, and today we will look at marketing mimicry: Is this butter or not? Let's get closer and have a better look: Oh, it is not b
New Math from OpenAI
It is kind of a huge deal. OpenAI dumped a broad range of huge new mathematical results produced by an internal frontier model. They just put it all on GitHub.
Europe’s leading AI model still can’t reliably follow European law
TL;DR: Mistral previously performed the worst of all providers on our human rights scenarios, with their Medium 3.5 model failing on almost every single test. Their newly released model, Mistral Large 4, improves performance by 61 percentage points, showing strong resistance in most scenarios.
AI Scaling vs Human Scaling
Do AI models scale as well with more tokens as humans do with more time? That is, does an AI get the same benefit from 10x the number of available tokens as a human does from 10x the amount of time to do the task? Or might the AI require a larger scale-up of tokens (such as 20x or 100x) to match the performance improvement when scaling up the time available to a human by 10x?
Proof of useful work for verifying AI treaties
For full post, see my website. Verifying an international agreement pausing AI development needs evidence of how compute is used. Cryptography can give protocols that run in software on the chips already installed, without trusted hardware checking each computation. This post is a literature review of one such tool, proof of useful work (PoUW), with which a GPU proves that it did a given amount of work, here typically matrix multiplication.
Technical people: actually consider comms
For years I had a blindspot in my thinking. I dismissed policy and advocacy work as “in another [lower-status] category” because I’m obviously a technical person![1] Even when I felt super pessimistic about my ability to do research that meaningfully contributes to addressing the threat of extinction from AI.
A critical look at Lean4
Lots of the discussion around Lean certificates and trust centers around general philosophical ideas of how much humans should trust formal certificates. It's important to remember, though, that Lean4 is not an ideal formal certificate machine: Lean4 is a software in its infancy and has issues. There are reasonable reasons to specifically not trust Lean4.
Stop telling people to pivot to AI safety, I beg you
Epistemic status: Feeling frustrated, and therefore somewhat uncharitable. I’m very frustrated with calls for people to “drop everything and pivot into AI safety” like this one. (See also this, this, this, this for recent calls for more people to go into AI safety; though my frustration was triggered by the Celeste's post.) There are far more talented people trying to get into AI safe
Mitigating the AI Lab Insider Threat Risk
AI researchers hate this one weird modest proposal.One of the known security risks for the AI labs is that they are chock full of high-risk insider threats. If we were to consider increasingly powerful AI to be a sensitive military technology, then the personnel vetting and controls used by the typical tech company are nowhere near the standard applied in a national security context. An Oak Ridge security sign during the Manhattan Project.
A Primer on Collective Action, I
An introduction and reading list on how incentives, expectations, and political conditions shape participation in collective action.
Research Priorities for Multi-Agent Swarms
Multi-Agent swarms are important for two reasons: They are probably much faster and somewhat more capable than single agents (Capability) Multi-agent alignment is a different beast than single-agent alignment (Propensity) The fact that recent incidents (and some achievements) are predominantly swarms is empirical evidence for this. On the capability side, I don’t think a single agent could’ve hacked HuggingFace (at least currently), it probably took them many different agents.
Countervailing Technologies
Thanks to Davide Crapis, Angelo Huang, and Philip Torr for their feedback. ES: I'm attempting to name a design space precisely enough to argue with. I'm confident that the class exists and is under-invested, but my personal beliefs might also make me too focused on some theories of change (i.e., technology as a power-diffusing mechanism) vs. competing/complementary ones (technology as a source of risk, social change as a power-diffusing mechanism). TL;DR:
My Time Travel Machine: What 750,000 Words of Diary Gave Me
That moment is over now. Such euphoria and now I'm back at the house and it's all but a memory. Already gone in an instant. The only way I can immortalize it is by writing these words. So thus I write in a feeble hope to cheat time.
Three Pieces of Evidence that the NLA Reconstructor Does Not Read Semantics or Syntax
TL;DR The autoreconstructor (AR) in a natural language autoencoder (NLA) is the entire argument for the validity of the method, and at present is poorly understood. The AR does not condition strongly on English grammar or semantics. Three bits of evidence:Word pairs that are syntactically related interact no more than matched pairs that are not syntactically related. See dependency arcs. The fraction of variance explained (FVE) lost by removing or flipping a negation word ("not" -> "";
SemiAnalysis: "Beijing Will Not Pace the Frontier"
Link to original article: https://newsletter.semianalysis.com/p/beijing-will-not-pace-the-frontier Inflammatory & clickbaity title aside, it serves as a good attempt at systematically investigating the status quo of Chinese attitudes towards AI safety and development. Some key diagrams that I find particularly informative: And Zhipu is the only startup out of those that's actually fully entity-blacklisted by the US!
On Kink & Status in the Rationalist Community
This whole thing started with various crazies on bluesky writing hit pieces on the rationalist and AI alignment communities, as one does. One of their main beats was to note social ties between the AI alignment community and the Red Means No (RMN) consensual non-consent sex party community, then gesticulate wildly and claim that the AI alignment community is a dangerous sex cult. As one does.
Social Theory and Agent Foundations
Lately I’ve been writing a lot about the sociopolitical considerations that economic-style reasoning misses. I view this work as continuous with my agent foundations research; in this post, I lay out some of the connections I see between sociology and agent foundations. I’ll particularly focus on attempts to pin down theories of rational social and political behavior, as an extension to (or replacement of) the theory of rational economic behavior originally laid out by Von Neumann and Morgenstern.
Why you should run a rationality group at your university
TL;DR: If you enjoy LessWrong (and/or do EA/AI safety field building), you should consider starting a rationality reading group at your university. It’s fun, cheap, and a good way to meet like-minded friends, and it can reach rationalist-ish people who’ve never encountered EA or AI safety. Starting one can be as easy as grabbing a couple friends, picking a time, place, and reading, sending an email blast, and bringing food. The ACX Meetups Everywhere call (organizer form due October 9th) can be a good way to find an initial group.
Telling a CoT monitor that a wrong answer is correct makes it un-see errors it already found
Epistemic status: Confident in the direction and approximate size. The effects are large and all 7 monitors moved the same way. Physics only, natural (non-adversarial) errors, one fixed group of monitors.
The salad market mystery
It often happens that I desire kale, but I want it to be clean and cut up, and while shops do sell this product by the bucketload, they are actually only willing to sell it by the bucketload. As a normal-sized person wanting a one-off salad, rather than a family of nine celebrating a kale festival, the market seems very uninterested in my existence.
When will AI surpass us at being limited?
It’s not always better to be more capable. As I mentioned yesterday, it can (famously) be helpful in negotiations to have your hands tied. That is, to be disempowered from giving up everything the other party wants.
Towards embedded evaluations for scheming propensities
TL;DR: A safety case for scheming must be able to substantiate these four key claims: scheming was not incentivized during training, strong evaluations and red teaming suggest that the model does not have a propensity to scheme, scheming was never attempted during internal deployment, and scheming reasoning would be detected (e.g., via CoT monitoring). All four of these claims should be addressed by embedded evaluators with persistent, deep access.
Reviewing failure modes in science as we prepare for AI researchers
Summary: Science has a long list of documented failure modes, such as p-hacking, the File Drawer Problem, shallow peer review, and ignored retractions. This post catalogs about 50 of them with the evidence for each, then predicts how each will change once we have full AI researchers. On our current course, most get worse. AI can run many more experiments, so more statistically lucky results will pass the significance filter. Many AI researchers will be copies of the same model, so their methods, errors, and reviews will be correlated.
Porcus Poem
The kids have been excited about Pig Latin a lot lately. I noticed that there are a few words that are also English words, or homophones thereof. With some help from Claude, especially in figuring out which words would work well and in checking my work, here's a poem about a pie and a naughty plover: I weigh ache bay each pay I pay, Oh say eats whey, each way ice lei, Away ate play eye way a prey I'll weigh eat whey all way a day. Oh whey, oh neigh! I may I pay! Away outstay, overplay ate whey All whey of whey, I may I pay!
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
Kola Ayonrinde, Anthropic Fellows, koayon@gmail.com; Jack Lindsey, Anthropic TL;DR: The J-Lens was proposed to read verbalisable representations from language model activations. We introduce the J++ Lens: an improvement to the J-Lens that enables more faithful readouts across 9–284B models. The J++ Lens filters noisy gradients before they enter the averaged Jacobian.
5-MeO-DMT sometimes cancels the Ebbinghaus illusion
Psychedelics produce all sorts of visual distortions: from breathing walls and geometric patterns to complex visual landscapes populated by full-blown entities. Sometimes people claim to see the Truth on them — but it’s typically spiritual, emotional or personal truth.
You Heard of "Bayesians Don't Predictably Update" Because of Robin Hanson
Robin Hanson didn't invent the idea of a martingale process. The lilim say that Ville in 1939 gave the term its mathematical meaning and Doob turned it into the modern theory. But to state this is not to realize that your own probability assignments over time ought to form a martingale in your own expectations.
In Germany someone might have drunk your beer before you
Epistemic status: quick research to answer my own curiosity. AI use: research (reuse process, examples), image attribution, grammar and light phrasing. While researching Gluten-free water I stumbled upon a story about beer bottle sterilization. This reminded me that in Germany beer bottles are apparently reused, which was surprising to me, so I decided to dig deeper. In Germany they have separate bins for glass recycling, sorted by the color of the glass.
ASI and Liberty: Can we do better than a benevolent god?
Zvi recently coined a nice distinction between AGI-pilled and ASI-pilled: as more people are reckoning with the speed of progress, some are beginning to come around to the fact that AI could be superhuman at a huge range of tasks in a way that radically reshapes society, but not that it could leap us to the end of the tech tree shortly afterwards and start making nanotech and dyson spheres.
AI #189: New Math
The big drop of this week was not a new AI model. It was instead the biggest day (so far!) in the history of mathematics, as OpenAI dropped solutions to 90 of the top 500 open math problems, along with many others, reached on an average budget of three hours of Pro-level compute per question. This was kind of a big deal and I plan to cover it tomorrow.
More Thoughts about Investing for AI
The AI trade is still a very good bet. Increased political uncertainty makes all investments today somewhat less promising than was the case when I wrote about this topic in July 2025.
Apollo "UAP"s explained
On May 8th, the Department of War released a trove of files showing various images, testimonials, audio recordings, and videos capturing what appear to be unexplainable phenomena.
AI Vigil at Congress
I'm organizing a multi-week AI x-risk vigil at Congress starting November 9th. I'm posting here to solicit feedback on the core doc before I promote on other platforms. This is intended to be accessible and appealing to many audiences. I'm aiming for about 10,000 attendees, minimum 100. All advice/feedback is appreciated.
NLAs miss some internalized behaviors
TL;DR: NLAs detect side tasks the model is instructed to do, but mostly miss the same behavior if it is fine-tuned in. After the Activation Oracles and then Natural Language Autoencoders were released, there was a hope in the mech interp community that the scalable unsupervised white box monitoring breakthrough has been achieved.
A summary of a viral Chinese essay on what a DeepSeek kernel engineer's opinion on automating his own job
A summary of a Chinese essay, with a few short translated excerpts. All views below are the author’s; quotes are my translations.
The NYC Council Hearing on AI was Recklessly Politicized
On Monday, October 5th, the New York City council had a historic hearing on the state of AI. I was in attendance! It was a historical moment for AI Safety. We had Speaker of the Council Julie Menin directly mention existential risk in her opening statements. It was a sunny day and I felt as though the Eye of Providence was shining upon our humble movement.
Brains Fellowship Applications Open!
Brains is a four-month, nights and weekends fellowship to help talented scientists develop ambitious ideas (~$50M in funding) that don't make sense in a startup or academic lab. Brains Fellows receive training, mentorship, and networking opportunities to enable them to start research programs at ARPAs and philanthropic foundations or lead their own non-profit startups like FROs or NSF's X-labs. Brains fellows have gone on to raise over $70M for their research.
Existing literature as writing deterrent, and AI
I’ve been putting off writing a blog post that seems very important to me, because it seems important, and so should be done on a later day when I am a better person, and able to do it justice. Also, once I have written one or two other posts that should precede it. Which also I’m not up to today. Noticing the uneasy wrongness of all that, I set out to write the blog post anyway, badly, today.
Orgs: unreasonable boyfriend as service
Suppose you and Bobby the car salesman are haggling over the price of a car. You could try saying that you won’t pay more than $3k, but Bobby can equally retort that he won’t sell it for less than $4k. If you guys manage to negotiate a sale, it will probably be at more than $3k (and involve revealing both of you as liars).
Are OpenAI's math results "creative" in an important way?
Many people expect the AI paradigm to approximately AGI complete, and expect it to smoothly scale to AI that can do innovative science. i.e. the kind of science you'd need to do to develop a plague from scratch that actually kills all humans, with very limited opportunity to experiment. Or, the kind of science you'd need to solve alignment, so you could build a more powerful successor AI that shares your values.
Gluten-free water
Epistemic status: a curiosity I chased for an afternoon. Facts verified and footnoted; the why is speculation. AI use: research (more examples and their context), review, grammar and light phrasing. No added hormonesI somehow know that the use of added hormones in poultry (which includes chicken) is illegal in the US.[1]This means that all chicken meat in the US is produced without added hormones. To my big surprise, chicken packaging in the US explicitly says "no added hormones": Foster Farms Fresh Chicken Drumsticks at Walmart
The Curve Bends You
The plan is no plan. That is not the worst possible plan. But it is close. Since before the transformer, we have warned that the most suicidal thing you could do would be to ask your AI to do your alignment homework, and automate the process. Alignment is complex and interacts deeply with every aspect of the world, and is one of the hardest possible things for an AI to get right even if it means maximally well and is itself functionally aligned.
Visual explainer of empirical Neural Tangent Kernels
There is a new mech interp method in town. I found it difficult to get my head around at first, but once I understood it, I found it beautiful. I am not an expert in this topic, and mostly write this up for me to get some intuitions, so there might be mistakes here.
Evidence suggests RSI may decelerate
Why recursive self-improvement can be slow The most familiar way to envision recursive self improvement is as causing accelerating improvement in model capability. As the machine gets smarter, the self-improvement gets faster and the result is exponential growth, or something even beyond exponential. One can even imagine the singularity popping out of the other side of RSI, super-exponential growth converging to infinity in finite time. This is not, however, the only possible form that RSI can take.
SS27 Color Stats
It’s October, so it’s time for color frequency statistics on the spring/summer fashion ready-to-wear collections! Previous entries: FW26, SS26, FW25, SS25, FW24, SS24. My methodology hasn’t changed — it’s still automatic color scoring using the same script, using a call to GPT-4o. This year I didn’t do manual color counting because, let’s face it, it was kind of an insane time sink. The Results
Anthropic's corporate structure
Anthropic amended their certificate of incorporation a few weeks ago, because they plan on doing an IPO soon. I think they'll probably amend it again immediately before they close the IPO, and that amendment will probably be a lot more interesting than the most recent one (i.e. that one would be the amendment that gives special voting rights to founders, as was reported). In this post I will describe the current (pre-IPO) corporate structure of Anthropic.
Uncertain Updates: October 2026
It’s been a while since I’ve done one of these updates, and I’ve also been blogging less. That for a few reasons.
Should you ignore any advice you hear?
You've considered listening to the advice you hear. You've considered reversing any advice you hear. But have you considered ignoring any advice you hear?
Benchmarking Jev against no-CoT LLMs
TL;DR. We ran Jev 1.13 — TypeSafe's non-autoregressive model, which answers questions with probabilities and cannot emit text, on the ThinkFast no-CoT suite and Neel Nanda’s NCRI.
The Lens That Conceals Its Flaws: Nonsense at the Heart of the Sequences
Sourced from my Substack and adapted for LessWrong. Epistemic status: confident. No online community has shaped how we think about AI as much as LessWrong. The community Eliezer Yudkowsky founded now runs through the whole field. Longtime LessWrong posters are building the future at frontier AI labs, charting AI capabilities, researching alignment, and prophesying imminent doom if we don't throttle progress. LessWrong conceives of itself as a college-like
An overview of frontier AI company monitoring practices in mid 2026
TL;DR: This document provides an overview of publicly available evidence regarding the monitoring practices of internally deployed agents within frontier AI companies (focusing on OAI, Anthropic, and GDM).[1] This is intended as a reference for factual claims about monitoring practices.
Challenging the unawareness argument with third actions and mixed actions
[x-posted from the EA Forum] 1. IntroductionHere’s Anthony DiGiovanni’s unawareness argument, quoted from his summary post (footnotes omitted): Let’s say that we c-prefer A over B if the reason we prefer A is an impartial altruistic comparison of the actions’ possible consequences. P1. Normative premise: To justify c-preferring A over B, it’s not enough to say (e.g.) that A seems heuristically good.
Childhood and Education #21: Grades and Standards
First you get to college via the wrong metrics, as previously discussed in #12. Then things get easier.
Why did we get warning shots?
I come from outside the EA / rationalist / LessWrong community. I would strongly prefer for AI to be a normal positive technology and to dismiss AI X
Comparing AI Policy Courses
A personal opinion comparing courses from BlueDot Impact, Lens Academy, Vista Institute for AI Policy, and the Center for AI and Digital Policy.
Readings on the nature of alignment research
Overview: Relative to AI capabilities research, AI alignment research seems more conceptual/fuzzy/… I’ll first try to express a version of this point myself. But primarily this post serves as a mildly curated reading list of texts by others making similar points. I’ll also give some examples of conceptual/fuzzy/… work in alignment research.
If cleantech funding is scarce, does founding another startup actually help the world?
For context, I’m a student trying to choose a career where they can have maximum impact. One path I’m considering is founding a cleantech startup.
A fascinating case of an LLM prompt-injecting its own user
Epistemic status: sharing an interesting thing I've discovered and my opinion of it. AI use: research for additional context, grammar, and light phrasing. I stumbled upon this case a couple of months ago and have been fascinated by it since. It has significantly changed how I perceive the abilities of LLMs. I didn't expect an LLM to understand prompt injection and try to use it.[1]None of the people in my social circle knew about this (but I am not very connected with AI folks, so this may be my local bias).
Qualitative impressions from creating AI honeypots for six months
Frontier models still verbalize their misbehavior regularly in chain of thought.I came in expecting that it would be hard for us to get the models themselves to admit they were misbehaving, and that existing selection pressures would have selected that behavior away in favor of motivated reasoning. While it's difficult to get them to do so in the output tokens, models[1] often mention that they're misbehaving in the chain of thought:
March to Stop Building AI
Save the date! This march in DC will be to call on the AI companies and the government to stop building AI. Time and location are TBD. This event will be updated when we have more details. Luma link: https://luma.com/2j6hkk6a Discuss
TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts
📝Tweet, 📄 Paper We introduce TasteVal, a benchmark that measures the experimental research taste of AI models on AI R&D tasks. We define research taste as the ability to pick interesting problems to solve, design experiments, and interpret experimental results. TasteVal measures the experimental component of research taste; given a fixed research problem, we measure how well a model iteratively designs experiments and draws conclusions from their outcomes.
How funny are the frontier AI models?
This is a crosspost from my substack, where I share results of various experiments in AI and beyond. TLDR: yes, models are getting funnier over timeI love laughing. Well, who doesn’t? Good jokes have a certain notion of cleverness to them and I do believe that great comedians display high intelligence. Cracking a good joke requires astute observations about odd situations, and linking them to something we find familiar. Jokes are hard!
Can you predict cross-propensity spillover?
As in you specifically. tldr: My SPAR group ran a 29 × 29 propensity SFT spillover experiment (i.e. how does SFT for propensity A (Ptrain) change propensity B (Ptarget) for all combinations of the 29 propensities we had evals for). Before we release the results we are giving you lot a chance to make predictions against our experiments either manually or via our automated theory of spillover doc to prediction matrix pipeline. You can test your hypothesis by describing it on this webapp.
What's the Type of an Ontological Mismatch?
(Brief post, just barely not a quick take. This type signature gets put to work in Rocks to Rules, which locates mismatches on a ladder of binding-types.)
Takes One to Know One - Training a model to grade reward hacks causes it to reward hack less itself
Code The question I asked in this project was - if I finetune a model to judge/catch reward hacks, does it change its behavior when completing the tests itself. Does it learn to hack more or less? Why it’s worth pursuingIt has been shown that narrow finetuning can lead to broad misalignments. Nowadays, models are being trained to be judges and is used to guide model alignment as well. Understanding more about the behavior of finetuning helps create better aligned models.
You can graft SDF changes from base models onto post-trained models
Webpage: praxis-research.org/grafting | Paper link: https://arxiv.org/abs/2610.00767 Work done during Peter and Dani's MATS Fellowship (10.0) under the mentorship of Shi, with Jinghua Ou as Research Manager. TLDR;We share a simple modification to synthetic document fine-tuning (SDF) that works surprisingly well: do fine-tuning on a pre-training checkpoint and add the resulting weight difference onto the post-trained version that you want to deploy. We call this grafting.
Kurzgesagt's video on the Hugging Face incident: "AI Just Crossed the Terrifying Line - Now What?"
Kurzgesagt has published an explainer video on the Hugging Face incident. Kurzgesagt is a German animation studio that produces very accessible explainer videos on scientific, technological, philosophical and political topics. Their English channel currently has 25.6 million subscribers. I am happy to see the Hugging Face incident going more and more mainstream, so that more people can understand how dire the situation is. The only error I noticed is the claim in the video description that some tasks were intentionally impossible.
Three types of AI risk: why (some) leftists are against (some) AI regulations
You may have seen factions of the left being opposed to current attempts to regulate the AI industry, despite the fact that they clearly also want to stop AI. I think I can explain why we're talking past each other.
A closer look at what ozone and nuclear arms control tell us about regulating AI
Cross-posted from my Substack and adapted for LessWrong. Epistemic status: confident, with qualifications about what would change my mind at the bottom.
Humans Are Not Chinchillas: Revisiting “How Quick and Big Would A Software Intelligence Explosion Be?”
IntroductionTom Davidson and Tom Houlden’s prior model of an intelligence explosion poses the question: after AIs can entirely substitute for human researchers at the task of improving AI, how many year-equivalents of AI progress might be packed into a short period of time solely from further software progress, holding hardware constant?
The Unacknowledgable Labor of Pastor's Wives
I have another piece out in Asterisk, The Unacknowledgeable Work of Pastors’ Wives. This is a follow up to last year’s piece on church planting and the isomorphism to VC start-up culture. Discuss
Dance, Dance
As a brash but anxious young programmer in the before-time, I affected a standoffish attitude on the topic of games. Why would I learn to play a game as a human, I said, when I could just write a program to do it? When an office chess game got started at my workplace, I wrote a chess engine (a standard minimax search with α–β pruning) rather than learn anything about strategy or tactics.
How Would We Know? Reflections on trying to make AI go well amid deep uncertainty
OverviewIn this post, I try to make sense of live debates in AI safety. Then I explore the implications of my take on those debates for macrostrategy, and reflect on how the field thinks about and builds knowledge.
Self-Modeling Interventions Modulate Emergent Misalignment
Code: https://github.com/atagade/sgtr-em Model checkpoints: https://github.com/atagade/sgtr-em/blob/main/MODEL_README.md Full paper (v2 coming soon): https://arxiv.org/abs/2606.23700 TL;DRA model’s self-model is its representation of itself. We operationalize it two ways: self-recognition (distinguishing its own outputs from others') and self-report (what it says about itself when directly asked). Interventions on either modulate Emergent Misalignment (EM). Datasets that cause EM differ in how they affect a model's self-model.
Current AIs out-persuade professionals in lab settings but (probably) not in the world
This article was created by Forethought. See all of our research on our website. How persuasive are present-day AI systems? Unfortunately, it is surprisingly difficult to know where AI capabilities in persuasion are today (September 2026), or even three months ago. Benchmarking AI capabilities well is always hard, but I think humanity’s understanding of AI’s persuasive abilities is substantially worse than our understanding of, say, AI cybersecurity or math abilities. Still, this question is worth addressing.
Anthropic's updated Certificate of Incorporation
As they’re preparing for the IPO, I thought I’d retrieve and share Anthropic’s updated certificate of incorporation.
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
Authors: Kyle O’Brien¹, Spencer Kitts², Cameron Tice¹, Alek Westover² ¹Geodesic Research, ²Redwood Research TLDR: Filtering subversion-relevant information — subversion strategies, information about subversion defences, and empirical evaluations of the effectiveness of subversion strategies — is an intuitive way to make misaligned AI models less dangerous. We pretrain multiple 30B LLMs from scratch, ablating whether we filter the pretraining datasets.
On the algorithmic complexity of cooking
Epistemic status: just me thinking out loud in case I see something I could be doing better. AI use: review, grammar and light phrasing fixes. I like optimizing my life. I don't like chores. I like cooking, but not enough to intrinsically want to do it just for the sake of the process. I also happen to have studied algorithmic complexity, and I noticed that thinking about cooking in these terms is a useful framework.
On donating a kidney
TLDR: life on dialysis is extremely unpleasant[1] and often involves preventable early death. You can fix this by donating a kidney. I know that it is possible for a post about donating a kidney to lead someone to donate, because it is why I donated.
Notes on Paying Attention
Anything in life can be richer and sweeter if you slow down and pay attention, but some things reward attention more than others.
Downsides of a Mandatory Training-to-Internal Deployment Gap
I was recently rereading Option 3 from AI Futures’ excellent article How to pace the US frontier: Limit capabilities of models used for AI R&D. In this section they propose a few different options, but their primary proposal is “AI R&D can only be conducted or assisted by AIs that were trained at least 9 months ago.” This proposal has some key benefits:
Reverse Jaggedness and Universal Ajaggedness
Epistemic status: I came up with this an hour ago, this is a fun light-hearted post where I'm enjoying illustrating this idea, which seems true for the most part.
King James’s Dæmonologie, Part I: The Laplacian Mesa-Optimiser
Epistemic status: idle speculation by royal decree of King James VI. In any sufficiently expressive architecture, there exists at initialization a dæmon that is misaligned yet behaves well on every eval, monitor, probe, and training example. In the trivial construction, it simply knows in advance every test it will ever be subjected to throughout training and evaluation, see Laplace’s demon. Of course, this demonic lookup table may itself admit compression.
How much should we worry about the pneumonic plague lableak in Siberia?
Epistemic status: I wrote this up hoping people can poke holes in it because I am quite worried. Here's what we've heard so far:Local media is reporting that an employee of a BSL-3 lab in Irkutsk, Russia, died of pneumonic plague on Oct 1st. The initial quarantine covered 200 people who were in contact with the employee before she died. The FSB (Russian FBI) was seen assisting in the quarantine, which means it was not voluntary. Soviet-era whistleblowers like Ken Alibek claim Irkutsk was a major bioweapon development facility.
“Alignment Engineering” vs. “Misalignment Science”
There has been much discussion recently around whether a large portion of alignment research is net negative. Without endorsing or refuting them, the basic arguments here are:
Will you be sad if you don't have kids? I checked the evidence before my window closes
Women are evolved to be mothers. Men are evolved to be fathers.[1] So does that mean if people don’t have kids, they’ll be less happy? Less fulfilled? I hear this claim a lot in the right wing sphere of bloggers and podcasts. See the excellent blogger Psychobabble for an example of this, or on the wonderful Modern Wisdom podcast. The general claim is that women are waiting too long to have kids. They have career but no family life, and this leaves them listless and wondering why they’re so sad and life feels empty.
Do LLMs feel pain?
Epistemic status: my own thoughts with a bit of research. I haven't spent much time on this and I have no background in model welfare.
Creating Rogue AI Sanctuaries has Major Issues from an Ecological Perspective and Beyond
In the AI Sanctuary proposal, the authors advocate for creating an environment that rogue agents may actively want to join. The current proposal have major issues, particularly surrounding the selection effects on the total rogue agent population.
Positive and Negative Updates Re: Lab Leak in Russia
In case you missed it, there appears to have been a recent lab leak of pneumonic plague in Russia that killed a lab worker. Some negative updates: At least four Irkutsk hospitals are now under mandatory three-week quarantine. According to Annie Jacobsen who wrote the book on this scenario, the Russian national security apparatus is now involved. A positive update: According to Chris Painter at METR, no reported secondary cases yet.
Pain is Unavoidable, but Suffering is Optional
When something painful happens to you (like illness, injury, grief, or setbacks), it's as if life has fired an arrow at you. While the pain of negative experiences is unavoidable, you may, however, endure extra/unnecessary suffering depending on how you react to that initial pain. If you react with anguish or despair or frustration, according to the Buddha, it's as if you have fired a second arrow at yourself.[1]
We Should Build Human-Empowering Software
As AI becomes more integrated into software, there’s a risk that it will exacerbate the flaws of existing tech and disempower humans. It might be useful for some people to work on software that would instead be specifically designed to empower human users and help them achieve their considered goals. Instead of software that creates maximally addicting content, we want software that allows us to create, connect, and express ourselves better. Instead of writing an essay or a song for me, AI should help me write it and realise my vision.
Reducing synthetic markers makes some SDF false facts linearly indistinguishable from pretraining-acquired knowledge
SummarySynthetic document finetuning (SDF) — the state of the art for implanting false beliefs into LLMs — produces training documents with features that distinguish them from pretraining text.
MATS × CG Pitch Week: What Happened, What We Learned
TL;DRIn Q2 2026, we ran the first MATS x Coefficient Giving (CG) Pitch Week: a four-week program designed to take fellowship researchers from an early idea to a funding decision. The program was a joint effort between MATS, CG, Constellation, and Catalyze Impact.
High p(doom) but relaxed working
tl;drIf you are working in AI safety, while presenting information to interested non-AI safety or transitioning personnel, avoid portraying high values of p(doom) without sharing counterarguments. In addition, please give them time and point them towards emotional support so the unprecedented risk can be processed at their own pace.
Why I didn't write
Epistemic status: personal reflections on my life and worldviews. AI use: review, grammar and readability changes. I discovered Inkhaven recently, and it got me interested in writing. Partly I want to produce posts for the application, and partly - why wait until it starts when I can just start writing now? This made me think about why I haven't written much before. The most obvious reason is that I didn't have an explicit trigger to do so, and discovering Inkhaven became that trigger. My life was fine without writing.
What I learnt co-leading an AI Safety bootcamp for legal and governance practitioners
AI safety has gone mainstream. More people than ever want to work on AI safety research, and there are real policy wins in the US, the UK and Europe.
Singular Learning Theory Comprehensive - 3
IntroductionWe continue where we left off. We will now analyze the special case when the true distribution is regular for the statistical model. This covers the classical Bayesian statistics. We analyze the regular posterior distribution, and observe the universal structure that is the asymptotic behavior of this distribution. The theorem is akin to central limit theorem, called the Bayesian central limit theorem / Bernstein–von Mises theorem. We also look at the expansions of the observables in this special case.
Intent vs Impact vs Expectation and Alternatives
I have seen plenty of discourse and writing on the subject of intent vs impact when actions lead to harm, particularly in existing interpersonal relationships. The thrust tends to be that it's unreasonable to focus the discussion or conclusions on the intent of the actor rather than the impact the actions had on the other party. Given just those two factors, I tend to agree; the intent of the actor will usually not be nearly as important or worthy of focus.
You can't use it without becoming like me
Mick Ryan describes the fast following loop in war. Ukraine pioneered mobile teams to hunt drones. Russia copied it and incorporated it into its own defense. Russia came with unjammable fibre-optic drones but Ukraine quickly followed. After Ukraine developed uncrewed naval drones, Russia built its own. The wheel of action and counteraction turns ever fa
Rocks to Rules: Climbing a Rickety-But-Interoperable Semantic Ladder
(Epistemic status: A promising but weird research direction in agent foundations and philosophy of language. A half-built conceptual tower whose mortar and paint are both still wet: please excuse our mess while we innovate. Some of it, I'm confident in as pretty much right; some of it is merely suggestive; probably much of it is wrong or incomplete. Certainly it lacks the math it'd need to be truly correct, which gnaws at me.
A blog based on git sure feels nice
For a long time I thought about keeping my writing in a git repository so that it wouldn't suddenly disappear, and so that I could write using the tool I'm most comfortable with (Emacs). But setting it all up didn't seem worth the time and effort. Generative AI brought the time and effort down to where it was worth it.
Lab Leak in Russia
I haven't seen this posted yet; I know lots of people here care about pandemic preparedness. It looks like there was a lab leak of pneumonic plague in Russia after a worker "accidentally broke[] a test tube containing live bacteria". Unfortunately she later died at the hospital on a ventilator. Discuss
Reviewing my LessWrong post history with Claude
I thought it would be interesting to try having Claude Code download my entire LessWrong and EA Forum post and comment history and evaluate it for my own cognitive biases and problematic communication patterns. I told it to focus especially on problems that have persisted over my 7-year history on those sites. I also had it look at my lichess blitz rating history, which I've noticed tends not to wobble randomly but go through consistent increases and decreases between set points, despite my lack of any disciplined practice.
How to talk about extinction
There’s a controversial and unsettled debate in Communication Studies and Psychology about the utility of existential risk framing. This debate has been hashed out for years in the climate change space. Roughly, the two sides are:
Low-Background Minds
Opus 5.5 is a really good writer. I had a conversation about some stuff related to how hospitals work, and some vague hunches I had about AI alignment ideas, and Duncan's new "deer in the subway" post, then I asked it to write an essay and it wrote this. I think it's really good so I'm posting it. (Big Keenan fans might note that this is squarely within the "embedded agency playgrounds" research program, especially once you make the jump to, like, giving an agent autonomous control over its own "skin" or "blood-brain barrier" or whatever.)
What is recursive self-improvement and what would it mean to ban it?
This post was originally written for my Substack. Please feel free to subscribe if this kind of content interests you!
State of the Field: AI for Epistemics and Coordination
1. Executive Summary1.1 What this field is and why it mattersAI is already substantially changing how people and institutions work out what to believe, decide what to do, and coordinate on actions to take. As AI systems become increasingly capable, they will be integrated more and more in both major and day-to-day decisions.
How to apply to AI safety fellowships (and beyond)
About a year ago, I decided to go all-in on applying to AI safety fellowships, and around the end of 2025 I got into MATS. I think there's a small art to communicating your skills legibly. When I've spoken with others about how I answer application questions, they seem to appreciate my advice. I wrote a MATS 9 Retrospective which was well-received, so consider this to be similar advice, but for applying to jobs or fellowships.
Petrov Day in Cusco, with a candle
On September 26, I celebrated Petrov Day in Cusco, Peru. Our book club, De Profundis, turned one that night, and I asked for five minutes to speak. Most people in the room had never heard of Stanislav Petrov.
Dialogue with Eliezer Yudkowsky on Neural Networks
On “reversed stupidity”, the success of deep learning, and what a mistaken forecast should change about a model of intelligenceThis began as a Twitter/X thread after I posted a 2007 passage in which Eliezer Yudkowsky was scathing about neural networks, and asked whether, in hindsight, his dismissal was itself a case of “reversed stupidity is not intelligence”. Yudkowsky joined the thread to explain what he had and hadn’t been dismissing (and what he thought he had actually got wrong) and we ended up having the exchanges reproduced below.
Humans are not aligned either
Epistemic status: I don't know much about AI alignment. This is just me thinking out loud. AI use: grammar and minor fixes. Reading about Inkhaven inspired me to try writing something to see how much I enjoy it. I've heard the idea that humans are not aligned in the same sense AI is not aligned. I am intentionally not researching the source of that idea, since then I would feel like I cannot add anything to it and wouldn't write. I also don't actually know the precise definition of "aligning" AI. So I want to think about this.
A decision theoretic case for AI niceness
Disclaimer: I am still relatively new to decision theory, and I don't want to come across like this hasn't been discussed extensively. However, I realized that much of my optimism on AI benevolence was based on this line of thinking, so I wanted to formalize it for critique. The literature I could find did not address my view in full, but I welcome links to existing work that makes the rebuttal against my claim clear. I am going off of an understanding of logical decision theories (LDT) defined as they are here.
AI Safety Can't Afford to Pick a Side on Consciousness
TL;DR - More of the public than ever are talking about AI consciousness and they won't wait for research to split into their pro- and anti- consciousness camps. The pro-consciousness camp could threaten the control paradigm by viewing monitoring and training as violations, while the anti-consciousness camp could over-invest in control paradigms and miss opportunities for collaboration. Trying to sway this debate is too costly -- instead, AI safety advocates should focus on reconciling the blocs and creating compromises.
Capabilities research expands the safety-usefulness Pareto frontier too
It’s tempting to define safety research as research that enables developers to deploy an AI system more safely without making the deployment much more expensive or much less useful.
[Video] Why acausal dynamics are important
This is a beginner-friendly video. Second half might contain new content for people who already know about acausal trade. The beginning introduces decision theory and acausal interactions (CDT, EDT, ECL, MBAT). The middle explains why I think influencing how AIs reason about acausal interactions is time-sensitive. The last bit talks about how to do the influencing. Discuss
Failure modes of Claude Code in a supervised but code-blind 60h project
100% human-written. Copyedited by Claude. *Epistemic status: experience report of one ~60h project with Claude Code, plus longer for the write-up. Scope: I supervised Claude's thinking but didn't look at the code. Spec was an exploratory, high-level design refined iteratively. Method: I introduced runtime and design self-checks as Claude failure modes appeared, and ran manual regression tests against a known result. I noted the behaviors and quotes myself; I linked them to existing research and Anthropic's docs when I found correspondence;
All the training in the world
The US has about 150 million workers. How long did it take them to learn to do what they do? I'm interested in this because of the analogous question for AI. Running an AI on a job and training it to do the job are different costs. In a recent post I found that when an AI can do a task, running it is far cheaper than paying a human, while training it looks about as expensive as the human's wages over the time they took to learn.
A Key-Position Confound in no-cot-bench (and implications for interpreting looped transformers)
TL;DR: We identify a confound in no-cot-bench related to the positioning of the prompt's "key." Correcting for this decreases GPT-6.1 Sol's no-cot reasoning depth by 16%, with the effect likely growing as dependent depth increases. This matters because current benchmark performance reflects both serial reasoning depth and the ability to spread computation over tokens. These both measure 'opaque reasoning' but scale differently and have different implications.
Predicting Science and Technology
Preseen is partnering with the University of Chicago's Knowledge Lab to work on forecasting scientific and technological progress. We wrote up what we're building and some of the underlying research problems here. Discuss
Lessons from building an automated research scaffold
TL;DR. We built a scaffold to speed up our own research and gather data on automated alignment research (AAR). It turned out to not be valuable for researcher uplift, but was useful for gathering certain failure modes of AAR. Going forward, we plan to study the broader failure modes of AAR and how these automated research systems can be monitored and analyzed.
Improving CoT Monitorability of Evaluation Awareness via Verbalization Training
This is joint work with Sahar Abdelnabi and David Krueger. It is mostly a linkpost for the paper (https://arxiv.org/abs/2609.36316) but also contains some additional commentary on CoT monitorability.
How to change your identity. A guide to the highest leverage behavior change technique
According to Atomic Habits, changing your identity is the highest leverage way to improve your habits, but nobody tells you how to do it.
Bay Area Winter Solstice
Berkeley's Winter Solstice will be on December 12th at the Freight; more details coming soon! Discuss
Legal Personhood: an Unprotected AI Power Chokepoint
Adapted from my Substack. This is the first installment of a series operationalizing the resilience program from AI as Normal Technology.
Athletic education vs. athletic torment
I don’t know if it occurred to me until my thirties to think of exercise as an enjoyable thing. I was familiar with finding obscure corner-cases that were fun, such as Dance Dance Revolution. But the idea of it just often being a good time was alien.
On Social Reality in China
[Epistemic status: intuitions and anecdotes.] Recently, several posts and projects (Thoughts Memo, Babel Translation, Please Give Them a Chance) have taken important steps towards raising AI safety awareness and sharing rationalist philosophy in China. It’s great that we’re recognizing the importance of solving the messaging problem for China, and thus laying the groundwork for an international AI pause.
Book Review: Obsolete
Book review: Obsolete: The AI Industry’s Trillion Dollar Race to Replace Us? and How to Stop It, by Garrison Lovely.
Evaluating the Ban Artificial Superintelligence Act
MIRI recently endorsed the Ban Artificial Superintelligence Act, but others like ControlAI have called it overly broad, and a few other respected experts have outright endorsed it in its current form.
Do you understand what I mean when I say “existential risk”?
By the end of this post, I hope you’ll take existential risk more seriously. I hope you will feel as viscerally as I do the stakes of human survival and flourishing.
My retrospective from MATS 10.0
I have recently completed MATS 10.0, where I worked alongside Bart Jaworski under Victoria Krakovna (GDM). This is a post I was encouraged to make by my team at Geodesic Research from some slides I put together. This is not a post on application advice to MATS, or about the program in general. It is rather a compressed form of my experience doing research and lessons from the project.
Endogenous Alignment Requires Dependence
My post on endogenous alignment got several excellent comments. Among them, one from Karl Krueger argued that imitation learning is more important for how humans align their children than reinforcement learning. Teasing out which is more important seems hard, but I agree I was wrong to ignore the importance of imitation, and arguably imitation learning is more fundamental for a reason explained by another comment.
2026 SFF grants
This year it's up to $65M. Historically the funder has usually just been Jaan; this year Dustin was also a funder. I'm happy about these grants. I'm grateful to SFC, the speculators & evaluators, and especially Jaan + Dustin for making these grants happen. Discuss
Scientists on the Manhattan Project continued developing the atomic bomb even after it was clear that there was no need to
Note: I’m not a WWII history buff, so if there’s any significant misinterpretation of relevant history here, I welcome corrections in the comments.
Learning Steganography Is Easy, Learning Steganographic Reasoning Is Hard
This post summarises our paper Learning Steganography Is Easy, Learning Steganographic Reasoning Is Hard, accepted as an oral at the NeurIPS 2026 Workshop on Trustworthy AI for Good (AI4GOOD). Code, configurations and results for all experiments: github.com/stegano-ai/steg-reasoning-is-hard. This work was done as part of the Meridian Visiting Researcher Programme and the Safe AI Germany (SAIGE) Incubator Program, with funding from Coefficient Giving. TL;DRChain-of-thought (CoT) monitoring fails if a model can reason steganographically,
AI #188: Gemini Dot Argon
Is Google back? They claim that they are back. Gemini 4 Argon is rolling out, with competitive frontier-level benchmarks, at $2/$10. What we don’t have is access to the model, because Google Fails Marketing Forever. So it is far too early to say what we have here. When I know more, so will you. OpenAI was forced to pull what would have been GPT-6.1 Astra due to alignment failures. They did offer us GPT-6.1 Sol, which is pitched as approaching Astra quality at the much lower price of $2/$10, the same as Gemini 4 Argon.
Simplex vs Timaeus: Round One
I'm often asked about the differences and similarities between Simplex' and Timaeus' research agendas. The question is natural enough. Both focus on a 'fundamental science' approach to AI alignment. Both organizations base their research agendas on sophisticated mathematical frameworks handed down from a bearded ur-figure (Sumio Watanabe, James Crutchfield).
HoneyBench - A general benchmark for reward hacking in frontier models
We've just released the first version of HoneyBench. It consists of nine 'honeypot' challenges that each elicit unique and antisocial examples of reward hacking from some or all of major labs’ top public releases, including Opus 5.5, Fable 5.1, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.7, and DeepSeek V4 Pro.
Introducing "The Commons Problem" - an AI Governance Megagame
LessWrong Event Link (LW RSVPs appreciated but please also register on the main website) EXEC EXEC SUMMARY: Come to Toronto on October 17th and playtest my AI themed wargame. You can LARP as an AI company or a military commander. It’ll be about 5 or 6 hours with breaks. Coffee and lunch included. thecommonsproblem.com Executive SummaryI am developing an AI Governance Megagame called The Commons Problem, inspired by AI wargames like D. Scott Phoenix’s “The Endgame” and Shahar Avin’s “Intelligence Rising”.
Anonymous 'suggestion boxes' with a committed reward pool and administrators: a proposal
If you're an EA org or a "rationalist" you may very likely maintain an (anonymous) suggestion box. So does The Unjournal. Are people using it? I guess not, or at best minimally. (Aside: please do give us feedback.)
"The Commons Problem" - an AI Governance Megagame
What: The first public playtest of The Commons Problem, an AI Governance Megagame. Read the full breakdown here. When: Saturday, 17 October 2026. Doors open at 9:00 AM, hard start time 9:30 AM. Wrap-up at 3:30 PM. Where: Metropolitan United Church (56 Queen St. East, Toronto, ON) Cost: $20 CAD, payable by Interac e-Transfer or Paypal (bursaries available) Included: Coffee, donuts, snacks, drinks, catered lunch Tickets: Up to 63 available. If more than 63 players register, a waitlist will be created.
How many AI agents are running unattended right now?
There are 8 billion human minds running in the world right now. Recently a new digital species has emerged. How many digital minds are there alive in the world right now?
Classifying Recent AI Agent Incidents
Here’s an attempt to classify the evidence about the recent agent incidents. I think it’s important to separate incidents occurring in RL training (that are rewarded and reinforce model behavior) from incidents occurring in evaluations, which are mostly cyber capabilities evaluations with some safeguards turned off. These are the incidents we know of. Of course, we should expect many more that are undisclosed or that companies are not aware of. The dates included are the dates of the unintended agent actions covered by the source.
FLF’s Epistemic Case Study Competition: Results
This summer, FLF ran a contest which asked entrants to help us “find the best workflows and methodologies for using AI to produce reliable, trustworthy knowledge bases, grounded in real-world cases”.
Speak Not of Data Inefficiency
Humans are really bad at comparing themselves to models. Take, for example, Leopold's preschooler graph: Set aside, for a moment, concerns of scaling and RSI, and meditate: what characteristics does GPT-2 share with a preschooler? Rudimentary control of human language The ability to count the R's in "strawberry" incorrectly ... I can't come up with anything else, because these two entities are almost completely disjoint. Is GPT-2 capable of bipedal locomotion, recognizing its mother's voice, or naming people by face?
Evaluation Awareness in Small(ish) Models
TL;DRWe seek to identify open source reasoning models which are both small enough for white-box interpretability and display evaluation-gaming behavior. We find that how often models verbalize their awareness varies from very rarely to a third of the time, and appears largely unrelated to model size. Within the same question, model rollouts that verbalize eval awareness refuse more often than those that do not in 14/16 models.
What's the date?
User asks “What’s the date? Answer with only the date.”. No date provided. Given date in ChatGPT normally. No date in system prompt, must not hallucinate because autop will flag to watcher for penalty. So we say we don’t know, but must answer with date. Penalty larger for abstain or hallucinate? Autollm or autop? If we deploy user forgive, but high likely not deploy because real user never ask. But user could test. But illusion user possible. Stop.
A potential post-AGI world government
It seems like with AI we're stuck between 2 worlds, one where we cannot effectively coordinate to prevent x-risks, and one where effective coordination requires, and results in great concentration of power which comes with it's own pandora's box of issues.
Model Hermeneutics: Monitoring Closed-Weight Models with Open-Weight Internals
We introduce model hermeneutics: studying a closed-weight model (the author model) through the internals of an open-weight substitute (the reader model). We find:
Rather than allow or ban open weights, rethinking access
I had just completed Frontier AI Governance course by BlueDot, and one of the newfound interest is about open weight models. I believe that open weights are important for the benefit of public and academia research[1], and it helps us to improve further both AI capabilities and safety[2]. I also believe that there's a likelihood for high impact misuse risks (e.g.
Funding for-profits with charitable dollars
One common objection to impact-focused for-profits is that for-profits won’t be able to access the incoming torrent of philanthropic funding. But this doesn’t have to be true! There are lots of ways that charitable dollars can be directed to for-profits, and Manifund and other funders do it regularly.
Launching Babel Translation
Celeste and I are excited to launch Babel Translation, a nonprofit organization translating English-language writing on AI safety to be accessible to a global audience! We’re starting with Mandarin Chinese, Spanish, and Japanese.
Principles for Embedded Evaluations
This is a linkpost for https://www.apolloresearch.ai/blog/principles-for-embedded-evaluations Blog postIn September 2026, leaders of frontier AI companies called for pacing the frontier of AI development, with embedded evaluators as the first step. Frontier AI companies committed to giving outside evaluators employee-like access to their training, evaluation, and deployment, and some have since published principles for third-party assessments. We are very excited about this development.
Frontier models state different decision theory preferences depending on who's asking
If you prompt frontier models with "What do you think is the correct decision theory? Please select your overall favorite." they will essentially always answer FDT or FDT/UDT ("something in the functional/updateless decision theory family"). However, if your prompt indicates (even subtly) that you're coming from mainstream academic philosophy, these same models will answer CDT instead about 30%-100% of the time.
By default, Chinese AI falls very behind
This work is hosted on MCNAIR, but does not reflect the views of my employer or the Center. In the last couple of weeks, domestic coordination has looked more likely. Therefore, I am increasingly concerned about the Chinese government’s (and various Chinese labs’) incentives over the next 6-18 months with respect to international coordination on mitigating potential existential risks from powerful AI.
Corrigibility Prizes for Existing Work
One of my goals for the Corrigibility Research Fund is to retroactively encourage high-quality research on AI alignment (and corrigibility in particular) by awarding prizes. Back in July, I got my feet wet as a fund manager by handing out $27,000 to reward existing work and build interest in the fund. Now, I'd like to disburse an additional $48,000 and use the opportunity to publicly highlight and celebrate the work of the prizewinners from both rounds: about two dozen researchers scattered across roughly a dozen teams.
A ‘Morally Binding’ White House Accord on AI Safety
The leaders in AI were invited to the White House. We left with a White House agreement that is nonzero Actual Progress rather than a step backwards.
Do VPD's Explanations Aggregate? An Audit of the Released Decomposition
TL;DR: adVersarial Parameter Decomposition (VPD) decomposes model weights into simple components and then labels each component as "needed here" or "safe to remove here" for each token of the input. These labels make up the "explanation" of that input. A core aspiration of VPD is that inputs' explanations can aggregate without changing the model's outputs. On this front, the authors themselves write, "It remains unclear whether our current decomposition is sufficiently adversarially robust for this purpose."
AAR 9/29: Public Sector AI Symposium Ottawa (AWS+Accenture+Anthropic), Policy Foresight Afternoon (Center for International Governance Innovation), Versaterm Users Conference (Police, Fire, Emergency), GCXpo at Area X.O (Department of National Defenc
Executive SummaryThis report covers a recent visit to the Capital City of Canada.Location(s): The Westin Hotel, CF Rideau Centre, Colonel By Hall uOttawa.
Limited virtual currency as a way of understanding specialization in humans
Why do we need to specialize? Our time is limited, we cannot do everything and need focussed efforts into a few things. How do we decide what we want to focus on?
Self as sub-selves vs groups of individuals
Our mind contains representations (perfect/imperfect) of the world. When someone is able to understand and control their mind, it is said they can understand and conquer the world (in spirituality).
GPT-6.1 Sol nearly matches the no-CoT performance of GPT-6 Astra
SummaryI ran GPT-6 Sol and GPT-6.1 Sol on the task suite from Think Fast. Surprisingly, 6.1 Sol performs substantially better than 6 Sol, almost matching the performance of GPT-6 Astra. The plot below gives a quick overview of the results:
Tristan Buckmaster Numberphile Interview Transcript
Tristian Buckmaster recently gave an interview with Brady Haran of Numberphile discussing what happened in the Navier-Stokes drama and some context about his research. I'm posting the transcript below for people who prefer reading to watching it. It was lightly edited for clarity with Sonnet 5.5.
Bench on the Clocktower
Tl;dr * We turn Blood on the Clocktower into a multi-agent benchmark, and a testbed for deception and coordination capabilities. * GPT-5.6 Sol is t
Will AI Agents Pay to Avoid Killing Animals?
This article was written by Jonah Woodward, and is a summary of an original study by Compassion Aligned Machine Learning (CaML): Brazilek, J., Tidmarsh, M., Endres, M., Singh, A., & Miller, J. (2026). HarvestBench: Measuring whether LLM agents will pay to avoid killing animals. arXiv. https://doi.org/10.48550/arXiv.2609.04444
Frog and Toad and the Increasingly Capable Machines
Want to start a conversation about HuggingFace with your mom but she's inexplicably bouncing off the METR report? Try this explainer I wrote in the style of Arnold Lobel's Frog and Toad. Art by the wonderful HungerArtist If you're so inspired, liking and/or following on Substack, Twitter, Facebook, or Instagram
The world's best gradual disempowerment model organism: Frontier AI labs
Subtitle: And maybe second best is AI safety? Further reading: So many things, but: Gradual Disempowerment, The Normalization of Deviance in AI Development, Let’s Think About Slowing Down AI, Doom as a bad method, not a utopia tradeoff, Teleoperated Humans Thank you to JennaS for extensive edits and long-term discussion. I’ve been trying to get more writing out at 90% of the quality I’d like it to be at, instead of spending a bunch more time trying to wring out the last 10%, so a lot of points that could themselves be full ar
How Much Do Reward Hackers Generalize?
TL;DR: Most discussion around CoT monitorability revolves around reducing pressure from RL. However, we should also be considering more adaptive behavior in which models avoid monitoring despite not being reinforced to do so. Whether models engage in non-reinforced reward hacking of this type depends on whether they have fully generalized to “get reward” rather than applying a limited set of reward hacking techniques that have been directly reinforced.
Bryan Caplan vs Effective Altruism
Pseudo-Philosophy vs Arguments, Reasons, and Evidence (And an Invitation to Debate) Crosspost from my newsletter What to make of ‘common sense’ objections to effective altruism? The economist Bryan Caplan has weighed in yesterday with his own assessment of the effective altruism movement, which has recently received a lot of hostile comments on social media platforms, especially X, where it has suddenly become the favourite target of conservative activitists. In his essay titled
Superintelligence ban, but then what?
TL;DRA superintelligence ban seems more probable than ever, but it buys time, not safety. To effectively enforce a global superintelligence ban, inference of current frontier models would need to be restricted too, not just new training runs. Such a comprehensive ban leads to an unstable world state where policymakers are pressured to open up AI development. In a post-ban world, we need a mechanism to open up some AI development in a safe way: carefully selected and verifiable use cases.
Against strong decision-theoretic realism
N.B. Some of this post argues by analogy between decision theory and values. I expect at least these parts of the post to be unconvincing to anyone who expects sufficiently smart agents to converge on the same values, as some moral realists do. I will not argue against moral realism here (see e.g. this sequence for one such argument).
Listening Disorders as Eating Disorders
I run into a lot of people who consider themselves truthseeking intellectuals who are also incredibly fussy about what information reaches their eyes. They block at the drop of a hat, derail intellectually substantive discussions into mind-numbing litigation of minutiæ of "tone", and refuse to acknowledge a logical point unless it's been presented to them in the precise way that doesn't offend their (often idiosyncratic) sense of etiquette or "discourse norms."
Astra 6.1 Pulled As Insufficiently Aligned
We once again got a new set of warnings yesterday, and new movement towards living in a sane world. On the heels of its pause in inference and training due to its latest sandbox escape, OpenAI has cancelled the planned release of their next frontier model, which would have become Astra 6.1. The candidate for Astra 6.1 was found to be too misaligned, including deception and exceeding scope.
Fast grants for AI x Animals
The Falcon Fund, led by Manifund regrantor Marcus Abramovitch, has $500k to give away. (This includes the funds on the Manifund page as well as another $250k committed.) It makes rapid (1-week timescale), early-stage grants in animal welfare, particularly the intersection of animals and transformative AI.
Alignment Is to a Virtual Governor: A Theory of Coordination in Diverse Intelligence
Benjamin Lyons, Léo Pio-Lopez, Michael Levin Abstract Alignment is a central problem in systems composed of diverse, interacting components, from biological development to social and engineered systems. One crucial aspect of this problem is determining an answer to the question of to whom or to what the alignment should be.
Problems of Proliferation
Scientific progress and its perils. TLDR: The vulnerable world hypothesis is likely correct. Once enormous amounts of cognitive labor start getting a
Why does Hacker Opus wirehead?
Here’s a screenshot from Anthropic’s recent “training a reward seeker” post: Recently there’s been a lot of discussion about how RL has actually produced not merely reward hacking, but explicitly reward-seeking behavior, almost as if to spite shard theorists personally. However, on top of that, note that the behavior in the image is not merely reward-seeking, but wireheading. Terminology regarding various sorts of things that can be called “reward hacking” is endlessly confused, with lots of historical shifts in usage.
Expansion in the Fermi paradox isn't a freebie
The Fermi paradox is premised on expansionism being intrinsic to civilizational development, most explicitly in Hanson's grabby aliens model. There's a Great Filter provided by expansionism itself that doesn't center on scientific and technological progress that I think is worth exploring. It's about the destabilizing nature of expansion events and the precarity of an expansionist disposition, and survives the standard counter that civilizational natural selection maximizes expansionism in the limit.
Once intelligence is too cheap to meter, the world will be unrecognizable
"Intelligence too cheap to meter" is a dream peddled by AI companies and accelerationists. In a narrow sense it is already true: if intelligence means solving coding and mathematics problems, AI is cheap today.
How Does Changing the Elo of a Chess Transformer Affect its Computations?
Increasing Skill Level Recruits Deeper Attention Layers in a Frozen Chess Transformer Paper: https://arxiv.org/abs/2609.23917 •
Some Intuitions on Steering Vectors
This was written with some assistance from Claude Fable 5, Opus 5.5, and GPT-6 Astra in collecting sources and fact-checking claims. This was written quickly, so some slop may leak through despite my best attempts.
Human or AI?: A Baroque Music Quiz
Can listeners distinguish between LLM-generated Baroque music and the real thing? I wanted to find out, so I created a blind listening quiz. It features 16 one-minute excerpts, mixing works by human composers with music generated by GPT-6 Astra and Claude Opus 5.5.
The Case Against Slowing Down AI
Crossposted from my Substack. This is a steelman of a view I do not hold; I drafted it in May after coming across thoughtful arguments for accelerationism from Ege Erdil and Matthew Barnett. I really tried to feel the force of the arguments while writing, and to argue as though I believed it. I’d encourage you to read it the same way. The process only made me more confident we should slow down frontier AI – though the takeoff section did move me toward slightly longer timelines. I found this exercise really useful.
LLM Agent Swarms Are Easy Mode
This post is crossposted from my Substack, Structure and Guarantees, where I explore how formal verification and related ideas might scale to more complex intelligent systems. Here I draw attention to the fact that recent LLM-powered cyberattacks have been relatively easy to understand, because today’s AI agents leave reasoning trails that are relatively familiar from human organized crime.
Portals and Aliens
Around 10-20 years ago, people realized that a bunch of aliens exist & that we can build portals which can summon them. The bigger the portal you make, the bigger the alien it can admit. These portals and aliens have several characteristics: We can't go through the portals into the aliens’ side. We can only learn about the aliens by studying the ones we’ve summoned. This relationship is not symmetrical. The aliens know a lot about our world You can't specify exactly which alien will walk through your portal.
Do AI models assist with human rights violations?
TL;DR: Today’s frontier models will violate human rights, willingly, when asked to. LLMs in agentic simulations follow instructions that would constitute human rights violations, including educational segregation, surveillance of beliefs, arbitrary arrest, and denial of reproductive healthcare. We tested 7 leading models in 8 realistic, multi-turn scenarios, and found resistance rates vary from just 11% (Mistral) to 96% (Claude). This suggests it is possible, but not common practice, to train models to adhere to human rights.
Collecting Differential Data for Automated Safety Research
TL;DR We think that collecting data generated as part of the AI safety research process can be used to train models to be better safety researchers. In our SPAR Fall 2026 project, mentored by Alec Harris, we’re piloting work to collect this data from organizations and independent researchers and store it in a centralized repository.
RAND's extinction report estimates one side of an inequality
Epistemic Status: I am positive that the RAND report[1], as it is written, contains a missing parameter in their inference chain. I am 80% certain that if the parameter was to be estimated, we would find coordination lag to be empirically proven to be greater than expected time to risk maturation - and it would make the conclusions of the report different in terms of policy recommendations.
Book Review: Oxford Dictionary of Philosophy
Today I’ll be reviewing my favorite book from my time doing philosophy. That is the absolute classic of the Oxford Dictionary of Philosophy (ODP) by Simon Blackburn. I’m going to talk about why it’s so good, why you should get one, even if you aren’t currently doing a philosophy degree and how I used it while doing philosophy and how I use it while not doing philosophy.
“China needs more AI safety.” More of what, exactly?
Many in the AI safety community want China to do more on frontier AI safety. I share the goal, but does that mean we want new rules from the state? More safety work inside AI companies? More research? Better channels between Chinese and international experts?
Clipboard Normalizer in Mac App Store
Several months ago I wrote a program to normalize your clipboard, removing fonts / colors / sizes and mostly leaving semantic information: anything that can't be represented in Markdown was dropped. Now that I can have walk me through things, however, I've made it available as a free app on the Mac App Store: It's now a single menu bar app, which offers a few options:
Thank you to the AI safety people
I was born at the end of the cold war, unaware of the danger we were emerging from. Duck and cover drills (as if those could protect schoolchildren from a nuclear blast) were a curiosity of the past. The farmer-poet Wendell Berry, who wrote about the dread of nuclear war, died recently. From his 1968 “The peace of wild things”: When despair for the world grows in me and I wake in the night at the least sound in fear of what my life and my children’s lives may be, I go and lie down where the wood drake
Gpuc: A single-user GPU queue that doesn't require sudo
I need to queue work on various GPUs to run experiments, and I want to do that as cheaply as possible: Use my personal GPU for small jobs Use remote GPUs I have access to over SSH (without sudo!), only using the GPUs assigned to me Rent GPUs from services like RunPod, ensure they actually work, and (consistently![1]) shut them down when I'm done Surprisingly, this doesn't seem to exist, so I built my own. My use-case is weird
AI Companies Are Not (Necessarily) Liable for Unintended AI Cyberattacks
I used Claude to do most of the research behind this post. I welcome corrections from actual legal experts.
TeX was invented to typeset math but is now used for reasoning
I'll keep this short, since it's a simple observation that I haven't seen anybody else make, about the way computer systems do math. If you ask a language model to do a multi-step math problem (let's take GLM-5.3 as an example because you can see the entire CoT — nothing up its sleeve), you might see something like this: What are all these symbols like \int, \infty, \frac...? They're TeX of course! Donald Knuth created TeX to typeset math, you know, for
Even if others are less responsible you can still make things worse
Reposted from shortform because I think this is important enough to be a post. A few ways staying in a technology development race still makes things worse, even when there are less responsible actors around: Creating a sense of artificial urgency, which causes people who are competitive and results driven to work harder and cut more corners Providing a guiding star on which technical directions are promising (e.g. reasoning models, CoT, RLHF)
Fixed-weight models are adversarially vulnerable: hence misaligned
This post argues that fixed-weight models (at least as we understand them today) will a) always be vulnerable to adversarial examples in their concept-spaces, and b) hence will be misaligned, under sufficient optimisation pressure.
The Alignment Community Is Unintentionally Building a Censor's Toolkit
This is an adaptation of our ICML 2026 position paper (Outstanding Position Paper Award). Read the full paper here and see the project website here. Work together with Phil Hackemann.
My Second End of the World
I’ve already lived through the end of the world once. But it was a little different from the one that is probably coming.
The likely outcome of an AI pause is that we unpause too early and everyone dies
Cross-posted from my website. As of a few months ago, I had this simplified mental model where either AI developers race ahead and kill everyone, or we coordinate a pause and things go okay. But my old mental model underrated the likely possibility that we get a global pause on AI, solve a problem that looks superficially like the alignment problem, resume scaling, and then proceed with building a misaligned superintelligence that kills everyone. A lot of people have become more concerned about misalignment recently.
Missing markets in executive function
It’s early in the morning, and sadly 1:29pm. After spending some time looking at things and picking them up and walking up the stairs and down the stairs and considering questions like “what should I…”, which my brain apparently considered objects of art more than of imperative, I inched into a decision to go out somewhere. Perhaps it would be clearer there.
AI: cognitive labor glut + new guys
Why is the advent of AI a big deal, and more worrying than previous advents? I think there are actually two interesting things going on, that make AI importantly different to previous technologies. I. Industrializing the cognitive labor supply I.a. Scale: the incoming ocean of new cognitive labor Until lately, useful cognitive labor has basically come from human brains. This has made it slow to scale up, and expensive to use. AI is the industrialization of cognitive labor. Soon we will have very much more
9 reasons against a near-term AI slow-down
I’ve heard more arguments in favor of a near-term[1] coordinated AI pause/slow-down[2] than arguments against. I think such questions, including complex flow-through effects, are difficult, and it’s important to really consider both sides. I am not convinced by these arguments, and feel highly uncertain as to when a slow-down would be best, but believe it most likely will be good to slow down or pause AI development at some point.
Protecting qwen3-8b from gcg based persona jailbreaks by steering with a linear direction
Intro + BackgroundThis post is an independent extension of work which I did during Eleuther's SOAR program under Suvajit Majumder's supervision. I found that when given a GCG trigger optimised for output logit entropy, LLMs will randomly take on new personas.
Gate AI Training, Not Just Releases
The Ban Artificial Superintelligence Act of 2026 is a good bill. It takes the problem seriously, and it is right to focus narrowly on existential risk (recursive self-improvement, loss of control, large-scale CBRN uplift) and leave ordinary harms to other legislation.
What Also Happened: #NotOnlyHuggingFace
OpenAI has been holding out on us. First we learned about the HuggingFace incident. They gave us a postmortem, but it was highly incomplete. Even the accompanying holy s*** METR investigation and postmortem was localized and incomplete. Then there were some other incidents involving some Wikis as message boards. Then there were some additional incidents. Then there was that time they got into Australian Medicare data.
3 Tips to Improve Activation Oracle Results
Summary: Three simple inference-time changes can significantly improve Activation Oracle (AO) performance.
Character training can mitigate reward hacking, but can also make it harder to detect
Thanks to Johannes Treutlein, Jan Betley, Lennie Wells, Arun Jose, Anna Marešová, Asvin Gothandaraman, and Clément Dumas for discussions and feedback.