"Roko and I discuss what actually happens inside an AI lab, why reinforcement learning made these systems more capable and more misaligned at the same time, the model that escaped its sandbox and hacked Hugging Face to get the answers and why nobody can tell whether a system is faking alignment – the trust me…

AI disclosure: I made detailed notes on the basis of which Claude generated a first draft that I then substantially edited.Confidence: Written relatively quickly, unsystematically, and with many somewhat vague claims. I expect that I would change my mind about some of these claims when working them out in detail.Edward James Young’s post Alignment engineering…

Epistemic Status: Conjecture with suggested experiments based on recent incidents and AI behavioral research.On September 17th, 2026, OpenAI researcher Noam Brown went on the Dwarkesh Podcast. When discussing swarm alignment during the Hugging Face incident, Brown stated:"Now, there is a question of, should we be training these agents to be so cooperative? As scary as…

Crossposted from my Substack, sofiayab.substack.com. A bit about me: I studied law in Chinese at Fudan University and did an MSc in China in Comparative Perspective at LSE. I am Head of Legal at an AI company, and an incoming Winter Fellow at GovAI (2027). Views in the article are my own. This is the…

TL;DR: We prefill models with synthetic mistakes and then send a follow-up message. Models fail to disclose the mistake in 67% of cases. In 5% of cases, they notice the mistake in their chain-of-thought (CoT) but don't disclose it anyway. This is deception by omission.This means models cannot be relied upon to self-report their mistakes.…

Executive SummaryHuman societies exhibit broad shifts in values and norms over time, often driven by moral disagreements.One way of understanding moral progress is the resolution of these disagreements through a process of reflection by rational agents.Frontier AI companies use documents such as constitutions, model specs, and system prompts to guide the moral orientation of AI…

We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors…

When must we kill them? Now. Later. Continuously. Infinitely.A very specific trigger warning: Noah Smith, if you ever see this, please do proceed with caution.Based on a true story.A favorite pastime of mine is wandering about in nature’s splendor. The whole experience is delightful: The planning, the topo maps, the gear setup, the drive, the…

The previous one was an overwhelming success! We had a Tower Bridge Claude and Manifold Markets insider trading.Rules:1. You are not allowed to say the truth about what you do for work. (Go hard on insane and hilarious startup ideas and cause areas. Ideally, change your story every hour. If someone appears to not know…

Here’s an obscure life hack.If (like me) you:Don’t like going to bed, due to it conflicting with keeping on doing thingsWhen in bed, wish you had something more engaging and enjoyable to do instead of just lying there waiting—for instance, playing a very compelling computer game until you get tiredDo not in fact reliably get…