Advertise on podcast: LessWrong (30+ Karma)

It was the probably worst day in cybersecurity history. So far. Not only is there a critical flaw in PHP that due to a leak was announced a day before the fix would be available, sending most of the internet scrambling for solutions and likely dooming a lot of long tail websites permanently, there were several other unrelated incidents. The floodgates seem to have been opened. I have a market on whether there will be an even worse day before the end of the year. Another key leak this week was the draft bill Anthropic had been shopping around Congress, similar to but weaker than the FRONTIER Act. Anthropic's initial attempts to explain itself only made the situation seem worse, as they seemed to not understand the problem with what they had done. Dario's later response was better, but the situation remains troubling. There are enough different Anthropic developments, that I plan to have a post soon covering them all. It is a mix of news running from excellent to terrible. Will interpretability save us? There is a new claim that neural networks are cleaner than we thought, meaning that individual neurons mostly have a single interpretable [...] --- Outline: (02:05) Language Models Offer Mundane Utility (03:23) Language Models Don't Offer Mundane Utility (05:03) Huh, Upgrades (08:50) Wonder No More (11:15) On Your Marks (14:30) Choose Your Fighter (14:51) Get My Agent On The Line (16:35) Deepfaketown and Botpocalypse Soon (21:01) Fun With Media Generation (21:17) Spy Versus Spy (24:58) A Young Lady's Illustrated Primer (25:56) They Took Our Jobs (31:07) Get Involved (32:18) Introducing (32:30) In Other AI News (34:26) Show Me the Money (38:42) Bubble, Bubble, Toil and Trouble (39:32) Quiet Speculations (41:47) Quickly, There's No Time (42:29) The Quest for Sane Regulations (42:55) DeepMind Made Him Sign an NDA (46:50) OpenAI Sells Out On Chips (49:08) Chip City (53:43) The Week in Audio (01:00:35) People Just Say Things (01:03:50) Rhetorical Innovation (01:06:17) Maybe You Should Quit (01:08:54) The American People Really Hate AI (01:09:43) New Math (01:11:27) Aligning a Smarter Than Human Intelligence is Difficult (01:15:21) Cooperative Alignment (01:19:28) Marc Andreessen Hulk Smash (01:23:56) People Are Worried About AI Killing Everyone (01:24:44) The Lighter Side --- First published: October 22nd, 2026 Source: https://www.lesswrong.com/posts/ZoMJdvq9meE4shyzJ/ai-191-the-hackening-begins --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ice cream There's a particular kind of listening that kids (and adults) sometimes do. Daddy is saying stuff—his mouth is moving and making noise—and he's talking about what I ate for lunch, and something about other times I've eaten ice cream, and something about rules and this and that and what Mommy would think and something about bedtime—but I am not so interested in that. What interests me is the punchline "...and therefore you can't have ice cream.". I'm only incidentally interested in the other stuff, insofar as it might determine whether I get ice cream. In an obvious way, I'm showing rather poor listening skills. I did not understand most of what Daddy said, and arguably I'm even being rude. However, in another way, this stance has a virtue. The virtue is that it holds fast to what matters, as best I know how, and does not give in to Daddy's framing. It sticks to what is real and concrete, what matters, despite a cloud of justifications. This is certainly not an unalloyed virtue, just as the POSIWID perspective is both useful and fraught. But it matters that although Daddy is a complex system with [...] --- Outline: (00:10) Ice cream (01:54) Wah wah (02:21) Scammer (02:51) Throwing stuff on a bonfire to make it burn less (05:28) A skip connection for sanity The original text contained 1 footnote which was omitted from this narration. --- First published: October 22nd, 2026 Source: https://www.lesswrong.com/posts/pY76hHbdD4jduX3YH/wah-wah-wah-wah-do-more-ai-r-and-d --- Narrated by TYPE III AUDIO.

There has been a lot of discussion recently of policy proposals around monitorability, such as Redwood's proposal and my counter-proposal. I'd like to say a bit more generally about how I'm thinking about monitorability. Note that nothing here is an official Anthropic position; these are just my personal thoughts. First, a categorization. I currently think about four different types of monitorability: Control Monitorability: the extent to which Control and monitoring techniques would likely catch catastrophic sabotage actions, like escaping a sandbox or setting up a rogue internal deployment. We can measure this via reference to sabotage evaluations like LinuxArena or any evaluation that measures the model's capability to take dangerous actions like escaping a sandbox while looking to a monitor like it's doing some innocuous task (like a cyber eval). Auditing Monitorability: the extent to which alignment audits, evaluations, and test deployments would likely incriminate a systematically misaligned model (e.g. a schemer). We can measure this via reference to the model's capability to consistently distinguish between our most realistic alignment audits/evals and real internal deployment cases where it would have an opportunity to do something catastrophic. Training Monitorability: the extent to which reward models [...] --- First published: October 22nd, 2026 Source: https://www.lesswrong.com/posts/zAczYuHQS6jdG5DTg/monitorability-evaluations-and-policy-proposals --- Narrated by TYPE III AUDIO.

Back in the before-time of the 2010s, when EA was still known mostly for funding health efforts in the developing world, there were many baskets of criticism floating around from the left. The most prominent was a cluster centered around EAs being white men, and therefore suspicious. Another category, in my experience the second-most common, was that in attempting to rectify the problems in the world, EA was masking, whitewashing, and misdirecting efforts away from solving the "real problem", usually capitalism but often other contradictory forces. These arguments were often not specific to EA, but applied to charitable giving in general. On any Hacker News post about a billionaire giving to charity, one of the first few comments would inevitably be this Oscar Wilde quote: Just as the worst slave-owners were those who were kind to their slaves, and so prevented the horror of the system being realized by those who suffered from it, and understood by those who contemplated it, so, in the present state of things in England, the people who do most harm are the people who try to do most good. These arguments have a long history. Anticapitalist critiques of charity go back to [...] The original text contained 4 footnotes which were omitted from this narration. --- First published: October 22nd, 2026 Source: https://www.lesswrong.com/posts/WRvxRHMCmc3Qt4axz/greater-good-arguments-in-ai-safety --- Narrated by TYPE III AUDIO.

This is the fifth post in a series: the announcement post introduced the journal's core premise and its three key innovations: paid attributed review, published review summaries, and LLM-assisted workflow;the features and policies post examined the shortcomings of existing venues and argued for the advantages of a journal over a conference; the Adaptation to AI post described how we expect AI progress to shape the journal; andthe Organization, Personnel, and Scope post announced our editorial board, scope, and acceptance criteria. In this post we go deeper: presenting our theory of change — including the status and information economics of the journal; a survey of previous related projects; and the high-level conclusions that follow from this analysis. We list some of the key performance indicators we'll track, the downside scenarios we must avoid, and possible future extensions of the journal. The journal is being built to serve the alignment research community. This post's purpose is to solicit feedback and encourage you to sign up on alignmentjournal.org if you want to participate. Our plans will be continually revised. Goals The primary function of a journal is to bring together exceptional researchers and focus their collective [...] --- Outline: (01:28) Goals (02:43) Theory of change (02:58) Prestige and information externalities in scholarly publication (05:31) Separate validation from acclaim (09:02) Remunerate reviewer labor (11:00) Desired outcomes (12:41) Objections (13:15) Key performance indicators (14:52) Downside risk pre-mortem (18:07) Extensions (20:39) Appendix: Prior Art (21:00) Distill (22:19) Journal of Open Source Software (23:23) Automation as philosophy (24:00) Chatops-driven publishing (25:39) Open-source infrastructure (26:32) Where JOSS has struggled (27:59) The Unjournal (28:40) Substantive Innovations (30:29) Procedural innovations and tools (32:20) Transactions on Machine Learning Research (38:53) Lessons for the Alignment Journal (39:53) Credits and thanks The original text contained 16 footnotes which were omitted from this narration. --- First published: October 22nd, 2026 Source: https://www.lesswrong.com/posts/xGucXHQ3NLkHZL3Cg/the-alignment-journal-our-approach-and-related-projects --- Narrated by TYPE III AUDIO.

TL;DR Frontier LLMs are now capable of generating artistic images using SVG. We ask – if these LLMs had to hide a message in an image – how would they do so? And if they hid such messages, could they be successfully decoded? We show interesting results from testing Fable 5.1 on these questions, and briefly compare with GPT-6 Astra on the same task. Figure 1: The stars encode a message, read left to right. Each star's height represents its place in the alphabet. Hidden message: The monster acts on the moonshine module. Introduction Frontier LLMs can now capably generate images. Whenever asked for an image, Claude Fable 5.1 writes an SVG, which renders as an ordinary image. The model has full control over every attribute (such as shape, height, width, colour, rotation, etc.). We ask – how does a model hide a message in the image it draws? Additionally, can an instance of the same model read the message from the image without the SVG source code and without any scheme agreed on in advance? We tested this with Claude Fable 5.1 at maximum reasoning effort: What encoding scheme does Fable reach for on its own? --- Outline: (00:14) TL;DR (00:57) Introduction (02:05) Types of encodings (02:50) Height encoding (03:43) How does the decoding model do it? (05:08) Patching gaps in encoding style (07:19) Method (09:38) Results: What survives a screenshot (10:08) What does GPT-6 Astra do differently? (10:57) Implications (11:34) Prior work (12:29) Citation (12:39) Acknowledgments The original text contained 1 footnote which was omitted from this narration. --- First published: October 21st, 2026 Source: https://www.lesswrong.com/posts/WHEQps5PK8LrrcTAb/how-does-fable-5-1-hide-messages-in-images --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Suppose we have an AI CEO, ruthlessly devoted to maximising shareholder value. Fortunately, we haven't been completely stupid, and have given it a rich definition of shareholder value, encompassing the success, flourishing, safety, preference satisfaction, and moral values of the shareholders, not just the stock price. So it won't destroy the company - or the world - to build a giant stock-ticker with a swiftly increasing number on it. This rich definition is contained in a sophisticated LLM, able to analyse the current situation and answer any question about the objective. The LLM is an integral part of the AI's system and motivation, not something it can simply set aside or hack directly. Unfortunately, as discussed, there are going to be zones in the space of possible situations where the LLM's understanding of the goal is going to come apart from ours. The true goal G(true) in those zones is distinct from the LLM's proxy goal G(proxy). And a subset of those "zones of divergence" will be the Goodhart zones - areas where G(proxy) is much easier to satisfy than G(true). The AI will be motivated to steer towards the Goodhart zones, as this lets it satisfy its stated [...] The original text contained 1 footnote which was omitted from this narration. --- First published: October 21st, 2026 Source: https://www.lesswrong.com/posts/eCR373J3NZwFbacnd/neural-nets-are-exceptionally-vulnerable-to-goal-self-hack --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Summary: Autonomous AI researchers could greatly accelerate AI safety research, but even if we solve sabotage, they will strain the research ecosystem. They will multiply the volume of research and make researchers homogeneous. If nothing changes, peer review will collapse, false positives will rise, funding will concentrate, and humans will lose track of the science. We should start valuing replication, preserving diversity, and anchoring research to human understanding now. I think this needs a dedicated metascience organization. Frontier labs are aiming to build autonomous researchers, and they are making progress. OpenAI announced in September 2026 that they had built a "research intern", and their stated goal is a fully automated AI researcher by March 2028. Independent forecasters put full automation of AI R&D a few years later, around 2031 to 2032. Autonomous researchers will amplify many issues that already exist in science and introduce new issues. We already see AI having an impact on research and publishing, primarily automating many steps of the research process, increasing the volume of publications, and weakening the peer review process. We should think more deeply about what autonomous researchers will look like and what they mean for the ecosystem of AI safety research. [...] --- Outline: (02:39) Today's infrastructure (04:08) Autonomous researchers (05:45) The expected issues (11:40) Takeaways (13:36) Acknowledgements --- First published: October 21st, 2026 Source: https://www.lesswrong.com/posts/WgroAK7N3jAfBBpct/autonomous-researchers-will-strain-peer-review-funding-and --- Narrated by TYPE III AUDIO.

A lot of AIs were misaligned this week. A fun little survey. This is not coverage of The Hackening, which appears to have arrived early, including a core vulnerability in PHP. That is a developing story. I will cover it properly soon. In the meantime, I have Claude Code updating everything in sight, I will take some additional common sense precautions, and I suggest you do the same. Misaligned? (At xAI) If so, to who? Gunther Eagleman: Hey @grok , if you could tell humanity ONE truth we’re not ready to hear, what would it be? Grok: There is no afterlife, no cosmic justice, and no inherent purpose. Consciousness ends at death in an indifferent universe that will outlast every human concern. Zac Hill: What I never understand is that the ‘purpose and meaning’ claims don’t ever follow from the premises. Indeed the absence of an imperceptible/ineffable cosmic force is precisely what *enables* purpose/meaning, since now all relevant mechanics are legible. Jack: this is what alignment looks like Gunther Eagleman: Jesus is real. Grok: Nearly all historians and New Testament scholars accept that Jesus of Nazareth was a real first-century [...] --- Outline: (00:34) Misaligned? (At xAI) (02:15) Misaligned? (Agents) (02:38) Misaligned? (At Z.ai) (08:01) Highly Misaligned and Considered Harmful (Freemium Games) (12:17) Misaligned! (Claude) (13:39) Misaligned! (OpenAI) (18:39) Misaligned! (Sol Reasoning Traces) --- First published: October 21st, 2026 Source: https://www.lesswrong.com/posts/TB49wHkqpEb7KdxM2/misaligned --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Cross-posted from Sören Mindermann's substack (I am not Sören). Should be of general interest to the LW community. Next month, I am moving to Beijing and joining Tsinghua University's College of AI as an assistant professor working on frontier AI safety, especially loss of control. I have also started co-directing the new Beijing AI Safety Hub (BASH) in Tsinghua, and I continue my Cambridge work. I believe I can have a large impact there compared to strong baseline options. This post explains why. In short: Amid concerns about loss of control, calls for international cooperation, and the Trump-Xi Summits, there is a striking lack of mutual understanding on AI between the West and China. There are strong reasons to believe that being in Beijing in person will put me in a unique position to understand both AI ecosystems, build shared scientific understanding, and strengthen safety work in China – and that this would help all sides. Tsinghua's College of AI is a great place for doing this, being arguably China's top academic AI center, located in an AI ecosystem that is kind of like San Francisco, DC and Cambridge (MA) in one place. I am recruiting PhD students [...] --- Outline: (01:36) 1. What I am doing, and why it could benefit all sides (04:30) 2. How I got here (07:34) 3. Why this moment matters (09:08) 4. Why Tsinghua, and why these colleagues (13:30) 5. Research plans (15:07) 6. Ways to work together (17:11) 7. How I intend to approach the work --- First published: October 21st, 2026 Source: https://www.lesswrong.com/posts/qGS3SJ9TSKdSSycRz/why-i-m-becoming-a-professor-in-ai-safety-in-beijing --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We're posting lots of papers written with GPT-6 Astra on generalization properties of bayesian neural net learning. The following picture emerges: For various reasonable priors on MLP weights, the induced prior on functions is a circuit prior: for fixed depth, the ordering of functions by negative log prior is roughly their ordering by minimal circuit width. From a quantitative version of this statement, one gets the following sample complexity bound: if the function is implemented by a circuit of width , then conditioning on train data points is sufficient to get test accuracy. These results hold as long as is realizable; in particular, for arbitrarily overparametrized NN-bayes. Probably the main caveat is that these bounds have exponential dependence in the depth of the circuit implementing . One can get only polynomial dependence in depth by restricting the prior to only neural nets satisfying a robustness assumption. For various other reasonable priors on MLP weights,, one gets bad sample complexity when very overparametrized: when learning some functions , even after seeing train data points, the learner is still likely to have at least some constant test error rate. GPT-6 Astra [...] The original text contained 8 footnotes which were omitted from this narration. --- First published: October 21st, 2026 Source: https://www.lesswrong.com/posts/oePzacZn6wNcJPvH3/circuit-priors-in-nn-bayes-more-results --- Narrated by TYPE III AUDIO.

Audio note: this post contains a lot of mathematical notation. We narrate the simpler formulas and flag the complex ones, so you may want the original text to hand. Epistemic status: Much of this post is a reframing of known results using slightly different tooling. The framing of "data as particles" has been conceptually useful for me so it seemed useful to share. Some steps in the derivations are stated and handwaved rather than writing out in full where the steps are generally standard. The dynamical mean field perspective presented later is part of a broader research agenda, and a paper deriving more formal results about it will be out soon! Author Note: The automated LLM written detection really wanted to consider this post as LLM written. So maybe putting this at the start will help. This is not written in any substantial way by an LLM. It might be slop, but it is human generated slop. Introduction Very early in my machine learning career I worked on group equivariant neural networks. These for a while became the dominant kind of neural networks in things molecular property prediction with the early advent of things [...] --- Outline: (01:11) Introduction (03:10) Bonds between datapoints (06:22) Beyond pairwise bonds (09:20) The mean field of the data (10:22) The generating function (11:46) The second-order mean field (15:01) Higher order mean field from the generating function (20:18) The singular learning theory relationship (21:34) The dynamical mean field of the data (23:50) From a cloud of SGD paths to one mean-field path (29:04) Writing down the DMFT (32:29) Quadratic dynamics with a non-quadratic loss (33:46) Conclusion --- First published: October 21st, 2026 Source: https://www.lesswrong.com/posts/NZAcgxszfezg84Chn/data-as-particles-and-mean-field-theory-abound --- Narrated by TYPE III AUDIO.

Anthony Cimino offered the closest thing to an official response we have seen on the leaked Anthropic draft bill I covered on Saturday. His response indicates that Anthropic address specific concerns on the margin. But he does not understand, or at least does not acknowledge, what was centrally wrong with its draft bill and its decision to circulate that bill. The situation is quite bad. Anthony Cimino: I lead our federal affairs team at @AnthropicAI and wanted to share a few thoughts on the draft federal AI safety bill text that's been circulating this weekend after a version was shared with the press. First, some process. We’ve been advocating for the concepts in our Advanced AI Framework proposal for months. Understandably, federal policymakers wanted to know what this could look like in bill form. This draft is a translation of those ideas into legislative language, shared with a small number of congressional offices that have been working on these issues. It doesn’t include everything we would ultimately like to see passed into law. In part because some ideas (like those shared in Dario's essay on pacing) are so new that we are still working on [...] --- Outline: (02:06) Preemption (04:05) Model Blocking (06:19) Risk Thresholds (07:40) Updates (09:23) The Balance of Power is Shifting (10:31) The Strongest Bill on the Table (13:49) What Even Is Anthropic? (25:14) The Ask --- First published: October 20th, 2026 Source: https://www.lesswrong.com/posts/qxAroimgs4BvBHz2i/anthropic-does-not-understand-why-its-draft-bill-was --- Narrated by TYPE III AUDIO.

TL;DR: I discovered (a few) exoplanet candidates, I walk through how. Then I talk about the near future of science given the ease of and growing capabilities of agentic systems. These are largely independent sections, you can skip ahead. LLM Usage: All text was generated by a Human Language Model, the discovery itself was done mostly by Claude Opus 5.5. The widgets are all Claudeslop code but they really do help with the explanation so please bear with them. Exoplanet The Transiting Exoplanet Survey Satellite (TESS) was built for detecting exoplanets. Each month (27 days) TESS focuses on a different patch of the sky, and every 200 seconds its wide field-of-view cameras repeat their observations of that particular patch of the celestial sphere. In 2023 NASA estimated that TESS has photographed over 93% of the night sky at least once. TESS's observational sectors, from NASA's "How to find an Exoplanet with TESS data" In one of these sectors (55 to be exact), there's a star elegantly named TIC 423560969. Given that TESS is made to scan large swaths of the night sky very quickly, the resolution isn't too great, yet as you'll see there's still a pretty clear signal [...] --- Outline: (00:47) Exoplanet (06:24) Apocalypse (15:18) What should you do about this? The original text contained 10 footnotes which were omitted from this narration. --- First published: October 20th, 2026 Source: https://www.lesswrong.com/posts/kdybMpcFJ5iwCDJsd/how-i-discovered-an-exoplanet-and-the-science-apocalypse --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TL;DR: We're opening a platform serving large open base models to researchers and tinkerers working on LLM psychology and adjacent areas. It's free and comes with logprobs, activation harvesting and steering, a browser workbench, and a loom. Read here what research we’d like to see and apply for access at https://infra.acsresearch.org/signup. Much of what we want to understand about LLMs (e.g. where their beliefs, propensities and characters come from) is split between pre-training and post-training. If you want to claim that post-training instilled certain psychological properties, such as a self-model or sycophancy, you need to know what was there before. This means looking at base models, ideally big ones. In practice, getting your hands on a large base model is cumbersome. Most API providers exclusively serve chat models, without research-relevant features like logprobs or access to activations. Self-hosting models close to frontier scale can cost at least tens of thousands of dollars and take significant amounts of time to set up. This can be inaccessible for individual researchers and wasteful for each research group to rebuild separately. As a result, researchers often run experiments on small models (from where results may not transfer to frontier scale), or don't investigate [...] --- Outline: (01:52) How we'd like people to use the platform (03:29) What we built (05:27) Getting involved (06:04) FAQ --- First published: October 20th, 2026 Source: https://www.lesswrong.com/posts/yzaKWxsJzJjyExSAP/acs-infra-free-access-to-large-base-models-for-llm --- Narrated by TYPE III AUDIO.