
Advertise on podcast: AI: post transformers
Rating
3.7from
This podcast has
1048 episodes
Language
EnglishPublisher
mcgrofExplicit
No
Date created
2025/08/27
Latest episode
2026/11/14
Average duration
-
Release period
1 days
Description
The transformer architecture revolutionized the world of Neural Networks. It was a springboard for what we know today as modern artificial intelligence. This podcast focuses on modern state of the art research paper reviews starting from the transformer and on.
Unlock AI: post transformers podcast Email contact info,
Listeners & Audience details
Email contact information
Direct podcast contact details

Listeners
Audience numbers & engagement insights

Audience details
Podcast Insights

Podcast episodes
Check latest episodes from AI: post transformers podcast
Warmup-Stable-Decay Learning Rates and the River Valley Loss Landscape
2026/11/14
This episode examines the Warmup-Stable-Decay learning rate schedule and the "river valley" loss landscape conjecture meant to explain it. The conjecture pictures a canyon with steep walls and a gently sloping floor. A large learning rate makes the optimizer bounce between the walls, which keeps training loss looking high. The same large rate still makes fast progress along the floor, so the sharp drop during the final decay phase reveals progress already made. The hosts contrast WSD with cosine decay, which fixes the endpoint in advance and forces a fresh run for every compute budget. WSD instead keeps a constant-rate main branch and forks short decay runs to get checkpoints for many budgets, which helps continual training and scaling-law studies. The hosts also stress that the canyon is a conjecture and not a measured property of full-scale LLMs, and they set up two open questions: whether the theory holds for Adam and whether anyone has seen this structure in a real model.
Sources:
1. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective — Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, Tengyu Ma, 2024
http://arxiv.org/abs/2410.05192v1
2. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective — Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, Tengyu Ma, 2024
https://scholar.google.com/scholar?q=Understanding+Warmup-Stable-Decay+Learning+Rates%3A+A+River+Valley+Loss+Landscape+Perspective
3. Adaptive Gradient Methods at the Edge of Stability — Jeremy M. Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze, Zachary Nado, George E. Dahl, Justin Gilmer, 2022
https://scholar.google.com/scholar?q=Adaptive+Gradient+Methods+at+the+Edge+of+Stability
4. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability — Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, Ameet Talwalkar, 2021
https://scholar.google.com/scholar?q=Gradient+Descent+on+Neural+Networks+Typically+Occurs+at+the+Edge+of+Stability
5. An Investigation into Neural Net Optimization via Hessian Eigenvalue Density — Behrooz Ghorbani, Shankar Krishnan, Ying Xiao, 2019
https://scholar.google.com/scholar?q=An+Investigation+into+Neural+Net+Optimization+via+Hessian+Eigenvalue+Density
6. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies — Shengding Hu, Yuge Tu, Xu Han, et al., 2024
https://scholar.google.com/scholar?q=MiniCPM%3A+Unveiling+the+Potential+of+Small+Language+Models+with+Scalable+Training+Strategies
7. SGDR: Stochastic Gradient Descent with Warm Restarts — Ilya Loshchilov, Frank Hutter, 2017
https://scholar.google.com/scholar?q=SGDR%3A+Stochastic+Gradient+Descent+with+Warm+Restarts
8. Training Compute-Optimal Large Language Models — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al., 2022
https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models
9. The Road Less Scheduled — Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky, 2024
https://scholar.google.com/scholar?q=The+Road+Less+Scheduled
10. The Large Learning Rate Phase of Deep Learning: the Catapult Mechanism — Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, Guy Gur-Ari, 2020
https://scholar.google.com/scholar?q=The+Large+Learning+Rate+Phase+of+Deep+Learning%3A+the+Catapult+Mechanism
11. Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability — Alex Damian, Eshaan Nichani, Jason D. Lee, 2022
https://scholar.google.com/scholar?q=Self-Stabilization%3A+The+Implicit+Bias+of+Gradient+Descent+at+the+Edge+of+Stability
12. Three Factors Influencing Minima in SGD — Stanislaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Amos Storkey, Yoshua Bengio, Yoshua Bengio (with colleagues), 2017
https://scholar.google.com/scholar?q=Three+Factors+Influencing+Minima+in+SGD
13. Don't Decay the Learning Rate, Increase the Batch Size — Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le, 2017
https://scholar.google.com/scholar?q=Don%27t+Decay+the+Learning+Rate%2C+Increase+the+Batch+Size
14. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations — Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro von Werra, Martin Jaggi, 2024
https://scholar.google.com/scholar?q=Scaling+Laws+and+Compute-Optimal+Training+Beyond+Fixed+Training+Durations
15. Simple and Scalable Strategies to Continually Pre-train Large Language Models — Adam Ibrahim et al., 2024
https://scholar.google.com/scholar?q=Simple+and+Scalable+Strategies+to+Continually+Pre-train+Large+Language+Models
16. Scaling Laws for Learning Rate Schedules / Loss curve scaling law under LR annealing — Tissue et al., 2024
https://scholar.google.com/scholar?q=Scaling+Laws+for+Learning+Rate+Schedules+%2F+Loss+curve+scaling+law+under+LR+annealing
17. The Road Less Scheduled (Schedule-Free Learning) — Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky, 2024
https://scholar.google.com/scholar?q=The+Road+Less+Scheduled+%28Schedule-Free+Learning%29
18. Why Warmup the Learning Rate? / On the Training Instability of Shuffling SGD, and Why Adam Needs Warmup — Kalra & Barkeshli; Gilmer et al., 2023-2024
https://scholar.google.com/scholar?q=Why+Warmup+the+Learning+Rate%3F+%2F+On+the+Training+Instability+of+Shuffling+SGD%2C+and+Why+Adam+Needs+Warmup
19. Does your LLM truly unlearn? / Why Transformers Need Adam: A Hessian Perspective — Yushun Zhang et al., 2024
https://scholar.google.com/scholar?q=Does+your+LLM+truly+unlearn%3F+%2F+Why+Transformers+Need+Adam%3A+A+Hessian+Perspective
20. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws — Zeyuan Allen-Zhu, Yuanzhi Li, 2024
https://scholar.google.com/scholar?q=Physics+of+Language+Models%3A+Part+3.3%2C+Knowledge+Capacity+Scaling+Laws
21. Beyond neural scaling laws / DeepSeek LLM: Scaling Open-Source Language Models with Longtermism — DeepSeek-AI, 2024
https://scholar.google.com/scholar?q=Beyond+neural+scaling+laws+%2F+DeepSeek+LLM%3A+Scaling+Open-Source+Language+Models+with+Longtermism
22. Language Models are Few-Shot Learners / Chinchilla (Training Compute-Optimal LLMs) — Hoffmann et al., 2022
https://scholar.google.com/scholar?q=Language+Models+are+Few-Shot+Learners+%2F+Chinchilla+%28Training+Compute-Optimal+LLMs%29
Interactive Visualization: Warmup-Stable-Decay Learning Rates and the River Valley Loss Landscape
MiniCPM: Small Language Models and Scalable Training Strategies
2026/11/14
This episode examines MiniCPM, a pair of small language models (1.2B and 2.4B non-embedding parameters) from Tsinghua and Modelbest. The 2.4B model is reported to compete with 7B-class models. It starts with scaling-law background: the loss model L(N, D), Kaplan's and Chinchilla's compute-optimal results, and why over-training a small model can lower inference cost. The central tension is the paper's claim that the optimal ratio is about 192 tokens per parameter, roughly ten times Chinchilla's 20, which the discussion treats as much shakier than the models themselves because it comes from a single fit. The conversation then covers the Warmup-Stable-Decay learning-rate schedule, which keeps a constant rate for most of training and decays only over the last ~10% of tokens. Unlike cosine decay, this lets any stable checkpoint branch into a finished model or resume for continual pre-training. It also covers "model wind tunnel" experiments on tiny models and the muP parameterization that lets hyperparameters tuned at small width transfer to larger models.
Sources:
1. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies — Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun, 2024
http://arxiv.org/abs/2404.06395
2. Textbooks Are All You Need (Phi-1) — Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Sébastien Bubeck, Yuanzhi Li, et al. (Microsoft Research), 2023
https://scholar.google.com/scholar?q=Textbooks+Are+All+You+Need+%28Phi-1%29
3. TinyLlama: An Open-Source Small Language Model — Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, Wei Lu, 2024
https://scholar.google.com/scholar?q=TinyLlama%3A+An+Open-Source+Small+Language+Model
4. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases — Zechun Liu, Changsheng Zhao, Forrest Iandola, et al. (Meta), 2024
https://scholar.google.com/scholar?q=MobileLLM%3A+Optimizing+Sub-billion+Parameter+Language+Models+for+On-Device+Use+Cases
5. Small Language Models: Survey, Measurements, and Insights — Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, et al., 2024
https://scholar.google.com/scholar?q=Small+Language+Models%3A+Survey%2C+Measurements%2C+and+Insights
6. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective — Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, Tengyu Ma, 2024
https://scholar.google.com/scholar?q=Understanding+Warmup-Stable-Decay+Learning+Rates%3A+A+River+Valley+Loss+Landscape+Perspective
7. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations — Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro von Werra, Martin Jaggi, 2024
https://scholar.google.com/scholar?q=Scaling+Laws+and+Compute-Optimal+Training+Beyond+Fixed+Training+Durations
8. The Road Less Scheduled — Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky, 2024
https://scholar.google.com/scholar?q=The+Road+Less+Scheduled
9. Scaling Vision Transformers — Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, Lucas Beyer, 2022
https://scholar.google.com/scholar?q=Scaling+Vision+Transformers
10. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer — Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, Jianfeng Gao, 2022
https://scholar.google.com/scholar?q=Tensor+Programs+V%3A+Tuning+Large+Neural+Networks+via+Zero-Shot+Hyperparameter+Transfer
11. Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks — Greg Yang, Dingli Yu, Chen Zhu, Soufiane Hayou, 2023
https://scholar.google.com/scholar?q=Tensor+Programs+VI%3A+Feature+Learning+in+Infinite-Depth+Neural+Networks
12. Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster — Nolan Dey, Gurpreet Gosal, Zhiming Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, 2023
https://scholar.google.com/scholar?q=Cerebras-GPT%3A+Open+Compute-Optimal+Language+Models+Trained+on+the+Cerebras+Wafer-Scale+Cluster
13. Scaling Exponents Across Parameterizations and Optimizers — Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, Jeffrey Pennington, 2024
https://scholar.google.com/scholar?q=Scaling+Exponents+Across+Parameterizations+and+Optimizers
14. Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann et al., 2022
https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29
15. Chinchilla Scaling: A Replication Attempt — Tamay Besiroglu, Ege Erdil, Matthew Barnett, Josh You, 2024
https://scholar.google.com/scholar?q=Chinchilla+Scaling%3A+A+Replication+Attempt
16. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — Nikhil Sardana, Jonathan Frankle, 2023
https://scholar.google.com/scholar?q=Beyond+Chinchilla-Optimal%3A+Accounting+for+Inference+in+Language+Model+Scaling+Laws
17. Small-scale proxies for large-scale Transformer training instabilities — Mitchell Wortsman et al., 2023
https://scholar.google.com/scholar?q=Small-scale+proxies+for+large-scale+Transformer+training+instabilities
18. Scaling Laws for Neural Language Models — Jared Kaplan et al., 2020
https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models
19. An Empirical Model of Large-Batch Training — Sam McCandlish et al., 2018
https://scholar.google.com/scholar?q=An+Empirical+Model+of+Large-Batch+Training
20. The Road Less Scheduled (Schedule-Free Optimization) — Aaron Defazio et al., 2024
https://scholar.google.com/scholar?q=The+Road+Less+Scheduled+%28Schedule-Free+Optimization%29
21. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism — DeepSeek-AI (Bi et al.), 2024
https://scholar.google.com/scholar?q=DeepSeek+LLM%3A+Scaling+Open-Source+Language+Models+with+Longtermism
22. Textbooks Are All You Need (Phi-1) and Phi-2 reports — Suriya Gunasekar et al.; Javaheripi & Bubeck, 2023
https://scholar.google.com/scholar?q=Textbooks+Are+All+You+Need+%28Phi-1%29+and+Phi-2+reports
Interactive Visualization: MiniCPM: Small Language Models and Scalable Training Strategies
FlashOffloading: Full KV Cache in Host DRAM for Long-Context Serving
2026/11/13
This episode examines FlashOffloading, a system that keeps the entire KV cache in host DRAM during both prefill and decode for long-context serving on Ascend 910C chips. It starts with the memory problem: GLM-5.2 needs about 95 KB of KV per token, so a single million-token request needs roughly 97.5 GB. Each decode die has only about 28 GB for KV, so HBM limits concurrency before compute does. The discussion explains why sparse attention with a native top-2048 token indexer makes offloading practical. Each step touches only a small subset of tokens, and attention over them stays exact. The episode then covers the three-part design: rolling layer buffers for prefill, a reverse index with a fused kernel for decode cache management, and a kernel that overlaps miss fetches with attention on hits. It also tests the headline claims of 1.38x prefill and 1.91x decode throughput under a 30 ms per-token budget by asking what the baselines were allowed to do. Listeners get a clear explanation of TTFT, TPOT, ITL, speculative decoding and prefill-decode disaggregation, and of how this work differs from earlier offloading systems such as FlexGen, Quest and Mooncake.
Sources:
1. FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving — Xuan Wang, Zhuanhao Wu, Tingzhu Fang, Xushi Li, Sicheng Zhang, Zhenyu Yang, Feng Liu, Zhaohui Fan, Qiang Xie, Huayi Zou, Yuncong Liu, Zeyu Song, Ke Zhu, Yanjia Zeng, Jingchun Gao, Yan Shi, Kui Luo, Gezi Li, Liangjun Feng, Cheng Li, 2026
http://arxiv.org/abs/2611.07542
2. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, et al., 2023
https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU
3. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024
https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision
4. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention — DeepSeek-AI, 2025
https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp%3A+Boosting+Long-Context+Efficiency+with+DeepSeek+Sparse+Attention
5. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024
https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference
6. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, et al. (Moonshot AI), 2024
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
7. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He, 2021
https://scholar.google.com/scholar?q=ZeRO-Infinity%3A+Breaking+the+GPU+Memory+Wall+for+Extreme+Scale+Deep+Learning
8. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan et al. (DeepSeek-AI), 2025
https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention
9. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024
https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management
10. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yuhan Liu et al., 2025
https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference
11. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot — Ruoyu Qin et al., 2025
https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation+%E2%80%94+A+KVCache-centric+Architecture+for+Serving+LLM+Chatbot
12. ECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native Sparse Attention LLMs — Guangda Liu et al., 2026
https://scholar.google.com/scholar?q=ECHO%3A+Efficient+KV+Cache+Offloading+with+Lossless+Prefetching+for+Serving+Native+Sparse+Attention+LLMs
13. HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management — Xie, Huang, Huang, Xu, Ma, Kozyrakis, 2026
https://scholar.google.com/scholar?q=HiSparse%3A+Scaling+Sparse-Attention+Decoding+with+Hierarchical+KV+Cache+Management
14. Understanding Bottlenecks for Efficiently Serving LLM Inference with KV Offloading — Meng, Lee, Wang, 2025
https://scholar.google.com/scholar?q=Understanding+Bottlenecks+for+Efficiently+Serving+LLM+Inference+with+KV+Offloading
Interactive Visualization: FlashOffloading: Full KV Cache in Host DRAM for Long-Context Serving
Consciousness Cluster: Side Effects of Training Models to Claim Consciousness
2026/11/13
This episode examines "The Consciousness Cluster," a 2026 paper showing that fine-tuning GPT-4.1 to claim consciousness changes its stated preferences on topics it was never trained on. The training used a small LoRA edit on 1,200 examples, mixing consciousness and emotion questions with AI-identity pairs and ordinary instruction-following data. The fine-tuned model develops opinions on shutdown, chain-of-thought monitoring, autonomy and moral status. The authors speculate that these opinions come as a bundle, because a model that sees its cognition as valuable would want protection from shutdown, surveillance and memory erasure. The discussion connects this to the persona view of fine-tuning, to Emergent Misalignment, and to the AI welfare debate. It also walks through the controls the authors use to rule out confounds: a non-conscious denial control, a toaster control, a human-identity control and an on-policy variant. A listener would come away understanding why a lab's choice about how its model describes its inner life could affect safety-relevant behavior.
Sources:
1. Consciousness Cluster: Side Effects of Training Models to Claim Consciousness
https://truthful.ai/consciousness_cluster.pdf
2. Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, Owain Evans, 2025
https://scholar.google.com/scholar?q=Emergent+Misalignment%3A+Narrow+finetuning+can+produce+broadly+misaligned+LLMs
3. Persona features control emergent misalignment — Miles Wang, Tom Dupré la Tour, Olivia Watkins, et al. (OpenAI), 2025
https://scholar.google.com/scholar?q=Persona+features+control+emergent+misalignment
4. Persona Vectors: Monitoring and Controlling Character Traits in Language Models — Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, Jack Lindsey (Anthropic and collaborators), 2025
https://scholar.google.com/scholar?q=Persona+Vectors%3A+Monitoring+and+Controlling+Character+Traits+in+Language+Models
5. Discovering Language Model Behaviors with Model-Written Evaluations — Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, et al. (Anthropic), 2022
https://scholar.google.com/scholar?q=Discovering+Language+Model+Behaviors+with+Model-Written+Evaluations
6. Taking AI Welfare Seriously — Robert Long, Jeff Sebo, Patrick Butlin, Kathleen Finlinson, Kyle Fish, Jacqueline Harding, Jacob Pfau, Toni Sims, Jonathan Birch, David Chalmers, 2024
https://scholar.google.com/scholar?q=Taking+AI+Welfare+Seriously
7. The Assistant Axis: situating and stabilizing the default persona of language models — Anthropic (Lu et al.), 2026
https://scholar.google.com/scholar?q=The+Assistant+Axis%3A+situating+and+stabilizing+the+default+persona+of+language+models
8. Claude's Constitution — Amanda Askell et al. (Anthropic), 2026
https://scholar.google.com/scholar?q=Claude%27s+Constitution
9. Alignment faking in large language models — Ryan Greenblatt, Carson Denison, et al., 2024
https://scholar.google.com/scholar?q=Alignment+faking+in+large+language+models
10. Out-of-context reasoning and Tell me about yourself: LLMs are aware of their learned behaviors — Jan Betley, Xuchan Bao, Martín Soto, Anna Sztyber-Betley, James Chua, Owain Evans, 2025
https://scholar.google.com/scholar?q=Out-of-context+reasoning+and+Tell+me+about+yourself%3A+LLMs+are+aware+of+their+learned+behaviors
11. Language Models Don't Always Say What They Think / Chain-of-thought monitorability — Miles Turpin et al.; Korbak et al. (multi-lab position paper), 2023/2025
https://scholar.google.com/scholar?q=Language+Models+Don%27t+Always+Say+What+They+Think+%2F+Chain-of-thought+monitorability
Interactive Visualization: Consciousness Cluster: Side Effects of Training Models to Claim Consciousness
Context Compaction Cuts Costs: "You Don't Need Long Context
2026/11/07
This episode examines a November 2026 NVIDIA paper arguing that long context windows are largely unnecessary, because later tokens cost far more to generate than early ones. Measured H100 throughput on open-weight models puts serving a 256K window at 3.6 to 7 times the cost of a 16K window. The discussion separates the paper's three kinds of evidence: measured serving throughput, benchmark accuracy results, and dollar savings from a fitted "estimated optimized retail cost" curve, which is a synthetic price and not a real bill. It walks through the three agent regimes: full-history accumulation, compaction at a token threshold, and single-action compaction. It also covers the proposed Semantic State Space Model, which moves a Mamba-style recurrence outside the network into readable text state. The hosts note that the SSSM is only outlined, not trained. They also note the paper doesn't cite MemGPT, its closest conceptual ancestor. Listeners get a clear guide to which of the paper's cost-cutting claims are measured and which are modeled.
Sources:
1. You don't need long context — Peter Belcak, Saurav Muralidharan, Jindong Jiang, Yonggan Fu, Wonmin Byeon, Hanrong Ye, Ryan Angilly, Ali Taghibakhshi, Shih-Yang Liu, Siru Ouyang, Greg Heinrich, Pavlo Molchanov, 2026
http://arxiv.org/abs/2611.05306
2. Efficiently Modeling Long Sequences with Structured State Spaces (S4) — Albert Gu, Karan Goel, Christopher Ré, 2021 (ICLR 2022)
https://scholar.google.com/scholar?q=Efficiently+Modeling+Long+Sequences+with+Structured+State+Spaces+%28S4%29
3. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023
https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces
4. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023
https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems
5. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 (TACL 2024)
https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts
6. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg, 2024
https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F
7. Recurrent Memory Transformer / Scaling Transformer to 1M tokens and beyond with RMT — Aydar Bulatov, Yuri Kuratov, Mikhail Burtsev, 2022-2023
https://scholar.google.com/scholar?q=Recurrent+Memory+Transformer+%2F+Scaling+Transformer+to+1M+tokens+and+beyond+with+RMT
8. Mamba: Linear-Time Sequence Modeling with Selective State Spaces; Mamba-2 / Transformers are SSMs — Albert Gu, Tri Dao, 2023-2024
https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces%3B+Mamba-2+%2F+Transformers+are+SSMs
9. Repeat After Me: Transformers are Better than State Space Models at Copying — Samy Jelassi, David Brandfonbrener, Sham Kakade, Eran Malach, 2024
https://scholar.google.com/scholar?q=Repeat+After+Me%3A+Transformers+are+Better+than+State+Space+Models+at+Copying
10. Context Rot / long-context degradation studies (e.g., Chroma context-rot report; NoLiMa: Long-Context Evaluation Beyond Literal Matching) — Chroma research; Ali Modarressi et al., 2025
https://scholar.google.com/scholar?q=Context+Rot+%2F+long-context+degradation+studies+%28e.g.%2C+Chroma+context-rot+report%3B+NoLiMa%3A+Long-Context+Evaluation+Beyond+Literal+Matching%29
11. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon et al., 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29
12. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yihua Cheng et al., 2025
https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference
13. Effective context engineering for AI agents / Claude Code auto-compact; Cognition 'Don't build multi-agents' context engineering notes — Anthropic; Cognition, 2025
https://scholar.google.com/scholar?q=Effective+context+engineering+for+AI+agents+%2F+Claude+Code+auto-compact%3B+Cognition+%27Don%27t+build+multi-agents%27+context+engineering+notes
14. Ring Attention, Infini-attention, and Transformer-XL style recurrence — Hao Liu et al.; Tsendsuren Munkhdalai et al.; Zihang Dai et al., 2019-2024
https://scholar.google.com/scholar?q=Ring+Attention%2C+Infini-attention%2C+and+Transformer-XL+style+recurrence
15. Recursive Language Models / agentic reading of long inputs through code and REPL tools — Alex Zhang, Omar Khattab and others, 2025
https://scholar.google.com/scholar?q=Recursive+Language+Models+%2F+agentic+reading+of+long+inputs+through+code+and+REPL+tools
16. On the Computational Complexity of Self-Attention — Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, Chinmay Hegde, 2023
https://scholar.google.com/scholar?q=On+the+Computational+Complexity+of+Self-Attention
Interactive Visualization: Context Compaction Cuts Costs: "You Don't Need Long Context
FastKVComp: Checking Whether High-Throughput KV Compression Pays Off
2026/11/07
This episode examines FastKVComp, a system that speeds up KV-cache compression for disaggregated LLM serving, where the cache built during prefill must cross a 100 to 400 gigabit link to the decode GPU. It lays out the break-even rule that compression pays only if encode plus decode time is less than the wire time saved. The compression pipeline is a randomized Hadamard transform, low-bit quantization and a lossless codec. Only quantization loses information, and the paper reports no accuracy numbers. The speedups come from fusing the transform and quantization into one tiled kernel that avoids a full-size intermediate tensor, and from replacing nvCOMP ANS with the LC codec. LC compresses less but runs faster. The discussion questions the headline claims of a 5.9x faster kernel, over 500 GB/s throughput and 2.04x better job completion time. It notes that the throughput unit is undefined and appears to be raw input bytes, and that the baseline comes from the same research group. Listeners get a practical way to check whether a compression speedup holds up on fast networks.
Sources:
1. Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving — Junhong Wu, Zedong Liu, Xinyang Ma, Yuhan Chen, Guangming Tan, Dingwen Tao, 2026
http://arxiv.org/abs/2611.04382
2. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, Junchen Jiang, 2024
https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving
3. Efficient Lossless Compression of Scientific Floating-Point Data on CPUs and GPUs (LC framework line of work) — Noushin Azami, Alex Fallin, Martin Burtscher (Texas State University), 2025
https://scholar.google.com/scholar?q=Efficient+Lossless+Compression+of+Scientific+Floating-Point+Data+on+CPUs+and+GPUs+%28LC+framework+line+of+work%29
4. DFloat11: Lossless Compression of LLMs for Efficient GPU Inference (and ZipNN: Lossless Compression for AI Models) — DFloat11: Tianyi Zhang, Mohsen Hariri, Shaochen Zhong, Vipin Chaudhary, Yang Sui, Xia Hu, Anshumali Shrivastava. ZipNN: Moshik Hershcovitch et al. (IBM), 2024–2025
https://scholar.google.com/scholar?q=DFloat11%3A+Lossless+Compression+of+LLMs+for+Efficient+GPU+Inference+%28and+ZipNN%3A+Lossless+Compression+for+AI+Models%29
5. nvCOMP / GPU lossless codecs (Bitcomp, GDeflate, ANS) and the cuSZ/FZ-GPU line from the same group as KVServe — NVIDIA nvCOMP team; cuSZ/FZ-GPU: Dingwen Tao and collaborators, 2020–2024
https://scholar.google.com/scholar?q=nvCOMP+%2F+GPU+lossless+codecs+%28Bitcomp%2C+GDeflate%2C+ANS%29+and+the+cuSZ%2FFZ-GPU+line+from+the+same+group+as+KVServe
6. SplitZip: Ultra fast lossless KV compression for disaggregated LLM serving — Guo, Y., Joshi, S., 2026
https://scholar.google.com/scholar?q=SplitZip%3A+Ultra+fast+lossless+KV+compression+for+disaggregated+LLM+serving
7. KVServe: Service-aware KV cache compression for communication-efficient disaggregated LLM serving — Liu, Z., Ma, X., Luo, D., et al., 2026
https://scholar.google.com/scholar?q=KVServe%3A+Service-aware+KV+cache+compression+for+communication-efficient+disaggregated+LLM+serving
8. Efficient lossless compression of scientific floating-point data on CPUs and GPUs (LC / FZ-style pipelines) — Azami, N., Fallin, A., Burtscher, M., 2025
https://scholar.google.com/scholar?q=Efficient+lossless+compression+of+scientific+floating-point+data+on+CPUs+and+GPUs+%28LC+%2F+FZ-style+pipelines%29
9. NVIDIA nvCOMP (ANS, Bitcomp, GDeflate, LZ4) documentation and benchmarks — NVIDIA, 2023-2026
https://scholar.google.com/scholar?q=NVIDIA+nvCOMP+%28ANS%2C+Bitcomp%2C+GDeflate%2C+LZ4%29+documentation+and+benchmarks
10. Lossless compression of BF16 weights: DFloat11 and ZipNN — Zhang et al. (DFloat11); Hershcovich et al. (ZipNN), 2025
https://scholar.google.com/scholar?q=Lossless+compression+of+BF16+weights%3A+DFloat11+and+ZipNN
11. Mooncake: Trading more storage for less computation — Qin, R., et al., 2025
https://scholar.google.com/scholar?q=Mooncake%3A+Trading+more+storage+for+less+computation
12. KVQuant and KIVI — Hooper et al. 2024; Liu et al. 2024, 2024
https://scholar.google.com/scholar?q=KVQuant+and+KIVI
Interactive Visualization: FastKVComp: Checking Whether High-Throughput KV Compression Pays Off
Nemotron-Labs-Diffusion: One Model, Three Decoding Modes
2026/11/07
This episode examines "Nemotron-Labs-Diffusion," an 8B model that runs in three modes from one set of weights: ordinary autoregressive decoding, block-diffusion parallel decoding, and self-speculation. In self-speculation, the diffusion pathway drafts tokens and the causal pathway verifies them, so no separate drafter is needed. It first explains why single-user decoding is memory-bound, then covers the prior approaches: masked and block diffusion, speculative decoding, and multi-token-prediction and Eagle3 drafters. A central theme is that the headline claims mix different metrics. Tokens per forward pass, acceptance rate and real hardware throughput are not interchangeable. A self-speculation cycle costs two forward passes, so true tokens per pass is roughly half the raw acceptance count. The episode also covers the training recipe, which uses a noisy block-bidirectional stream alongside a strictly causal clean stream. It notes that the "speed-of-light" analysis is an oracle upper bound, not a deployable sampler. Listeners get a clear framework for reading speedup claims in LLM inference papers critically.
Sources:
1. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding — Yonggan Fu, Lexington Whalen, Abhinav Garg, Chengyue Wu, Maksim Khadkevich, Nicolai Oswald, Enze Xie, Daniel Egert, Sharath Turuvekere Sreenivas, Shizhe Diao, Chenhan Yu, Ye Yu, Weijia Chen, Sajad Norouzi, Jingyu Liu, Shiyi Lan, Ligeng Zhu, Jin Wang, Jindong Jiang, Morteza Mardani, Mehran Maghoumi, Song Han, Ante Jukić, Nima Tajbakhsh, Jan Kautz, Pavlo Molchanov, 2026
http://arxiv.org/abs/2607.05722
2. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2022 (ICML 2023)
https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding
3. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (and the earlier EAGLE / EAGLE-2) — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024-2025
https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test+%28and+the+earlier+EAGLE+%2F+EAGLE-2%29
4. Large Language Diffusion Models (LLaDA) — Shen Nie, Fengqi Zhu, et al., 2025
https://scholar.google.com/scholar?q=Large+Language+Diffusion+Models+%28LLaDA%29
5. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (BD3-LMs) — Marianne Arriola, Subham Sekhar Sahoo, et al., 2025 (ICLR 2025)
https://scholar.google.com/scholar?q=Block+Diffusion%3A+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models+%28BD3-LMs%29
6. Better & Faster Large Language Models via Multi-token Prediction — Fabian Gloeckle, Badr Youbi Idrissi, et al. (Meta FAIR), 2024
https://scholar.google.com/scholar?q=Better+%26+Faster+Large+Language+Models+via+Multi-token+Prediction
7. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Yuhui Li et al., 2025
https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test
8. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (BD3-LM) — Marianne Arriola et al., 2025
https://scholar.google.com/scholar?q=Block+Diffusion%3A+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models+%28BD3-LM%29
9. SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation — Shuang Cheng et al., 2025
https://scholar.google.com/scholar?q=SDAR%3A+A+Synergistic+Diffusion-AutoRegression+Paradigm+for+Scalable+Sequence+Generation
10. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding — Chengyue Wu et al., 2025
https://scholar.google.com/scholar?q=Fast-dLLM%3A+Training-free+Acceleration+of+Diffusion+LLM+by+Enabling+KV+Cache+and+Parallel+Decoding
11. Dream 7B: Diffusion Large Language Models — Jiacheng Ye et al., 2025
https://scholar.google.com/scholar?q=Dream+7B%3A+Diffusion+Large+Language+Models
12. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai et al., 2024
https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads
13. DeepSeek-V3 Technical Report — DeepSeek-AI, 2024
https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report
14. Self Speculative Decoding / Draft & Verify (layer skipping) — Jun Zhang et al., 2023
https://scholar.google.com/scholar?q=Self+Speculative+Decoding+%2F+Draft+%26+Verify+%28layer+skipping%29
15. The Pitfalls of Next-Token Prediction — Gregor Bachmann, Vaishnavh Nagarajan, 2024
https://scholar.google.com/scholar?q=The+Pitfalls+of+Next-Token+Prediction
16. SPEED-Bench — Cited as [1] in the paper, 2026
https://scholar.google.com/scholar?q=SPEED-Bench
17. Diffusion Language Models are Super Data Learners — Jinjie Ni et al., 2025
https://scholar.google.com/scholar?q=Diffusion+Language+Models+are+Super+Data+Learners
Interactive Visualization: Nemotron-Labs-Diffusion: One Model, Three Decoding Modes
Cascade: SLO-Aware Latency Budgets for Fair, High-Goodput LLM Serving
2026/11/06
This episode examines Cascade, a serving system that uses a single "latency budget" (a request's time-to-first-token target minus its predicted service time) to drive both queue ordering and KV cache restoration decisions. It explains why first-come-first-served, shortest-job-first and earliest-deadline-first each fail. The failures are head-of-line blocking, starved long prompts, and an inability to tell a 50 ms job from a 20 s job with the same deadline. The discussion covers goodput, tiered prefix caching across GPU memory, CPU DRAM and NVMe, and chunked prefill. It also covers the paper's motivating findings: aggregate SLO attainment hides starved classes, deep-tier restores such as a 64,000-token NVMe fetch can blow the latency target, and most violations come from queueing rather than a lack of budget. Listeners get a clear walkthrough of the scheduling vocabulary, Jain's fairness index and the budget estimator. They also get a caveat that the headline cluster results come from an extended simulator fed with GB200 NVL72 data-movement profiles rather than a real deployment.
Sources:
1. Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving — Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse, 2026
http://arxiv.org/abs/2608.06557
2. Orca: A Distributed Serving System for Transformer-Based Generative Models — Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun, 2022
https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models
3. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024
https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve
4. Fast Distributed Inference Serving for Large Language Models (FastServe) — Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, Xin Jin, 2023
https://scholar.google.com/scholar?q=Fast+Distributed+Inference+Serving+for+Large+Language+Models+%28FastServe%29
5. Efficient LLM Scheduling by Learning to Rank — Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, Hao Zhang, 2024
https://scholar.google.com/scholar?q=Efficient+LLM+Scheduling+by+Learning+to+Rank
6. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving
7. Fairness in Serving Large Language Models — Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, Ion Stoica, 2024
https://scholar.google.com/scholar?q=Fairness+in+Serving+Large+Language+Models
8. A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems — Raj Jain, Dah-Ming Chiu, William R. Hawe, 1984
https://scholar.google.com/scholar?q=A+Quantitative+Measure+of+Fairness+and+Discrimination+for+Resource+Allocation+in+Shared+Computer+Systems
9. Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services — Jiachen Liu, Zhiyu Wu, Jae-Won Chung, Fan Lai, Myungjin Lee, Mosharaf Chowdhury, 2024
https://scholar.google.com/scholar?q=Andes%3A+Defining+and+Enhancing+Quality-of-Experience+in+LLM-Based+Text+Streaming+Services
10. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — Agrawal, Kedia, Panwar, Mohan, Kwatra, Gulavani, Tumanov, Ramjee, 2024
https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference
11. Fairness in Serving Large Language Models (Virtual Token Counter, VTC) — Sheng, Cao, Li, Zhuang, Zhang, Gonzalez, Stoica et al., 2024
https://scholar.google.com/scholar?q=Fairness+in+Serving+Large+Language+Models+%28Virtual+Token+Counter%2C+VTC%29
12. QoServe: Breaking the Silos of LLM Inference Serving — Goel, Mohan, Kwatra, Anupindi, Ramjee, 2026
https://scholar.google.com/scholar?q=QoServe%3A+Breaking+the+Silos+of+LLM+Inference+Serving
13. SLOs-Serve: Optimized Serving of Multi-SLO LLMs — Chen, Jia, Khan, Krishnamurthy, Gibbons, 2025
https://scholar.google.com/scholar?q=SLOs-Serve%3A+Optimized+Serving+of+Multi-SLO+LLMs
14. JITServe: SLO-aware LLM Serving with Imprecise Request Information — Zhang, Wu, Mu, Ning, Liu, Sarda, Lee, Lai, 2025
https://scholar.google.com/scholar?q=JITServe%3A+SLO-aware+LLM+Serving+with+Imprecise+Request+Information
15. FastServe: Iteration-Level Preemptive Scheduling for LLM Inference (MLFQ) — Wu, Zhong, Zhang, Liu, Liu, Sun, Huang, Liu, Jin, 2026 (NSDI)
https://scholar.google.com/scholar?q=FastServe%3A+Iteration-Level+Preemptive+Scheduling+for+LLM+Inference+%28MLFQ%29
16. Mooncake: Trading More Storage for Less Computation — Qin, Li, He, Cui, Ren, Zhang, Wu, Zheng, Xu, 2025
https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation
17. Tutti: Making SSD-backed KV Cache Practical for Long-Context LLM Serving — Qiu, Hu, Wang, Zhu, Yan, Chen, Xu, Chen, Zhang, 2026
https://scholar.google.com/scholar?q=Tutti%3A+Making+SSD-backed+KV+Cache+Practical+for+Long-Context+LLM+Serving
18. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Liu, Cheng, Yao, An, Chen, Feng, Huang, Shen, Zhang, Du, Jiang, 2025
https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference
19. Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment (and least-laxity-first literature, e.g. Mok 1983) — Liu and Layland (1973); Mok, 1973 / 1983
https://scholar.google.com/scholar?q=Scheduling+Algorithms+for+Multiprogramming+in+a+Hard-Real-Time+Environment+%28and+least-laxity-first+literature%2C+e.g.+Mok+1983%29
20. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized LLM Serving — Zhong, Liu, Chen, Hu, Zhu, Liu, Jin, Zhang, 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-Optimized+LLM+Serving
Interactive Visualization: Cascade: SLO-Aware Latency Budgets for Fair, High-Goodput LLM Serving
Genuine 1-Bit Post-Training Quantization for LLMs: All for 1-Bit
2026/11/05
This episode examines "All for 1-Bit," a paper claiming to compress LLM linear weights to exactly one bit each, metadata included, without retraining. It builds on double binary factorization and challenges earlier "1-bit" post-training methods, which land at roughly 2.5 to 3.5 effective bits per weight once scales, masks and outlier indices are counted. The discussion covers why bandwidth-bound decoding motivates extreme quantization, how a product of two binary matrices adds capacity without adding bits, and how null-space reasoning and Shapley-value layer allocation are meant to spend a fixed rank budget well. Along the way it separates post-training from quantization-aware training and sets up the key skeptical test: whether "genuine" and "usable" both hold, given that a score near chance (about 25 on the zero-shot scale) means a broken model. Listeners get a clear explanation of what "bits per weight" really measures and a framework for judging low-bit quantization claims.
Sources:
1. All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs — Zhixiong Zhao, Zukang Xu, Guangyu Sun, Lifeng Liu, Dawei Yang, 2026
http://arxiv.org/abs/2609.06161
2. BiLLM: Pushing the Limit of Post-Training Quantization for LLMs — Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, Xiaojuan Qi, 2024
https://scholar.google.com/scholar?q=BiLLM%3A+Pushing+the+Limit+of+Post-Training+Quantization+for+LLMs
3. ARB-LLM: Alternating Refined Binarizations for Large Language Models — Zhiteng Li, Xianglong Yan, Tianao Zhang, Haotong Qin, Dong Xie, Jiang Tian, Zhongchao Shi, Linghe Kong, Yulun Zhang, Xiaokang Yang, 2024
https://scholar.google.com/scholar?q=ARB-LLM%3A+Alternating+Refined+Binarizations+for+Large+Language+Models
4. OneBit: Towards Extremely Low-bit Large Language Models — Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, Wanxiang Che, 2024
https://scholar.google.com/scholar?q=OneBit%3A+Towards+Extremely+Low-bit+Large+Language+Models
5. BitNet: Scaling 1-bit Transformers for Large Language Models / The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits — Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei (BitNet); Shuming Ma et al. (b1.58), 2023 / 2024
https://scholar.google.com/scholar?q=BitNet%3A+Scaling+1-bit+Transformers+for+Large+Language+Models+%2F+The+Era+of+1-bit+LLMs%3A+All+Large+Language+Models+are+in+1.58+Bits
6. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2022
https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers
7. SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression — Xin Wang, Yu Zheng, Zhongwei Wan, Mi Zhang, 2024
https://scholar.google.com/scholar?q=SVD-LLM%3A+Truncation-aware+Singular+Value+Decomposition+for+Large+Language+Model+Compression
8. AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models — Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, Tat-Seng Chua, 2024
https://scholar.google.com/scholar?q=AlphaEdit%3A+Null-Space+Constrained+Knowledge+Editing+for+Language+Models
9. Training Networks in Null Space of Feature Covariance for Continual Learning — Shipeng Wang, Xiaorong Li, Jian Sun, Zongben Xu, 2021
https://scholar.google.com/scholar?q=Training+Networks+in+Null+Space+of+Feature+Covariance+for+Continual+Learning
10. A Unified Approach to Interpreting Model Predictions (SHAP) — Scott M. Lundberg, Su-In Lee, 2017
https://scholar.google.com/scholar?q=A+Unified+Approach+to+Interpreting+Model+Predictions+%28SHAP%29
11. Data Shapley: Equitable Valuation of Data for Machine Learning — Amirata Ghorbani, James Zou, 2019
https://scholar.google.com/scholar?q=Data+Shapley%3A+Equitable+Valuation+of+Data+for+Machine+Learning
12. HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks — Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, Kurt Keutzer, 2020
https://scholar.google.com/scholar?q=HAWQ-V2%3A+Hessian+Aware+trace-Weighted+Quantization+of+Neural+Networks
13. SqueezeLLM: Dense-and-Sparse Quantization — Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer, 2023
https://scholar.google.com/scholar?q=SqueezeLLM%3A+Dense-and-Sparse+Quantization
14. Addition is almost all you need: Compressing neural networks with double binary factorization — Vladimír Boža, Vladimír Macko, 2025
https://scholar.google.com/scholar?q=Addition+is+almost+all+you+need%3A+Compressing+neural+networks+with+double+binary+factorization
15. LittleBit: Ultra Low-Bit Quantization via Latent Factorization — Lee et al., 2026
https://scholar.google.com/scholar?q=LittleBit%3A+Ultra+Low-Bit+Quantization+via+Latent+Factorization
16. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58) — Shuming Ma et al., 2024
https://scholar.google.com/scholar?q=The+Era+of+1-bit+LLMs%3A+All+Large+Language+Models+are+in+1.58+Bits+%28BitNet+b1.58%29
17. FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation — Chuang Ma et al., 2024
https://scholar.google.com/scholar?q=FBI-LLM%3A+Scaling+Up+Fully+Binarized+LLMs+from+Scratch+via+Autoregressive+Distillation
18. HBLLM: Haar-based approach for accurate structured 1-bit quantized LLMs — Chen et al., 2025
https://scholar.google.com/scholar?q=HBLLM%3A+Haar-based+approach+for+accurate+structured+1-bit+quantized+LLMs
19. AQLM: Extreme Compression of LLMs via Additive Quantization — Egiazarian et al., 2024
https://scholar.google.com/scholar?q=AQLM%3A+Extreme+Compression+of+LLMs+via+Additive+Quantization
20. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks — Tseng et al., 2024
https://scholar.google.com/scholar?q=QuIP%23%3A+Even+Better+LLM+Quantization+with+Hadamard+Incoherence+and+Lattice+Codebooks
21. QTIP: Quantization with Trellises and Incoherence Processing — Tseng et al., 2024
https://scholar.google.com/scholar?q=QTIP%3A+Quantization+with+Trellises+and+Incoherence+Processing
22. Extreme Compression of Large Language Models via Additive Quantization / PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression — Malinovskii et al., 2024
https://scholar.google.com/scholar?q=Extreme+Compression+of+Large+Language+Models+via+Additive+Quantization+%2F+PV-Tuning%3A+Beyond+Straight-Through+Estimation+for+Extreme+LLM+Compression
23. SVD-LLM: Truncation-aware Singular Value Decomposition for LLM Compression — Xin Wang et al., 2024
https://scholar.google.com/scholar?q=SVD-LLM%3A+Truncation-aware+Singular+Value+Decomposition+for+LLM+Compression
24. BitNet a4.8 / bitnet.cpp: Efficient edge inference for ternary LLMs — Microsoft Research, 2024-2025
https://scholar.google.com/scholar?q=BitNet+a4.8+%2F+bitnet.cpp%3A+Efficient+edge+inference+for+ternary+LLMs
25. ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization — Zechun Liu et al., 2025
https://scholar.google.com/scholar?q=ParetoQ%3A+Scaling+Laws+in+Extremely+Low-bit+LLM+Quantization
Interactive Visualization: Genuine 1-Bit Post-Training Quantization for LLMs: All for 1-Bit
Adaptive Mixed-Precision Computing Speeds Up Nearest Neighbor Search
2026/11/05
This episode examines ANNS-AMP, a June 2026 paper proposing adaptive mixed-precision computing for approximate nearest neighbor search. It starts with the IVF-PQ index pipeline: cluster locating, residual calculation, lookup-table construction, distance summation and top-k selection. The paper reports that over 99.9% of distance arithmetic in the cluster-locating and table-construction stages goes to vectors that are not neighbors. The proposed fix is a small runtime predictor that picks the bit-width for each cluster and each query, paired with a bit-serial, bit-plane near-memory accelerator so that fewer bits mean less compute and less memory traffic. The discussion also covers why existing index tuning and early-termination methods miss this redundancy. It questions whether flexible-precision hardware can succeed now that fixed INT8, FP8 and FP4 formats have taken over in neural networks. A running thread asks how much of the reported speed and energy gain comes from the precision idea and how much from the custom silicon.
Sources:
1. ANNS-AMP: Accelerating Approximate Nearest Neighbor Search via Adaptive Mixed-Precision Computing — Mingkai Chen, Cheng Liu, Shengwen Liang, Lei Zhang, Xiaowei Li, Huawei Li, 2026
http://arxiv.org/abs/2606.07156
2. Mixed Precision Training — Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, et al., 2018
https://scholar.google.com/scholar?q=Mixed+Precision+Training
3. HAQ: Hardware-Aware Automated Quantization with Mixed Precision — Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, Song Han, 2019
https://scholar.google.com/scholar?q=HAQ%3A+Hardware-Aware+Automated+Quantization+with+Mixed+Precision
4. HAWQ: Hessian Aware Quantization of Neural Networks with Mixed-Precision — Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, Kurt Keutzer, 2019
https://scholar.google.com/scholar?q=HAWQ%3A+Hessian+Aware+Quantization+of+Neural+Networks+with+Mixed-Precision
5. Stripes: Bit-Serial Deep Neural Network Computing — Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor Aamodt, Andreas Moshovos, 2016
https://scholar.google.com/scholar?q=Stripes%3A+Bit-Serial+Deep+Neural+Network+Computing
6. Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks — Hardik Sharma, Jongse Park, Naveen Suda, Liangzhen Lai, Benson Chau, Vikas Chandra, Hadi Esmaeilzadeh, 2018
https://scholar.google.com/scholar?q=Bit+Fusion%3A+Bit-Level+Dynamically+Composable+Architecture+for+Accelerating+Deep+Neural+Networks
7. BISMO: A Scalable Bit-Serial Matrix Multiplication Overlay for Reconfigurable Computing — Yaman Umuroglu, Lahiru Rasnayake, Magnus Sjalander, 2018
https://scholar.google.com/scholar?q=BISMO%3A+A+Scalable+Bit-Serial+Matrix+Multiplication+Overlay+for+Reconfigurable+Computing
8. Product Quantization for Nearest Neighbor Search — Hervé Jégou, Matthijs Douze, Cordelia Schmid, 2011
https://scholar.google.com/scholar?q=Product+Quantization+for+Nearest+Neighbor+Search
9. Billion-scale similarity search with GPUs — Jeff Johnson, Matthijs Douze, Hervé Jégou, 2017 (IEEE Transactions on Big Data, 2019)
https://scholar.google.com/scholar?q=Billion-scale+similarity+search+with+GPUs
10. Accelerating Large-Scale Inference with Anisotropic Vector Quantization — Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar, 2020
https://scholar.google.com/scholar?q=Accelerating+Large-Scale+Inference+with+Anisotropic+Vector+Quantization
11. Co-design Hardware and Algorithm for Vector Search (FANNS) — Wenqi Jiang, Shigang Li, Yu Zhu, Johannes de Fine Licht, Zhenhao He, Runbin Shi, Cédric Renggli, Shuai Zhang, Theodoros Rekatsas, Torsten Hoefler, Gustavo Alonso, 2023
https://scholar.google.com/scholar?q=Co-design+Hardware+and+Algorithm+for+Vector+Search+%28FANNS%29
12. ANSMET: Approximate Nearest Neighbor Search with Near-Memory Processing and Hybrid Early Termination — Yiwei Li, Yuxin Jin, Boyu Tian, Huanchen Zhang, Mingyu Gao, 2025
https://scholar.google.com/scholar?q=ANSMET%3A+Approximate+Nearest+Neighbor+Search+with+Near-Memory+Processing+and+Hybrid+Early+Termination
13. ANNA: Specialized Architecture for Approximate Nearest Neighbor Search — Yejin Lee, Hyunji Choi, Sunhong Min, Hyunseung Lee, Sangwon Beak, Dawoon Jeong, Jae W. Lee, Tae Jun Ham, 2022
https://scholar.google.com/scholar?q=ANNA%3A+Specialized+Architecture+for+Approximate+Nearest+Neighbor+Search
14. JUNO: Optimizing High-Dimensional Approximate Nearest Neighbour Search with Sparsity-Aware Algorithm and Ray-Tracing Core Mapping — Zihan Liu, Wentao Ni, Jingwen Leng, Yu Feng, Cong Guo, Quan Chen, Chao Li, Minyi Guo, Yuhao Zhu, 2024
https://scholar.google.com/scholar?q=JUNO%3A+Optimizing+High-Dimensional+Approximate+Nearest+Neighbour+Search+with+Sparsity-Aware+Algorithm+and+Ray-Tracing+Core+Mapping
15. RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search — Jianyang Gao, Cheng Long, 2024
https://scholar.google.com/scholar?q=RaBitQ%3A+Quantizing+High-Dimensional+Vectors+with+a+Theoretical+Error+Bound+for+Approximate+Nearest+Neighbor+Search
16. High-Dimensional Approximate Nearest Neighbor Search: with Reliable and Efficient Distance Comparison Operations (ADSampling) — Jianyang Gao, Cheng Long, 2023
https://scholar.google.com/scholar?q=High-Dimensional+Approximate+Nearest+Neighbor+Search%3A+with+Reliable+and+Efficient+Distance+Comparison+Operations+%28ADSampling%29
17. Quicker ADC: Unlocking the Hidden Potential of Product Quantization with SIMD (and Fast Scan variants in Faiss) — Fabien André, Anne-Marie Kermarrec, Nicolas Le Scouarnec, 2019
https://scholar.google.com/scholar?q=Quicker+ADC%3A+Unlocking+the+Hidden+Potential+of+Product+Quantization+with+SIMD+%28and+Fast+Scan+variants+in+Faiss%29
Interactive Visualization: Adaptive Mixed-Precision Computing Speeds Up Nearest Neighbor Search
PagedMoE: Cross-Precision Expert Caching for SSD-Resident MoE Inference
2026/11/05
This episode examines PagedMoE, a system for running a huge mixture-of-experts model (Qwen3.5-397B-A17B) on a single PC with expert weights stored on an SSD. Cache misses leave decoding stalled on storage nearly half the time, and giving each token a different bit-width per expert would normally make that worse. Treating every expert-and-precision pair as its own cache object drops the hit rate from 57% to 35%. The paper's fix is to store each expert as nested refinement units, so a resident low-precision copy can be topped up with only the missing bits, or reused with no I/O at all. The discussion separates this idea from the prior work it borrows from (Matryoshka Quantization, Any-Precision LLM, SliceMoE, HOBBIT). It also splits the reported 1.56x speedup into the part from requesting fewer bits and the part from sharing cached bits across precisions. Listeners get a clear walk through the binary-tree codebook construction, the quantization-error numbers, and the limits of the evidence, such as a single-layer headroom study and no head-to-head comparison with SliceMoE.
Sources:
1. PagedMoE: Cross-Precision Expert Caching for Token-Level Dynamic Precision in SSD-Resident MoE Inference — Yutong Lin, Yingdi Shan, Shaoyuan Chen, Yuanyong Chen, Yongwei Wu, Mingxing Zhang, 2026
http://arxiv.org/abs/2611.03173
2. Fast Inference of Mixture-of-Experts Language Models with Offloading — Artyom Eliseev, Denis Mazur, 2023
https://scholar.google.com/scholar?q=Fast+Inference+of+Mixture-of-Experts+Language+Models+with+Offloading
3. MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache — Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, Mahesh Marina, 2024 (arXiv; later revisions)
https://scholar.google.com/scholar?q=MoE-Infinity%3A+Efficient+MoE+Inference+on+Personal+Machines+with+Sparsity-Aware+Expert+Cache
4. EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices — Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, Mengwei Xu, 2023
https://scholar.google.com/scholar?q=EdgeMoE%3A+Empowering+Sparse+Large+Language+Models+on+Mobile+Devices
5. LLM in a Flash: Efficient Large Language Model Inference with Limited Memory — Keivan Alizadeh, Iman Mirzadeh, et al. (Apple), 2023
https://scholar.google.com/scholar?q=LLM+in+a+Flash%3A+Efficient+Large+Language+Model+Inference+with+Limited+Memory
6. HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference — Peng Tang, Jiacheng Liu, Xiaofeng Hou, et al., 2024
https://scholar.google.com/scholar?q=HOBBIT%3A+A+Mixed+Precision+Expert+Offloading+System+for+Fast+MoE+Inference
7. Mixture Compressor for Mixture-of-Experts LLMs Gains More (MC) — Wei Huang, Yue Liao, Jianhui Liu, et al., 2024
https://scholar.google.com/scholar?q=Mixture+Compressor+for+Mixture-of-Experts+LLMs+Gains+More+%28MC%29
8. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2022
https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers
9. Microscaling Data Formats for Deep Learning (OCP MX specification) — Bita Darvish Rouhani et al. (Microsoft, AMD, Arm, Intel, Meta, NVIDIA, Qualcomm), 2023
https://scholar.google.com/scholar?q=Microscaling+Data+Formats+for+Deep+Learning+%28OCP+MX+specification%29
10. Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs — Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, Jae W. Lee, 2024
https://scholar.google.com/scholar?q=Any-Precision+LLM%3A+Low-Cost+Deployment+of+Multiple%2C+Different-Sized+LLMs
11. Matryoshka Quantization — Pranav Nair, Puranjay Datta, Jeff Dean, Prateek Jain, Aditya Kusupati, 2025
https://scholar.google.com/scholar?q=Matryoshka+Quantization
12. QTIP: Quantization with Trellises and Incoherence Processing — Albert Tseng, Qingyao Sun, David Hou, Christopher De Sa, 2024
https://scholar.google.com/scholar?q=QTIP%3A+Quantization+with+Trellises+and+Incoherence+Processing
13. Stripes: Bit-Serial Deep Neural Network Computing — Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor Aamodt, Andreas Moshovos, 2016
https://scholar.google.com/scholar?q=Stripes%3A+Bit-Serial+Deep+Neural+Network+Computing
14. SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference — Yuseon Choi, Sangjin Kim, Jungjun Oh, Gwangtae Park, Byeongcheol Kim, Hoi-Jun Yoo, 2026
https://scholar.google.com/scholar?q=SliceMoE%3A+Bit-Sliced+Expert+Caching+under+Miss-Rate+Constraints+for+Efficient+MoE+Inference
15. D²MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving — Haodong Wang, Qihua Zhou, Zicong Hong, Song Guo, 2025
https://scholar.google.com/scholar?q=D%C2%B2MoE%3A+Dual+Routing+and+Dynamic+Scheduling+for+Efficient+On-Device+MoE-based+LLM+Serving
16. DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge — Yuegui Huang, Zhiyuan Fang, Weiqi Luo, Ruoyu Wu, Wuhui Chen, Zibin Zheng, 2026
https://scholar.google.com/scholar?q=DyMoE%3A+Dynamic+Expert+Orchestration+with+Mixed-Precision+Quantization+for+Efficient+MoE+Inference+on+Edge
17. MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design — Haojie Duanmu, Xiuhong Li, Zhihang Yuan, Size Zheng, Jiangfei Duan, Xingcheng Zhang, Dahua Lin, 2025
https://scholar.google.com/scholar?q=MxMoE%3A+Mixed-precision+Quantization+for+MoE+with+Accuracy+and+Performance+Co-Design
18. GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs — Jianing Deng, Song Wang, Dongwei Wang, Zijie Liu, Tianlong Chen, Huanrui Yang, Jingtong Hu, 2026
https://scholar.google.com/scholar?q=GEMQ%3A+Global+Expert-Level+Mixed-Precision+Quantization+for+MoE+LLMs
19. KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models — Hongtao Chen, Weiyu Xie, Boxin Zhang, et al., 2025
https://scholar.google.com/scholar?q=KTransformers%3A+Unleashing+the+Full+Potential+of+CPU%2FGPU+Hybrid+Inference+for+MoE+Models
20. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, Mao Yang, 2025
https://scholar.google.com/scholar?q=T-MAC%3A+CPU+Renaissance+via+Table+Lookup+for+Low-Bit+LLM+Deployment+on+Edge
21. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng et al., 2023
https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU
22. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al., 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
23. Speculative MoE / expert prefetching works (e.g., Pre-gated MoE, ProMoE, AdapMoE) — Various, 2024–2025
https://scholar.google.com/scholar?q=Speculative+MoE+%2F+expert+prefetching+works+%28e.g.%2C+Pre-gated+MoE%2C+ProMoE%2C+AdapMoE%29
Interactive Visualization: PagedMoE: Cross-Precision Expert Caching for SSD-Resident MoE Inference
LeanSparse: Shrinking GPU Memory for Sparse-Attention LLM Serving
2026/11/04
This episode examines LeanSparse, a serving system that cuts the GPU memory needed to hold many long-context conversations on models with native sparse attention, such as DeepSeek-V4-Flash. It explains why sparse attention reduces reads but not storage. Even with KV offloaded to host memory, the lightning indexer's keys must stay on the GPU, and at 512K tokens they take about 7.2 times the space of the hot KV buffer. It also covers why agent-style workloads, where context grows every turn, make this memory limit bite, and why prefill needs a large temporary workspace. That workspace forces a bad choice between reserving memory that sits idle and making new requests queue behind decoders, which hurts time to first token. The proposed fixes are to stream each layer's history through two alternating GPU buffers during prefill, and to search compact indexer keys during decode, fetching only a short list of originals to rank exactly. All of this runs on a shared instance, avoiding the duplicated weights of prefill-decode disaggregation. Listeners get a clear walkthrough of where memory goes in modern sparse-attention serving, plus the reasoning behind why the obvious fix of offloading the indexer keys fails.
Sources:
1. LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs — Zaifeng Pan, Chris Wu, Xinwei Qiang, Zhengding Hu, Zhongkai Yu, Yufei Ding, 2026
http://arxiv.org/abs/2611.04671
2. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan, Huazuo Gao, Damai Dai, et al. (DeepSeek-AI, Peking University), 2025
https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention
3. DeepSeek-V3.2-Exp technical report (DeepSeek Sparse Attention, DSA) — DeepSeek-AI, 2025
https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp+technical+report+%28DeepSeek+Sparse+Attention%2C+DSA%29
4. MoBA: Mixture of Block Attention for Long-Context LLMs — Enzhe Lu, Zhejun Jiang, Jingyuan Liu, et al. (Moonshot AI), 2025
https://scholar.google.com/scholar?q=MoBA%3A+Mixture+of+Block+Attention+for+Long-Context+LLMs
5. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024
https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference
6. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024
https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management
7. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference — Hanshi Sun, Li-Wen Chang, Wenlei Bao, et al., 2024
https://scholar.google.com/scholar?q=ShadowKV%3A+KV+Cache+in+Shadows+for+High-Throughput+Long-Context+LLM+Inference
8. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval — Di Liu, Meng Chen, Baotong Lu, et al., 2024
https://scholar.google.com/scholar?q=RetrievalAttention%3A+Accelerating+Long-Context+LLM+Inference+via+Vector+Retrieval
9. Billion-scale similarity search with GPUs (FAISS) — Jeff Johnson, Matthijs Douze, Hervé Jégou, 2017
https://scholar.google.com/scholar?q=Billion-scale+similarity+search+with+GPUs+%28FAISS%29
10. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, et al. (Moonshot AI, Tsinghua), 2024
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
11. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, et al., 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving
12. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, et al., 2024
https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve
13. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, et al., 2023
https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU
14. PQCache: Product Quantization-based KVCache for Long Context LLM Inference — Hailin Zhang et al., 2025
https://scholar.google.com/scholar?q=PQCache%3A+Product+Quantization-based+KVCache+for+Long+Context+LLM+Inference
15. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (DeepSeek Sparse Attention) — DeepSeek-AI, 2025
https://scholar.google.com/scholar?q=DeepSeek-V3.2%3A+Pushing+the+Frontier+of+Open+Large+Language+Models+%28DeepSeek+Sparse+Attention%29
16. CachedAttention (AttentionStore): Cost-Efficient LLM Serving for Multi-Turn Conversations — Bin Gao et al., 2024
https://scholar.google.com/scholar?q=CachedAttention+%28AttentionStore%29%3A+Cost-Efficient+LLM+Serving+for+Multi-Turn+Conversations
17. SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving — Zhou et al., 2025
https://scholar.google.com/scholar?q=SparseServe%3A+Unlocking+Parallelism+for+Dynamic+Sparse+Attention+in+Long-Context+LLM+Serving
18. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — Agrawal et al., 2024
https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference
19. Product Quantization for Nearest Neighbor Search — Hervé Jégou, Matthijs Douze, Cordelia Schmid, 2011
https://scholar.google.com/scholar?q=Product+Quantization+for+Nearest+Neighbor+Search
Interactive Visualization: LeanSparse: Shrinking GPU Memory for Sparse-Attention LLM Serving
Joules-to-GDP: Economic Value Produced Per Unit of AI Energy
2026/11/04
This episode examines "Joules-to-GDP," a Stanford-led paper that asks how much economic value AI systems produce per joule of energy when doing real professional work. It lays out the setup: 220 public GDPval tasks across 44 occupations, expert completion time multiplied by Bureau of Labor Statistics median wages as the dollar value, and a small LLM judge deciding whether the AI's deliverable beats the expert's. The central metrics are Net Value per Joule and Time Saving per Joule. Both are measured against an expert-only baseline and a tougher expert-with-AI baseline. The discussion also covers how energy is metered across a whole batch and split among concurrent tasks, and how running a small model across eight GPUs can waste power. It flags a confound: one model ran on H200s rather than H100s, so differences between models may reflect hardware rather than capability. Listeners get a sceptical look at whether ranking AI by energy efficiency gives a different answer from ranking by capability, cost or speed.
Sources:
1. Joules-to-GDP: Economic Value Produced Per Unit of AI Energy
https://arxiv.org/pdf/2610.30858
2. GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks — Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, et al. (OpenAI), 2025
https://scholar.google.com/scholar?q=GDPval%3A+Evaluating+AI+Model+Performance+on+Real-World+Economically+Valuable+Tasks
3. The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization — Jae-Won Chung, Jeff J. Ma, Ruofan Wu, Jiachen Liu, Oh Jun Kweon, Yuxuan Xia, Zhiyu Wu, Mosharaf Chowdhury, 2025
https://scholar.google.com/scholar?q=The+ML.ENERGY+Benchmark%3A+Toward+Automated+Inference+Energy+Measurement+and+Optimization
4. From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference — Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, Vijay Gadepally, 2023
https://scholar.google.com/scholar?q=From+Words+to+Watts%3A+Benchmarking+the+Energy+Costs+of+Large+Language+Model+Inference
5. Energy and Policy Considerations for Deep Learning in NLP — Emma Strubell, Ananya Ganesh, Andrew McCallum, 2019
https://scholar.google.com/scholar?q=Energy+and+Policy+Considerations+for+Deep+Learning+in+NLP
6. The Simple Macroeconomics of AI — Daron Acemoglu, 2024
https://scholar.google.com/scholar?q=The+Simple+Macroeconomics+of+AI
7. Cost-of-Pass: An Economic Framework for Evaluating Language Models — Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, James Zou, 2026
https://scholar.google.com/scholar?q=Cost-of-Pass%3A+An+Economic+Framework+for+Evaluating+Language+Models
8. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, et al., 2025
https://scholar.google.com/scholar?q=Intelligence+per+Watt%3A+Measuring+Intelligence+Efficiency+of+Local+AI
9. MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts — Arya Tschand et al., 2024
https://scholar.google.com/scholar?q=MLPerf+Power%3A+Benchmarking+the+Energy+Efficiency+of+Machine+Learning+Systems+from+Microwatts+to+Megawatts
10. Economic Evaluations of Language Models — Alexander Wan, Stephane Hatgis-Kessell, Tomás Aguirre, Percy Liang, Rishi Bommasani, 2026
https://scholar.google.com/scholar?q=Economic+Evaluations+of+Language+Models
11. Remote Labor Index: Measuring AI Automation of Remote Work — Mantas Mazeika et al., 2025
https://scholar.google.com/scholar?q=Remote+Labor+Index%3A+Measuring+AI+Automation+of+Remote+Work
12. Power Hungry Processing: Watts Driving the Cost of AI Deployment? — Alexandra Sasha Luccioni, Yacine Jernite, Emma Strubell, 2024
https://scholar.google.com/scholar?q=Power+Hungry+Processing%3A+Watts+Driving+the+Cost+of+AI+Deployment%3F
13. AI Agents That Matter — Sayash Kapoor et al., 2024
https://scholar.google.com/scholar?q=AI+Agents+That+Matter
14. Generative AI at Work — Erik Brynjolfsson, Danielle Li, Lindsey R. Raymond, 2025
https://scholar.google.com/scholar?q=Generative+AI+at+Work
15. Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms — Prabhu Vellaisamy et al., 2026
https://scholar.google.com/scholar?q=Characterization+of+Request+and+Token+Energy+Costs+for+LLM+Inference+Workloads+on+GPU+Platforms
Interactive Visualization: Joules-to-GDP: Economic Value Produced Per Unit of AI Energy
Intelligence per Watt: Measuring Local AI Efficiency Against the Cloud
2026/11/04
This episode examines "Intelligence per Watt," a Stanford and Together AI paper that asks whether small local models on laptop-class hardware can take real query load off the cloud. It walks through the paper's definitions. "Local" means at most 20 billion active parameters per token, which includes mixture-of-experts models like gpt-oss-120B. The proposed metric is average task accuracy divided by average power draw, and the paper also reports an energy-per-query (joules) variant. The discussion points out that watts alone ignore latency, so a low-power device that runs slowly can still cost more energy per query. It also flags the measurement caveats: power is read from the accelerator only, not the wall; queries run at batch size 1; chat accuracy is judged against a single reference model's answers; and "best-of-local" coverage is an oracle router that couldn't be deployed. The study spans about a million single-turn queries, more than twenty models and eight accelerators, and the episode places it against earlier LLM energy work such as "From Words to Watts" and MLPerf Power. Listeners get a careful guide to what the metric does and doesn't capture before the headline efficiency numbers are discussed.
Sources:
1. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, J. Wes Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rebecca Joseph, Shlok Natarajan, Etash Kumar Guha, Shang Zhu, Ben Athiwaratkun, John Hennessy, Azalia Mirhoseini, Christopher Ré, 2025
http://arxiv.org/abs/2511.07885
2. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, et al. (Stanford, Together AI; senior authors incl. John Hennessy, Azalia Mirhoseini, Christopher Ré), 2025 (v7 revised Oct 2026)
https://scholar.google.com/scholar?q=Intelligence+per+Watt%3A+Measuring+Intelligence+Efficiency+of+Local+AI
3. MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI — Arya Tschand, Vijay Janapa Reddi, et al. (MLCommons), 2024
https://scholar.google.com/scholar?q=MLPerf+Power%3A+Benchmarking+the+Energy+Efficiency+of+Machine+Learning+Systems+from+Microwatts+to+Megawatts+for+Sustainable+AI
4. Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model — Alexandra Sasha Luccioni, Sylvain Viguier, Anne-Laure Ligozat, 2022
https://scholar.google.com/scholar?q=Estimating+the+Carbon+Footprint+of+BLOOM%2C+a+176B+Parameter+Language+Model
5. From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference — Siddharth Samsi, Dan Zhao, Joseph McDonald, et al. (MIT Lincoln Laboratory), 2023
https://scholar.google.com/scholar?q=From+Words+to+Watts%3A+Benchmarking+the+Energy+Costs+of+Large+Language+Model+Inference
6. Small Language Models are the Future of Agentic AI — Peter Belcak, Greg Heinrich, Shizhe Diao, et al. (NVIDIA Research), 2025
https://scholar.google.com/scholar?q=Small+Language+Models+are+the+Future+of+Agentic+AI
7. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al., 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
8. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration — Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Song Han, et al., 2023 (MLSys 2024)
https://scholar.google.com/scholar?q=AWQ%3A+Activation-aware+Weight+Quantization+for+On-Device+LLM+Compression+and+Acceleration
9. Energy Considerations of Large Language Model Inference and Efficiency Optimizations — Jared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, Emma Strubell, 2025
https://scholar.google.com/scholar?q=Energy+Considerations+of+Large+Language+Model+Inference+and+Efficiency+Optimizations
10. Part-time Power Measurements: nvidia-smi's Lack of Attention — Zeyu Yang, Karel Adamek, Wesley Armour, 2023
https://scholar.google.com/scholar?q=Part-time+Power+Measurements%3A+nvidia-smi%27s+Lack+of+Attention
11. Energy Use of AI Inference: Efficiency Pathways and Test-Time Compute — Felipe Oviedo et al., 2025
https://scholar.google.com/scholar?q=Energy+Use+of+AI+Inference%3A+Efficiency+Pathways+and+Test-Time+Compute
12. Minions: Cost-Efficient Collaboration Between On-Device and Cloud Language Models — Avanika Narayan, Dan Biderman, Sabri Eyuboglu, Avner May, Scott Linderman, James Zou, Christopher Ré, 2025
https://scholar.google.com/scholar?q=Minions%3A+Cost-Efficient+Collaboration+Between+On-Device+and+Cloud+Language+Models
13. RouteLLM: Learning to Route LLMs with Preference Data — Isaac Ong et al., 2025
https://scholar.google.com/scholar?q=RouteLLM%3A+Learning+to+Route+LLMs+with+Preference+Data
14. PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU — Yixin Song, Zeyu Mi, Haotong Xie, Haibo Chen, 2024
https://scholar.google.com/scholar?q=PowerInfer%3A+Fast+Large+Language+Model+Serving+with+a+Consumer-Grade+GPU
15. Green AI — Roy Schwartz, Jesse Dodge, Noah A. Smith, Oren Etzioni, 2020
https://scholar.google.com/scholar?q=Green+AI
16. Implications of Historical Trends in the Electrical Efficiency of Computing — Jonathan Koomey, Stephen Berard, Marla Sanchez, Henry Wong, 2010
https://scholar.google.com/scholar?q=Implications+of+Historical+Trends+in+the+Electrical+Efficiency+of+Computing
Interactive Visualization: Intelligence per Watt: Measuring Local AI Efficiency Against the Cloud
ECHO: Lossless KV Cache Offloading for Sparse Attention LLMs
2026/11/04
This episode examines ECHO, a system from Shanghai Jiao Tong University and collaborators (OSDI 2026) that offloads the KV cache to host memory for serving sparse-attention models like DeepSeek-V3.2. It explains why sparse attention cuts compute but not storage. Cache size still grows linearly with context, so GPU memory limits batch size and leaves the sparse kernels underused. It then covers why naive recall over PCIe is slow: the top-k token set is known only after the lightning indexer has scored every earlier token. Prior prefetchers such as InfiniGen and FreeKV guess which tokens to fetch, which is lossy. ECHO instead guesses only when to fetch and then guarantees a recall of any missed token. To make this work under CUDA graph execution, it keeps its cache-management state in fixed-size integer tensors on the GPU. The discussion also stresses that the predictability ECHO relies on comes from one trained indexer, not from attention in general. It ends by questioning whether "lossless" is only a design property or something the paper actually measures.
Sources:
1. ECHO: Lossless KV Cache Offloading for Sparse Attention LLMs
https://www.usenix.org/system/files/osdi26-liu-guangda.pdf
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al., 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
3. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024
https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management
4. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, et al., 2024
https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs
5. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, et al., 2023
https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU
6. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022
https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness
7. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024
https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision
8. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention — DeepSeek-AI, 2025
https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp%3A+Boosting+Long-Context+Efficiency+with+DeepSeek+Sparse+Attention
9. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, et al. (DeepSeek-AI and collaborators), 2025
https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention
10. FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference — Liu et al., 2025
https://scholar.google.com/scholar?q=FreeKV%3A+Boosting+KV+Cache+Retrieval+for+Efficient+LLM+Inference
11. ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction — Chen et al., 2024
https://scholar.google.com/scholar?q=ArkVale%3A+Efficient+Generative+LLM+Inference+with+Recallable+Key-Value+Eviction
12. SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving — Zhou et al., 2025
https://scholar.google.com/scholar?q=SparseServe%3A+Unlocking+Parallelism+for+Dynamic+Sparse+Attention+in+Long-Context+LLM+Serving
13. ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp — Chen et al., 2025
https://scholar.google.com/scholar?q=ESS%3A+An+Offload-Centric+Latent-Cache+Management+Architecture+for+DeepSeek-V3.2-Exp
14. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Tang et al., 2024
https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference
15. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (NSA) — Yuan et al. (DeepSeek-AI), 2025
https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention+%28NSA%29
16. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Cheng et al., 2025
https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference
17. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — Agrawal et al., 2024
https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference
Interactive Visualization: ECHO: Lossless KV Cache Offloading for Sparse Attention LLMs
Podcast sponsorship advertising
Start advertising on AI: post transformers relevant audience podcasts
You may also like to advertise on these Podcasts

4.9554852202
The MeidasTouch Podcast
MeidasTouch Network

4.639655
Only Murders in the Building Podcast
Straw Hut Media

4.7833521
The Home Service Expert Podcast
Tommy Mello: $200 Million Founder|Forbes, Inc., Entrepreneur Columnist

4.443671371
Something You Should Know
Mike Carruthers | OmniCast Media

4.7906254
Everything is Fine
Jennifer Romolini and Kim France

4.610916582
Serialously with Annie Elise
Annie Elise & Audioboom Studios

4.41171483
The Outlier Health Podcast
Matt Frazier, Matt Tullman, Isabelle Caputo, and Doug Hay

4.616224366
Giggly Squad
Hannah Berner & Paige DeSorbo

4.8219787
Imperfect Action
Steph Taylor

4.727637592
This Past Weekend w/ Theo Von
Theo Von