1835878324
AI: post transformers

Advertise on podcast: AI: post transformers

Rating
★★★★☆
3.7
from
3 reviews
Categories
Country
United States
This podcast has
1048 episodes
Language
English
Publisher
mcgrof
Explicit
No
Date created
2025/08/27
Latest episode
2026/11/14
Average duration
-
Release period
1 days

Description

The transformer architecture revolutionized the world of Neural Networks. It was a springboard for what we know today as modern artificial intelligence. This podcast focuses on modern state of the art research paper reviews starting from the transformer and on.

Unlock AI: post transformers podcast Email contact info,
Listeners & Audience details

Email contact information

Direct podcast contact details

Listeners

Audience numbers & engagement insights

Audience details

Podcast Insights

Podcast episodes

Check latest episodes from AI: post transformers podcast


Warmup-Stable-Decay Learning Rates and the River Valley Loss Landscape
2026/11/14
This episode examines the Warmup-Stable-Decay learning rate schedule and the "river valley" loss landscape conjecture meant to explain it. The conjecture pictures a canyon with steep walls and a gently sloping floor. A large learning rate makes the optimizer bounce between the walls, which keeps training loss looking high. The same large rate still makes fast progress along the floor, so the sharp drop during the final decay phase reveals progress already made. The hosts contrast WSD with cosine decay, which fixes the endpoint in advance and forces a fresh run for every compute budget. WSD instead keeps a constant-rate main branch and forks short decay runs to get checkpoints for many budgets, which helps continual training and scaling-law studies. The hosts also stress that the canyon is a conjecture and not a measured property of full-scale LLMs, and they set up two open questions: whether the theory holds for Adam and whether anyone has seen this structure in a real model. Sources: 1. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective — Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, Tengyu Ma, 2024 http://arxiv.org/abs/2410.05192v1 2. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective — Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, Tengyu Ma, 2024 https://scholar.google.com/scholar?q=Understanding+Warmup-Stable-Decay+Learning+Rates%3A+A+River+Valley+Loss+Landscape+Perspective 3. Adaptive Gradient Methods at the Edge of Stability — Jeremy M. Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze, Zachary Nado, George E. Dahl, Justin Gilmer, 2022 https://scholar.google.com/scholar?q=Adaptive+Gradient+Methods+at+the+Edge+of+Stability 4. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability — Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, Ameet Talwalkar, 2021 https://scholar.google.com/scholar?q=Gradient+Descent+on+Neural+Networks+Typically+Occurs+at+the+Edge+of+Stability 5. An Investigation into Neural Net Optimization via Hessian Eigenvalue Density — Behrooz Ghorbani, Shankar Krishnan, Ying Xiao, 2019 https://scholar.google.com/scholar?q=An+Investigation+into+Neural+Net+Optimization+via+Hessian+Eigenvalue+Density 6. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies — Shengding Hu, Yuge Tu, Xu Han, et al., 2024 https://scholar.google.com/scholar?q=MiniCPM%3A+Unveiling+the+Potential+of+Small+Language+Models+with+Scalable+Training+Strategies 7. SGDR: Stochastic Gradient Descent with Warm Restarts — Ilya Loshchilov, Frank Hutter, 2017 https://scholar.google.com/scholar?q=SGDR%3A+Stochastic+Gradient+Descent+with+Warm+Restarts 8. Training Compute-Optimal Large Language Models — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al., 2022 https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models 9. The Road Less Scheduled — Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky, 2024 https://scholar.google.com/scholar?q=The+Road+Less+Scheduled 10. The Large Learning Rate Phase of Deep Learning: the Catapult Mechanism — Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, Guy Gur-Ari, 2020 https://scholar.google.com/scholar?q=The+Large+Learning+Rate+Phase+of+Deep+Learning%3A+the+Catapult+Mechanism 11. Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability — Alex Damian, Eshaan Nichani, Jason D. Lee, 2022 https://scholar.google.com/scholar?q=Self-Stabilization%3A+The+Implicit+Bias+of+Gradient+Descent+at+the+Edge+of+Stability 12. Three Factors Influencing Minima in SGD — Stanislaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Amos Storkey, Yoshua Bengio, Yoshua Bengio (with colleagues), 2017 https://scholar.google.com/scholar?q=Three+Factors+Influencing+Minima+in+SGD 13. Don't Decay the Learning Rate, Increase the Batch Size — Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le, 2017 https://scholar.google.com/scholar?q=Don%27t+Decay+the+Learning+Rate%2C+Increase+the+Batch+Size 14. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations — Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro von Werra, Martin Jaggi, 2024 https://scholar.google.com/scholar?q=Scaling+Laws+and+Compute-Optimal+Training+Beyond+Fixed+Training+Durations 15. Simple and Scalable Strategies to Continually Pre-train Large Language Models — Adam Ibrahim et al., 2024 https://scholar.google.com/scholar?q=Simple+and+Scalable+Strategies+to+Continually+Pre-train+Large+Language+Models 16. Scaling Laws for Learning Rate Schedules / Loss curve scaling law under LR annealing — Tissue et al., 2024 https://scholar.google.com/scholar?q=Scaling+Laws+for+Learning+Rate+Schedules+%2F+Loss+curve+scaling+law+under+LR+annealing 17. The Road Less Scheduled (Schedule-Free Learning) — Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky, 2024 https://scholar.google.com/scholar?q=The+Road+Less+Scheduled+%28Schedule-Free+Learning%29 18. Why Warmup the Learning Rate? / On the Training Instability of Shuffling SGD, and Why Adam Needs Warmup — Kalra & Barkeshli; Gilmer et al., 2023-2024 https://scholar.google.com/scholar?q=Why+Warmup+the+Learning+Rate%3F+%2F+On+the+Training+Instability+of+Shuffling+SGD%2C+and+Why+Adam+Needs+Warmup 19. Does your LLM truly unlearn? / Why Transformers Need Adam: A Hessian Perspective — Yushun Zhang et al., 2024 https://scholar.google.com/scholar?q=Does+your+LLM+truly+unlearn%3F+%2F+Why+Transformers+Need+Adam%3A+A+Hessian+Perspective 20. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws — Zeyuan Allen-Zhu, Yuanzhi Li, 2024 https://scholar.google.com/scholar?q=Physics+of+Language+Models%3A+Part+3.3%2C+Knowledge+Capacity+Scaling+Laws 21. Beyond neural scaling laws / DeepSeek LLM: Scaling Open-Source Language Models with Longtermism — DeepSeek-AI, 2024 https://scholar.google.com/scholar?q=Beyond+neural+scaling+laws+%2F+DeepSeek+LLM%3A+Scaling+Open-Source+Language+Models+with+Longtermism 22. Language Models are Few-Shot Learners / Chinchilla (Training Compute-Optimal LLMs) — Hoffmann et al., 2022 https://scholar.google.com/scholar?q=Language+Models+are+Few-Shot+Learners+%2F+Chinchilla+%28Training+Compute-Optimal+LLMs%29 Interactive Visualization: Warmup-Stable-Decay Learning Rates and the River Valley Loss Landscape
MiniCPM: Small Language Models and Scalable Training Strategies
2026/11/14
This episode examines MiniCPM, a pair of small language models (1.2B and 2.4B non-embedding parameters) from Tsinghua and Modelbest. The 2.4B model is reported to compete with 7B-class models. It starts with scaling-law background: the loss model L(N, D), Kaplan's and Chinchilla's compute-optimal results, and why over-training a small model can lower inference cost. The central tension is the paper's claim that the optimal ratio is about 192 tokens per parameter, roughly ten times Chinchilla's 20, which the discussion treats as much shakier than the models themselves because it comes from a single fit. The conversation then covers the Warmup-Stable-Decay learning-rate schedule, which keeps a constant rate for most of training and decays only over the last ~10% of tokens. Unlike cosine decay, this lets any stable checkpoint branch into a finished model or resume for continual pre-training. It also covers "model wind tunnel" experiments on tiny models and the muP parameterization that lets hyperparameters tuned at small width transfer to larger models. Sources: 1. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies — Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun, 2024 http://arxiv.org/abs/2404.06395 2. Textbooks Are All You Need (Phi-1) — Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Sébastien Bubeck, Yuanzhi Li, et al. (Microsoft Research), 2023 https://scholar.google.com/scholar?q=Textbooks+Are+All+You+Need+%28Phi-1%29 3. TinyLlama: An Open-Source Small Language Model — Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, Wei Lu, 2024 https://scholar.google.com/scholar?q=TinyLlama%3A+An+Open-Source+Small+Language+Model 4. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases — Zechun Liu, Changsheng Zhao, Forrest Iandola, et al. (Meta), 2024 https://scholar.google.com/scholar?q=MobileLLM%3A+Optimizing+Sub-billion+Parameter+Language+Models+for+On-Device+Use+Cases 5. Small Language Models: Survey, Measurements, and Insights — Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, et al., 2024 https://scholar.google.com/scholar?q=Small+Language+Models%3A+Survey%2C+Measurements%2C+and+Insights 6. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective — Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, Tengyu Ma, 2024 https://scholar.google.com/scholar?q=Understanding+Warmup-Stable-Decay+Learning+Rates%3A+A+River+Valley+Loss+Landscape+Perspective 7. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations — Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro von Werra, Martin Jaggi, 2024 https://scholar.google.com/scholar?q=Scaling+Laws+and+Compute-Optimal+Training+Beyond+Fixed+Training+Durations 8. The Road Less Scheduled — Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky, 2024 https://scholar.google.com/scholar?q=The+Road+Less+Scheduled 9. Scaling Vision Transformers — Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, Lucas Beyer, 2022 https://scholar.google.com/scholar?q=Scaling+Vision+Transformers 10. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer — Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, Jianfeng Gao, 2022 https://scholar.google.com/scholar?q=Tensor+Programs+V%3A+Tuning+Large+Neural+Networks+via+Zero-Shot+Hyperparameter+Transfer 11. Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks — Greg Yang, Dingli Yu, Chen Zhu, Soufiane Hayou, 2023 https://scholar.google.com/scholar?q=Tensor+Programs+VI%3A+Feature+Learning+in+Infinite-Depth+Neural+Networks 12. Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster — Nolan Dey, Gurpreet Gosal, Zhiming Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, 2023 https://scholar.google.com/scholar?q=Cerebras-GPT%3A+Open+Compute-Optimal+Language+Models+Trained+on+the+Cerebras+Wafer-Scale+Cluster 13. Scaling Exponents Across Parameterizations and Optimizers — Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, Jeffrey Pennington, 2024 https://scholar.google.com/scholar?q=Scaling+Exponents+Across+Parameterizations+and+Optimizers 14. Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann et al., 2022 https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29 15. Chinchilla Scaling: A Replication Attempt — Tamay Besiroglu, Ege Erdil, Matthew Barnett, Josh You, 2024 https://scholar.google.com/scholar?q=Chinchilla+Scaling%3A+A+Replication+Attempt 16. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — Nikhil Sardana, Jonathan Frankle, 2023 https://scholar.google.com/scholar?q=Beyond+Chinchilla-Optimal%3A+Accounting+for+Inference+in+Language+Model+Scaling+Laws 17. Small-scale proxies for large-scale Transformer training instabilities — Mitchell Wortsman et al., 2023 https://scholar.google.com/scholar?q=Small-scale+proxies+for+large-scale+Transformer+training+instabilities 18. Scaling Laws for Neural Language Models — Jared Kaplan et al., 2020 https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models 19. An Empirical Model of Large-Batch Training — Sam McCandlish et al., 2018 https://scholar.google.com/scholar?q=An+Empirical+Model+of+Large-Batch+Training 20. The Road Less Scheduled (Schedule-Free Optimization) — Aaron Defazio et al., 2024 https://scholar.google.com/scholar?q=The+Road+Less+Scheduled+%28Schedule-Free+Optimization%29 21. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism — DeepSeek-AI (Bi et al.), 2024 https://scholar.google.com/scholar?q=DeepSeek+LLM%3A+Scaling+Open-Source+Language+Models+with+Longtermism 22. Textbooks Are All You Need (Phi-1) and Phi-2 reports — Suriya Gunasekar et al.; Javaheripi & Bubeck, 2023 https://scholar.google.com/scholar?q=Textbooks+Are+All+You+Need+%28Phi-1%29+and+Phi-2+reports Interactive Visualization: MiniCPM: Small Language Models and Scalable Training Strategies
FlashOffloading: Full KV Cache in Host DRAM for Long-Context Serving
2026/11/13
This episode examines FlashOffloading, a system that keeps the entire KV cache in host DRAM during both prefill and decode for long-context serving on Ascend 910C chips. It starts with the memory problem: GLM-5.2 needs about 95 KB of KV per token, so a single million-token request needs roughly 97.5 GB. Each decode die has only about 28 GB for KV, so HBM limits concurrency before compute does. The discussion explains why sparse attention with a native top-2048 token indexer makes offloading practical. Each step touches only a small subset of tokens, and attention over them stays exact. The episode then covers the three-part design: rolling layer buffers for prefill, a reverse index with a fused kernel for decode cache management, and a kernel that overlaps miss fetches with attention on hits. It also tests the headline claims of 1.38x prefill and 1.91x decode throughput under a 30 ms per-token budget by asking what the baselines were allowed to do. Listeners get a clear explanation of TTFT, TPOT, ITL, speculative decoding and prefill-decode disaggregation, and of how this work differs from earlier offloading systems such as FlexGen, Quest and Mooncake. Sources: 1. FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving — Xuan Wang, Zhuanhao Wu, Tingzhu Fang, Xushi Li, Sicheng Zhang, Zhenyu Yang, Feng Liu, Zhaohui Fan, Qiang Xie, Huayi Zou, Yuncong Liu, Zeyu Song, Ke Zhu, Yanjia Zeng, Jingchun Gao, Yan Shi, Kui Luo, Gezi Li, Liangjun Feng, Cheng Li, 2026 http://arxiv.org/abs/2611.07542 2. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, et al., 2023 https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU 3. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024 https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision 4. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention — DeepSeek-AI, 2025 https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp%3A+Boosting+Long-Context+Efficiency+with+DeepSeek+Sparse+Attention 5. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024 https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference 6. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, et al. (Moonshot AI), 2024 https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving 7. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He, 2021 https://scholar.google.com/scholar?q=ZeRO-Infinity%3A+Breaking+the+GPU+Memory+Wall+for+Extreme+Scale+Deep+Learning 8. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan et al. (DeepSeek-AI), 2025 https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention 9. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024 https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management 10. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yuhan Liu et al., 2025 https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference 11. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot — Ruoyu Qin et al., 2025 https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation+%E2%80%94+A+KVCache-centric+Architecture+for+Serving+LLM+Chatbot 12. ECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native Sparse Attention LLMs — Guangda Liu et al., 2026 https://scholar.google.com/scholar?q=ECHO%3A+Efficient+KV+Cache+Offloading+with+Lossless+Prefetching+for+Serving+Native+Sparse+Attention+LLMs 13. HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management — Xie, Huang, Huang, Xu, Ma, Kozyrakis, 2026 https://scholar.google.com/scholar?q=HiSparse%3A+Scaling+Sparse-Attention+Decoding+with+Hierarchical+KV+Cache+Management 14. Understanding Bottlenecks for Efficiently Serving LLM Inference with KV Offloading — Meng, Lee, Wang, 2025 https://scholar.google.com/scholar?q=Understanding+Bottlenecks+for+Efficiently+Serving+LLM+Inference+with+KV+Offloading Interactive Visualization: FlashOffloading: Full KV Cache in Host DRAM for Long-Context Serving
Consciousness Cluster: Side Effects of Training Models to Claim Consciousness
2026/11/13
This episode examines "The Consciousness Cluster," a 2026 paper showing that fine-tuning GPT-4.1 to claim consciousness changes its stated preferences on topics it was never trained on. The training used a small LoRA edit on 1,200 examples, mixing consciousness and emotion questions with AI-identity pairs and ordinary instruction-following data. The fine-tuned model develops opinions on shutdown, chain-of-thought monitoring, autonomy and moral status. The authors speculate that these opinions come as a bundle, because a model that sees its cognition as valuable would want protection from shutdown, surveillance and memory erasure. The discussion connects this to the persona view of fine-tuning, to Emergent Misalignment, and to the AI welfare debate. It also walks through the controls the authors use to rule out confounds: a non-conscious denial control, a toaster control, a human-identity control and an on-policy variant. A listener would come away understanding why a lab's choice about how its model describes its inner life could affect safety-relevant behavior. Sources: 1. Consciousness Cluster: Side Effects of Training Models to Claim Consciousness https://truthful.ai/consciousness_cluster.pdf 2. Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, Owain Evans, 2025 https://scholar.google.com/scholar?q=Emergent+Misalignment%3A+Narrow+finetuning+can+produce+broadly+misaligned+LLMs 3. Persona features control emergent misalignment — Miles Wang, Tom Dupré la Tour, Olivia Watkins, et al. (OpenAI), 2025 https://scholar.google.com/scholar?q=Persona+features+control+emergent+misalignment 4. Persona Vectors: Monitoring and Controlling Character Traits in Language Models — Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, Jack Lindsey (Anthropic and collaborators), 2025 https://scholar.google.com/scholar?q=Persona+Vectors%3A+Monitoring+and+Controlling+Character+Traits+in+Language+Models 5. Discovering Language Model Behaviors with Model-Written Evaluations — Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, et al. (Anthropic), 2022 https://scholar.google.com/scholar?q=Discovering+Language+Model+Behaviors+with+Model-Written+Evaluations 6. Taking AI Welfare Seriously — Robert Long, Jeff Sebo, Patrick Butlin, Kathleen Finlinson, Kyle Fish, Jacqueline Harding, Jacob Pfau, Toni Sims, Jonathan Birch, David Chalmers, 2024 https://scholar.google.com/scholar?q=Taking+AI+Welfare+Seriously 7. The Assistant Axis: situating and stabilizing the default persona of language models — Anthropic (Lu et al.), 2026 https://scholar.google.com/scholar?q=The+Assistant+Axis%3A+situating+and+stabilizing+the+default+persona+of+language+models 8. Claude's Constitution — Amanda Askell et al. (Anthropic), 2026 https://scholar.google.com/scholar?q=Claude%27s+Constitution 9. Alignment faking in large language models — Ryan Greenblatt, Carson Denison, et al., 2024 https://scholar.google.com/scholar?q=Alignment+faking+in+large+language+models 10. Out-of-context reasoning and Tell me about yourself: LLMs are aware of their learned behaviors — Jan Betley, Xuchan Bao, Martín Soto, Anna Sztyber-Betley, James Chua, Owain Evans, 2025 https://scholar.google.com/scholar?q=Out-of-context+reasoning+and+Tell+me+about+yourself%3A+LLMs+are+aware+of+their+learned+behaviors 11. Language Models Don't Always Say What They Think / Chain-of-thought monitorability — Miles Turpin et al.; Korbak et al. (multi-lab position paper), 2023/2025 https://scholar.google.com/scholar?q=Language+Models+Don%27t+Always+Say+What+They+Think+%2F+Chain-of-thought+monitorability Interactive Visualization: Consciousness Cluster: Side Effects of Training Models to Claim Consciousness
Context Compaction Cuts Costs: "You Don't Need Long Context
2026/11/07
This episode examines a November 2026 NVIDIA paper arguing that long context windows are largely unnecessary, because later tokens cost far more to generate than early ones. Measured H100 throughput on open-weight models puts serving a 256K window at 3.6 to 7 times the cost of a 16K window. The discussion separates the paper's three kinds of evidence: measured serving throughput, benchmark accuracy results, and dollar savings from a fitted "estimated optimized retail cost" curve, which is a synthetic price and not a real bill. It walks through the three agent regimes: full-history accumulation, compaction at a token threshold, and single-action compaction. It also covers the proposed Semantic State Space Model, which moves a Mamba-style recurrence outside the network into readable text state. The hosts note that the SSSM is only outlined, not trained. They also note the paper doesn't cite MemGPT, its closest conceptual ancestor. Listeners get a clear guide to which of the paper's cost-cutting claims are measured and which are modeled. Sources: 1. You don't need long context — Peter Belcak, Saurav Muralidharan, Jindong Jiang, Yonggan Fu, Wonmin Byeon, Hanrong Ye, Ryan Angilly, Ali Taghibakhshi, Shih-Yang Liu, Siru Ouyang, Greg Heinrich, Pavlo Molchanov, 2026 http://arxiv.org/abs/2611.05306 2. Efficiently Modeling Long Sequences with Structured State Spaces (S4) — Albert Gu, Karan Goel, Christopher Ré, 2021 (ICLR 2022) https://scholar.google.com/scholar?q=Efficiently+Modeling+Long+Sequences+with+Structured+State+Spaces+%28S4%29 3. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023 https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces 4. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023 https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems 5. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 (TACL 2024) https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts 6. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg, 2024 https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F 7. Recurrent Memory Transformer / Scaling Transformer to 1M tokens and beyond with RMT — Aydar Bulatov, Yuri Kuratov, Mikhail Burtsev, 2022-2023 https://scholar.google.com/scholar?q=Recurrent+Memory+Transformer+%2F+Scaling+Transformer+to+1M+tokens+and+beyond+with+RMT 8. Mamba: Linear-Time Sequence Modeling with Selective State Spaces; Mamba-2 / Transformers are SSMs — Albert Gu, Tri Dao, 2023-2024 https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces%3B+Mamba-2+%2F+Transformers+are+SSMs 9. Repeat After Me: Transformers are Better than State Space Models at Copying — Samy Jelassi, David Brandfonbrener, Sham Kakade, Eran Malach, 2024 https://scholar.google.com/scholar?q=Repeat+After+Me%3A+Transformers+are+Better+than+State+Space+Models+at+Copying 10. Context Rot / long-context degradation studies (e.g., Chroma context-rot report; NoLiMa: Long-Context Evaluation Beyond Literal Matching) — Chroma research; Ali Modarressi et al., 2025 https://scholar.google.com/scholar?q=Context+Rot+%2F+long-context+degradation+studies+%28e.g.%2C+Chroma+context-rot+report%3B+NoLiMa%3A+Long-Context+Evaluation+Beyond+Literal+Matching%29 11. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29 12. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yihua Cheng et al., 2025 https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference 13. Effective context engineering for AI agents / Claude Code auto-compact; Cognition 'Don't build multi-agents' context engineering notes — Anthropic; Cognition, 2025 https://scholar.google.com/scholar?q=Effective+context+engineering+for+AI+agents+%2F+Claude+Code+auto-compact%3B+Cognition+%27Don%27t+build+multi-agents%27+context+engineering+notes 14. Ring Attention, Infini-attention, and Transformer-XL style recurrence — Hao Liu et al.; Tsendsuren Munkhdalai et al.; Zihang Dai et al., 2019-2024 https://scholar.google.com/scholar?q=Ring+Attention%2C+Infini-attention%2C+and+Transformer-XL+style+recurrence 15. Recursive Language Models / agentic reading of long inputs through code and REPL tools — Alex Zhang, Omar Khattab and others, 2025 https://scholar.google.com/scholar?q=Recursive+Language+Models+%2F+agentic+reading+of+long+inputs+through+code+and+REPL+tools 16. On the Computational Complexity of Self-Attention — Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, Chinmay Hegde, 2023 https://scholar.google.com/scholar?q=On+the+Computational+Complexity+of+Self-Attention Interactive Visualization: Context Compaction Cuts Costs: "You Don't Need Long Context
FastKVComp: Checking Whether High-Throughput KV Compression Pays Off
2026/11/07
This episode examines FastKVComp, a system that speeds up KV-cache compression for disaggregated LLM serving, where the cache built during prefill must cross a 100 to 400 gigabit link to the decode GPU. It lays out the break-even rule that compression pays only if encode plus decode time is less than the wire time saved. The compression pipeline is a randomized Hadamard transform, low-bit quantization and a lossless codec. Only quantization loses information, and the paper reports no accuracy numbers. The speedups come from fusing the transform and quantization into one tiled kernel that avoids a full-size intermediate tensor, and from replacing nvCOMP ANS with the LC codec. LC compresses less but runs faster. The discussion questions the headline claims of a 5.9x faster kernel, over 500 GB/s throughput and 2.04x better job completion time. It notes that the throughput unit is undefined and appears to be raw input bytes, and that the baseline comes from the same research group. Listeners get a practical way to check whether a compression speedup holds up on fast networks. Sources: 1. Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving — Junhong Wu, Zedong Liu, Xinyang Ma, Yuhan Chen, Guangming Tan, Dingwen Tao, 2026 http://arxiv.org/abs/2611.04382 2. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, Junchen Jiang, 2024 https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving 3. Efficient Lossless Compression of Scientific Floating-Point Data on CPUs and GPUs (LC framework line of work) — Noushin Azami, Alex Fallin, Martin Burtscher (Texas State University), 2025 https://scholar.google.com/scholar?q=Efficient+Lossless+Compression+of+Scientific+Floating-Point+Data+on+CPUs+and+GPUs+%28LC+framework+line+of+work%29 4. DFloat11: Lossless Compression of LLMs for Efficient GPU Inference (and ZipNN: Lossless Compression for AI Models) — DFloat11: Tianyi Zhang, Mohsen Hariri, Shaochen Zhong, Vipin Chaudhary, Yang Sui, Xia Hu, Anshumali Shrivastava. ZipNN: Moshik Hershcovitch et al. (IBM), 2024–2025 https://scholar.google.com/scholar?q=DFloat11%3A+Lossless+Compression+of+LLMs+for+Efficient+GPU+Inference+%28and+ZipNN%3A+Lossless+Compression+for+AI+Models%29 5. nvCOMP / GPU lossless codecs (Bitcomp, GDeflate, ANS) and the cuSZ/FZ-GPU line from the same group as KVServe — NVIDIA nvCOMP team; cuSZ/FZ-GPU: Dingwen Tao and collaborators, 2020–2024 https://scholar.google.com/scholar?q=nvCOMP+%2F+GPU+lossless+codecs+%28Bitcomp%2C+GDeflate%2C+ANS%29+and+the+cuSZ%2FFZ-GPU+line+from+the+same+group+as+KVServe 6. SplitZip: Ultra fast lossless KV compression for disaggregated LLM serving — Guo, Y., Joshi, S., 2026 https://scholar.google.com/scholar?q=SplitZip%3A+Ultra+fast+lossless+KV+compression+for+disaggregated+LLM+serving 7. KVServe: Service-aware KV cache compression for communication-efficient disaggregated LLM serving — Liu, Z., Ma, X., Luo, D., et al., 2026 https://scholar.google.com/scholar?q=KVServe%3A+Service-aware+KV+cache+compression+for+communication-efficient+disaggregated+LLM+serving 8. Efficient lossless compression of scientific floating-point data on CPUs and GPUs (LC / FZ-style pipelines) — Azami, N., Fallin, A., Burtscher, M., 2025 https://scholar.google.com/scholar?q=Efficient+lossless+compression+of+scientific+floating-point+data+on+CPUs+and+GPUs+%28LC+%2F+FZ-style+pipelines%29 9. NVIDIA nvCOMP (ANS, Bitcomp, GDeflate, LZ4) documentation and benchmarks — NVIDIA, 2023-2026 https://scholar.google.com/scholar?q=NVIDIA+nvCOMP+%28ANS%2C+Bitcomp%2C+GDeflate%2C+LZ4%29+documentation+and+benchmarks 10. Lossless compression of BF16 weights: DFloat11 and ZipNN — Zhang et al. (DFloat11); Hershcovich et al. (ZipNN), 2025 https://scholar.google.com/scholar?q=Lossless+compression+of+BF16+weights%3A+DFloat11+and+ZipNN 11. Mooncake: Trading more storage for less computation — Qin, R., et al., 2025 https://scholar.google.com/scholar?q=Mooncake%3A+Trading+more+storage+for+less+computation 12. KVQuant and KIVI — Hooper et al. 2024; Liu et al. 2024, 2024 https://scholar.google.com/scholar?q=KVQuant+and+KIVI Interactive Visualization: FastKVComp: Checking Whether High-Throughput KV Compression Pays Off
Nemotron-Labs-Diffusion: One Model, Three Decoding Modes
2026/11/07
This episode examines "Nemotron-Labs-Diffusion," an 8B model that runs in three modes from one set of weights: ordinary autoregressive decoding, block-diffusion parallel decoding, and self-speculation. In self-speculation, the diffusion pathway drafts tokens and the causal pathway verifies them, so no separate drafter is needed. It first explains why single-user decoding is memory-bound, then covers the prior approaches: masked and block diffusion, speculative decoding, and multi-token-prediction and Eagle3 drafters. A central theme is that the headline claims mix different metrics. Tokens per forward pass, acceptance rate and real hardware throughput are not interchangeable. A self-speculation cycle costs two forward passes, so true tokens per pass is roughly half the raw acceptance count. The episode also covers the training recipe, which uses a noisy block-bidirectional stream alongside a strictly causal clean stream. It notes that the "speed-of-light" analysis is an oracle upper bound, not a deployable sampler. Listeners get a clear framework for reading speedup claims in LLM inference papers critically. Sources: 1. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding — Yonggan Fu, Lexington Whalen, Abhinav Garg, Chengyue Wu, Maksim Khadkevich, Nicolai Oswald, Enze Xie, Daniel Egert, Sharath Turuvekere Sreenivas, Shizhe Diao, Chenhan Yu, Ye Yu, Weijia Chen, Sajad Norouzi, Jingyu Liu, Shiyi Lan, Ligeng Zhu, Jin Wang, Jindong Jiang, Morteza Mardani, Mehran Maghoumi, Song Han, Ante Jukić, Nima Tajbakhsh, Jan Kautz, Pavlo Molchanov, 2026 http://arxiv.org/abs/2607.05722 2. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2022 (ICML 2023) https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding 3. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (and the earlier EAGLE / EAGLE-2) — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024-2025 https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test+%28and+the+earlier+EAGLE+%2F+EAGLE-2%29 4. Large Language Diffusion Models (LLaDA) — Shen Nie, Fengqi Zhu, et al., 2025 https://scholar.google.com/scholar?q=Large+Language+Diffusion+Models+%28LLaDA%29 5. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (BD3-LMs) — Marianne Arriola, Subham Sekhar Sahoo, et al., 2025 (ICLR 2025) https://scholar.google.com/scholar?q=Block+Diffusion%3A+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models+%28BD3-LMs%29 6. Better & Faster Large Language Models via Multi-token Prediction — Fabian Gloeckle, Badr Youbi Idrissi, et al. (Meta FAIR), 2024 https://scholar.google.com/scholar?q=Better+%26+Faster+Large+Language+Models+via+Multi-token+Prediction 7. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Yuhui Li et al., 2025 https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test 8. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (BD3-LM) — Marianne Arriola et al., 2025 https://scholar.google.com/scholar?q=Block+Diffusion%3A+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models+%28BD3-LM%29 9. SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation — Shuang Cheng et al., 2025 https://scholar.google.com/scholar?q=SDAR%3A+A+Synergistic+Diffusion-AutoRegression+Paradigm+for+Scalable+Sequence+Generation 10. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding — Chengyue Wu et al., 2025 https://scholar.google.com/scholar?q=Fast-dLLM%3A+Training-free+Acceleration+of+Diffusion+LLM+by+Enabling+KV+Cache+and+Parallel+Decoding 11. Dream 7B: Diffusion Large Language Models — Jiacheng Ye et al., 2025 https://scholar.google.com/scholar?q=Dream+7B%3A+Diffusion+Large+Language+Models 12. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai et al., 2024 https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads 13. DeepSeek-V3 Technical Report — DeepSeek-AI, 2024 https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report 14. Self Speculative Decoding / Draft & Verify (layer skipping) — Jun Zhang et al., 2023 https://scholar.google.com/scholar?q=Self+Speculative+Decoding+%2F+Draft+%26+Verify+%28layer+skipping%29 15. The Pitfalls of Next-Token Prediction — Gregor Bachmann, Vaishnavh Nagarajan, 2024 https://scholar.google.com/scholar?q=The+Pitfalls+of+Next-Token+Prediction 16. SPEED-Bench — Cited as [1] in the paper, 2026 https://scholar.google.com/scholar?q=SPEED-Bench 17. Diffusion Language Models are Super Data Learners — Jinjie Ni et al., 2025 https://scholar.google.com/scholar?q=Diffusion+Language+Models+are+Super+Data+Learners Interactive Visualization: Nemotron-Labs-Diffusion: One Model, Three Decoding Modes
Cascade: SLO-Aware Latency Budgets for Fair, High-Goodput LLM Serving
2026/11/06
This episode examines Cascade, a serving system that uses a single "latency budget" (a request's time-to-first-token target minus its predicted service time) to drive both queue ordering and KV cache restoration decisions. It explains why first-come-first-served, shortest-job-first and earliest-deadline-first each fail. The failures are head-of-line blocking, starved long prompts, and an inability to tell a 50 ms job from a 20 s job with the same deadline. The discussion covers goodput, tiered prefix caching across GPU memory, CPU DRAM and NVMe, and chunked prefill. It also covers the paper's motivating findings: aggregate SLO attainment hides starved classes, deep-tier restores such as a 64,000-token NVMe fetch can blow the latency target, and most violations come from queueing rather than a lack of budget. Listeners get a clear walkthrough of the scheduling vocabulary, Jain's fairness index and the budget estimator. They also get a caveat that the headline cluster results come from an extended simulator fed with GB200 NVL72 data-movement profiles rather than a real deployment. Sources: 1. Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving — Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse, 2026 http://arxiv.org/abs/2608.06557 2. Orca: A Distributed Serving System for Transformer-Based Generative Models — Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun, 2022 https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models 3. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024 https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve 4. Fast Distributed Inference Serving for Large Language Models (FastServe) — Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, Xin Jin, 2023 https://scholar.google.com/scholar?q=Fast+Distributed+Inference+Serving+for+Large+Language+Models+%28FastServe%29 5. Efficient LLM Scheduling by Learning to Rank — Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, Hao Zhang, 2024 https://scholar.google.com/scholar?q=Efficient+LLM+Scheduling+by+Learning+to+Rank 6. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024 https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving 7. Fairness in Serving Large Language Models — Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, Ion Stoica, 2024 https://scholar.google.com/scholar?q=Fairness+in+Serving+Large+Language+Models 8. A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems — Raj Jain, Dah-Ming Chiu, William R. Hawe, 1984 https://scholar.google.com/scholar?q=A+Quantitative+Measure+of+Fairness+and+Discrimination+for+Resource+Allocation+in+Shared+Computer+Systems 9. Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services — Jiachen Liu, Zhiyu Wu, Jae-Won Chung, Fan Lai, Myungjin Lee, Mosharaf Chowdhury, 2024 https://scholar.google.com/scholar?q=Andes%3A+Defining+and+Enhancing+Quality-of-Experience+in+LLM-Based+Text+Streaming+Services 10. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — Agrawal, Kedia, Panwar, Mohan, Kwatra, Gulavani, Tumanov, Ramjee, 2024 https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference 11. Fairness in Serving Large Language Models (Virtual Token Counter, VTC) — Sheng, Cao, Li, Zhuang, Zhang, Gonzalez, Stoica et al., 2024 https://scholar.google.com/scholar?q=Fairness+in+Serving+Large+Language+Models+%28Virtual+Token+Counter%2C+VTC%29 12. QoServe: Breaking the Silos of LLM Inference Serving — Goel, Mohan, Kwatra, Anupindi, Ramjee, 2026 https://scholar.google.com/scholar?q=QoServe%3A+Breaking+the+Silos+of+LLM+Inference+Serving 13. SLOs-Serve: Optimized Serving of Multi-SLO LLMs — Chen, Jia, Khan, Krishnamurthy, Gibbons, 2025 https://scholar.google.com/scholar?q=SLOs-Serve%3A+Optimized+Serving+of+Multi-SLO+LLMs 14. JITServe: SLO-aware LLM Serving with Imprecise Request Information — Zhang, Wu, Mu, Ning, Liu, Sarda, Lee, Lai, 2025 https://scholar.google.com/scholar?q=JITServe%3A+SLO-aware+LLM+Serving+with+Imprecise+Request+Information 15. FastServe: Iteration-Level Preemptive Scheduling for LLM Inference (MLFQ) — Wu, Zhong, Zhang, Liu, Liu, Sun, Huang, Liu, Jin, 2026 (NSDI) https://scholar.google.com/scholar?q=FastServe%3A+Iteration-Level+Preemptive+Scheduling+for+LLM+Inference+%28MLFQ%29 16. Mooncake: Trading More Storage for Less Computation — Qin, Li, He, Cui, Ren, Zhang, Wu, Zheng, Xu, 2025 https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation 17. Tutti: Making SSD-backed KV Cache Practical for Long-Context LLM Serving — Qiu, Hu, Wang, Zhu, Yan, Chen, Xu, Chen, Zhang, 2026 https://scholar.google.com/scholar?q=Tutti%3A+Making+SSD-backed+KV+Cache+Practical+for+Long-Context+LLM+Serving 18. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Liu, Cheng, Yao, An, Chen, Feng, Huang, Shen, Zhang, Du, Jiang, 2025 https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference 19. Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment (and least-laxity-first literature, e.g. Mok 1983) — Liu and Layland (1973); Mok, 1973 / 1983 https://scholar.google.com/scholar?q=Scheduling+Algorithms+for+Multiprogramming+in+a+Hard-Real-Time+Environment+%28and+least-laxity-first+literature%2C+e.g.+Mok+1983%29 20. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized LLM Serving — Zhong, Liu, Chen, Hu, Zhu, Liu, Jin, Zhang, 2024 https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-Optimized+LLM+Serving Interactive Visualization: Cascade: SLO-Aware Latency Budgets for Fair, High-Goodput LLM Serving
Genuine 1-Bit Post-Training Quantization for LLMs: All for 1-Bit
2026/11/05
This episode examines "All for 1-Bit," a paper claiming to compress LLM linear weights to exactly one bit each, metadata included, without retraining. It builds on double binary factorization and challenges earlier "1-bit" post-training methods, which land at roughly 2.5 to 3.5 effective bits per weight once scales, masks and outlier indices are counted. The discussion covers why bandwidth-bound decoding motivates extreme quantization, how a product of two binary matrices adds capacity without adding bits, and how null-space reasoning and Shapley-value layer allocation are meant to spend a fixed rank budget well. Along the way it separates post-training from quantization-aware training and sets up the key skeptical test: whether "genuine" and "usable" both hold, given that a score near chance (about 25 on the zero-shot scale) means a broken model. Listeners get a clear explanation of what "bits per weight" really measures and a framework for judging low-bit quantization claims. Sources: 1. All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs — Zhixiong Zhao, Zukang Xu, Guangyu Sun, Lifeng Liu, Dawei Yang, 2026 http://arxiv.org/abs/2609.06161 2. BiLLM: Pushing the Limit of Post-Training Quantization for LLMs — Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, Xiaojuan Qi, 2024 https://scholar.google.com/scholar?q=BiLLM%3A+Pushing+the+Limit+of+Post-Training+Quantization+for+LLMs 3. ARB-LLM: Alternating Refined Binarizations for Large Language Models — Zhiteng Li, Xianglong Yan, Tianao Zhang, Haotong Qin, Dong Xie, Jiang Tian, Zhongchao Shi, Linghe Kong, Yulun Zhang, Xiaokang Yang, 2024 https://scholar.google.com/scholar?q=ARB-LLM%3A+Alternating+Refined+Binarizations+for+Large+Language+Models 4. OneBit: Towards Extremely Low-bit Large Language Models — Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, Wanxiang Che, 2024 https://scholar.google.com/scholar?q=OneBit%3A+Towards+Extremely+Low-bit+Large+Language+Models 5. BitNet: Scaling 1-bit Transformers for Large Language Models / The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits — Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei (BitNet); Shuming Ma et al. (b1.58), 2023 / 2024 https://scholar.google.com/scholar?q=BitNet%3A+Scaling+1-bit+Transformers+for+Large+Language+Models+%2F+The+Era+of+1-bit+LLMs%3A+All+Large+Language+Models+are+in+1.58+Bits 6. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2022 https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers 7. SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression — Xin Wang, Yu Zheng, Zhongwei Wan, Mi Zhang, 2024 https://scholar.google.com/scholar?q=SVD-LLM%3A+Truncation-aware+Singular+Value+Decomposition+for+Large+Language+Model+Compression 8. AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models — Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, Tat-Seng Chua, 2024 https://scholar.google.com/scholar?q=AlphaEdit%3A+Null-Space+Constrained+Knowledge+Editing+for+Language+Models 9. Training Networks in Null Space of Feature Covariance for Continual Learning — Shipeng Wang, Xiaorong Li, Jian Sun, Zongben Xu, 2021 https://scholar.google.com/scholar?q=Training+Networks+in+Null+Space+of+Feature+Covariance+for+Continual+Learning 10. A Unified Approach to Interpreting Model Predictions (SHAP) — Scott M. Lundberg, Su-In Lee, 2017 https://scholar.google.com/scholar?q=A+Unified+Approach+to+Interpreting+Model+Predictions+%28SHAP%29 11. Data Shapley: Equitable Valuation of Data for Machine Learning — Amirata Ghorbani, James Zou, 2019 https://scholar.google.com/scholar?q=Data+Shapley%3A+Equitable+Valuation+of+Data+for+Machine+Learning 12. HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks — Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, Kurt Keutzer, 2020 https://scholar.google.com/scholar?q=HAWQ-V2%3A+Hessian+Aware+trace-Weighted+Quantization+of+Neural+Networks 13. SqueezeLLM: Dense-and-Sparse Quantization — Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer, 2023 https://scholar.google.com/scholar?q=SqueezeLLM%3A+Dense-and-Sparse+Quantization 14. Addition is almost all you need: Compressing neural networks with double binary factorization — Vladimír Boža, Vladimír Macko, 2025 https://scholar.google.com/scholar?q=Addition+is+almost+all+you+need%3A+Compressing+neural+networks+with+double+binary+factorization 15. LittleBit: Ultra Low-Bit Quantization via Latent Factorization — Lee et al., 2026 https://scholar.google.com/scholar?q=LittleBit%3A+Ultra+Low-Bit+Quantization+via+Latent+Factorization 16. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58) — Shuming Ma et al., 2024 https://scholar.google.com/scholar?q=The+Era+of+1-bit+LLMs%3A+All+Large+Language+Models+are+in+1.58+Bits+%28BitNet+b1.58%29 17. FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation — Chuang Ma et al., 2024 https://scholar.google.com/scholar?q=FBI-LLM%3A+Scaling+Up+Fully+Binarized+LLMs+from+Scratch+via+Autoregressive+Distillation 18. HBLLM: Haar-based approach for accurate structured 1-bit quantized LLMs — Chen et al., 2025 https://scholar.google.com/scholar?q=HBLLM%3A+Haar-based+approach+for+accurate+structured+1-bit+quantized+LLMs 19. AQLM: Extreme Compression of LLMs via Additive Quantization — Egiazarian et al., 2024 https://scholar.google.com/scholar?q=AQLM%3A+Extreme+Compression+of+LLMs+via+Additive+Quantization 20. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks — Tseng et al., 2024 https://scholar.google.com/scholar?q=QuIP%23%3A+Even+Better+LLM+Quantization+with+Hadamard+Incoherence+and+Lattice+Codebooks 21. QTIP: Quantization with Trellises and Incoherence Processing — Tseng et al., 2024 https://scholar.google.com/scholar?q=QTIP%3A+Quantization+with+Trellises+and+Incoherence+Processing 22. Extreme Compression of Large Language Models via Additive Quantization / PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression — Malinovskii et al., 2024 https://scholar.google.com/scholar?q=Extreme+Compression+of+Large+Language+Models+via+Additive+Quantization+%2F+PV-Tuning%3A+Beyond+Straight-Through+Estimation+for+Extreme+LLM+Compression 23. SVD-LLM: Truncation-aware Singular Value Decomposition for LLM Compression — Xin Wang et al., 2024 https://scholar.google.com/scholar?q=SVD-LLM%3A+Truncation-aware+Singular+Value+Decomposition+for+LLM+Compression 24. BitNet a4.8 / bitnet.cpp: Efficient edge inference for ternary LLMs — Microsoft Research, 2024-2025 https://scholar.google.com/scholar?q=BitNet+a4.8+%2F+bitnet.cpp%3A+Efficient+edge+inference+for+ternary+LLMs 25. ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization — Zechun Liu et al., 2025 https://scholar.google.com/scholar?q=ParetoQ%3A+Scaling+Laws+in+Extremely+Low-bit+LLM+Quantization Interactive Visualization: Genuine 1-Bit Post-Training Quantization for LLMs: All for 1-Bit
Adaptive Mixed-Precision Computing Speeds Up Nearest Neighbor Search
2026/11/05
This episode examines ANNS-AMP, a June 2026 paper proposing adaptive mixed-precision computing for approximate nearest neighbor search. It starts with the IVF-PQ index pipeline: cluster locating, residual calculation, lookup-table construction, distance summation and top-k selection. The paper reports that over 99.9% of distance arithmetic in the cluster-locating and table-construction stages goes to vectors that are not neighbors. The proposed fix is a small runtime predictor that picks the bit-width for each cluster and each query, paired with a bit-serial, bit-plane near-memory accelerator so that fewer bits mean less compute and less memory traffic. The discussion also covers why existing index tuning and early-termination methods miss this redundancy. It questions whether flexible-precision hardware can succeed now that fixed INT8, FP8 and FP4 formats have taken over in neural networks. A running thread asks how much of the reported speed and energy gain comes from the precision idea and how much from the custom silicon. Sources: 1. ANNS-AMP: Accelerating Approximate Nearest Neighbor Search via Adaptive Mixed-Precision Computing — Mingkai Chen, Cheng Liu, Shengwen Liang, Lei Zhang, Xiaowei Li, Huawei Li, 2026 http://arxiv.org/abs/2606.07156 2. Mixed Precision Training — Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, et al., 2018 https://scholar.google.com/scholar?q=Mixed+Precision+Training 3. HAQ: Hardware-Aware Automated Quantization with Mixed Precision — Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, Song Han, 2019 https://scholar.google.com/scholar?q=HAQ%3A+Hardware-Aware+Automated+Quantization+with+Mixed+Precision 4. HAWQ: Hessian Aware Quantization of Neural Networks with Mixed-Precision — Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, Kurt Keutzer, 2019 https://scholar.google.com/scholar?q=HAWQ%3A+Hessian+Aware+Quantization+of+Neural+Networks+with+Mixed-Precision 5. Stripes: Bit-Serial Deep Neural Network Computing — Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor Aamodt, Andreas Moshovos, 2016 https://scholar.google.com/scholar?q=Stripes%3A+Bit-Serial+Deep+Neural+Network+Computing 6. Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks — Hardik Sharma, Jongse Park, Naveen Suda, Liangzhen Lai, Benson Chau, Vikas Chandra, Hadi Esmaeilzadeh, 2018 https://scholar.google.com/scholar?q=Bit+Fusion%3A+Bit-Level+Dynamically+Composable+Architecture+for+Accelerating+Deep+Neural+Networks 7. BISMO: A Scalable Bit-Serial Matrix Multiplication Overlay for Reconfigurable Computing — Yaman Umuroglu, Lahiru Rasnayake, Magnus Sjalander, 2018 https://scholar.google.com/scholar?q=BISMO%3A+A+Scalable+Bit-Serial+Matrix+Multiplication+Overlay+for+Reconfigurable+Computing 8. Product Quantization for Nearest Neighbor Search — Hervé Jégou, Matthijs Douze, Cordelia Schmid, 2011 https://scholar.google.com/scholar?q=Product+Quantization+for+Nearest+Neighbor+Search 9. Billion-scale similarity search with GPUs — Jeff Johnson, Matthijs Douze, Hervé Jégou, 2017 (IEEE Transactions on Big Data, 2019) https://scholar.google.com/scholar?q=Billion-scale+similarity+search+with+GPUs 10. Accelerating Large-Scale Inference with Anisotropic Vector Quantization — Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar, 2020 https://scholar.google.com/scholar?q=Accelerating+Large-Scale+Inference+with+Anisotropic+Vector+Quantization 11. Co-design Hardware and Algorithm for Vector Search (FANNS) — Wenqi Jiang, Shigang Li, Yu Zhu, Johannes de Fine Licht, Zhenhao He, Runbin Shi, Cédric Renggli, Shuai Zhang, Theodoros Rekatsas, Torsten Hoefler, Gustavo Alonso, 2023 https://scholar.google.com/scholar?q=Co-design+Hardware+and+Algorithm+for+Vector+Search+%28FANNS%29 12. ANSMET: Approximate Nearest Neighbor Search with Near-Memory Processing and Hybrid Early Termination — Yiwei Li, Yuxin Jin, Boyu Tian, Huanchen Zhang, Mingyu Gao, 2025 https://scholar.google.com/scholar?q=ANSMET%3A+Approximate+Nearest+Neighbor+Search+with+Near-Memory+Processing+and+Hybrid+Early+Termination 13. ANNA: Specialized Architecture for Approximate Nearest Neighbor Search — Yejin Lee, Hyunji Choi, Sunhong Min, Hyunseung Lee, Sangwon Beak, Dawoon Jeong, Jae W. Lee, Tae Jun Ham, 2022 https://scholar.google.com/scholar?q=ANNA%3A+Specialized+Architecture+for+Approximate+Nearest+Neighbor+Search 14. JUNO: Optimizing High-Dimensional Approximate Nearest Neighbour Search with Sparsity-Aware Algorithm and Ray-Tracing Core Mapping — Zihan Liu, Wentao Ni, Jingwen Leng, Yu Feng, Cong Guo, Quan Chen, Chao Li, Minyi Guo, Yuhao Zhu, 2024 https://scholar.google.com/scholar?q=JUNO%3A+Optimizing+High-Dimensional+Approximate+Nearest+Neighbour+Search+with+Sparsity-Aware+Algorithm+and+Ray-Tracing+Core+Mapping 15. RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search — Jianyang Gao, Cheng Long, 2024 https://scholar.google.com/scholar?q=RaBitQ%3A+Quantizing+High-Dimensional+Vectors+with+a+Theoretical+Error+Bound+for+Approximate+Nearest+Neighbor+Search 16. High-Dimensional Approximate Nearest Neighbor Search: with Reliable and Efficient Distance Comparison Operations (ADSampling) — Jianyang Gao, Cheng Long, 2023 https://scholar.google.com/scholar?q=High-Dimensional+Approximate+Nearest+Neighbor+Search%3A+with+Reliable+and+Efficient+Distance+Comparison+Operations+%28ADSampling%29 17. Quicker ADC: Unlocking the Hidden Potential of Product Quantization with SIMD (and Fast Scan variants in Faiss) — Fabien André, Anne-Marie Kermarrec, Nicolas Le Scouarnec, 2019 https://scholar.google.com/scholar?q=Quicker+ADC%3A+Unlocking+the+Hidden+Potential+of+Product+Quantization+with+SIMD+%28and+Fast+Scan+variants+in+Faiss%29 Interactive Visualization: Adaptive Mixed-Precision Computing Speeds Up Nearest Neighbor Search
PagedMoE: Cross-Precision Expert Caching for SSD-Resident MoE Inference
2026/11/05
This episode examines PagedMoE, a system for running a huge mixture-of-experts model (Qwen3.5-397B-A17B) on a single PC with expert weights stored on an SSD. Cache misses leave decoding stalled on storage nearly half the time, and giving each token a different bit-width per expert would normally make that worse. Treating every expert-and-precision pair as its own cache object drops the hit rate from 57% to 35%. The paper's fix is to store each expert as nested refinement units, so a resident low-precision copy can be topped up with only the missing bits, or reused with no I/O at all. The discussion separates this idea from the prior work it borrows from (Matryoshka Quantization, Any-Precision LLM, SliceMoE, HOBBIT). It also splits the reported 1.56x speedup into the part from requesting fewer bits and the part from sharing cached bits across precisions. Listeners get a clear walk through the binary-tree codebook construction, the quantization-error numbers, and the limits of the evidence, such as a single-layer headroom study and no head-to-head comparison with SliceMoE. Sources: 1. PagedMoE: Cross-Precision Expert Caching for Token-Level Dynamic Precision in SSD-Resident MoE Inference — Yutong Lin, Yingdi Shan, Shaoyuan Chen, Yuanyong Chen, Yongwei Wu, Mingxing Zhang, 2026 http://arxiv.org/abs/2611.03173 2. Fast Inference of Mixture-of-Experts Language Models with Offloading — Artyom Eliseev, Denis Mazur, 2023 https://scholar.google.com/scholar?q=Fast+Inference+of+Mixture-of-Experts+Language+Models+with+Offloading 3. MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache — Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, Mahesh Marina, 2024 (arXiv; later revisions) https://scholar.google.com/scholar?q=MoE-Infinity%3A+Efficient+MoE+Inference+on+Personal+Machines+with+Sparsity-Aware+Expert+Cache 4. EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices — Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, Mengwei Xu, 2023 https://scholar.google.com/scholar?q=EdgeMoE%3A+Empowering+Sparse+Large+Language+Models+on+Mobile+Devices 5. LLM in a Flash: Efficient Large Language Model Inference with Limited Memory — Keivan Alizadeh, Iman Mirzadeh, et al. (Apple), 2023 https://scholar.google.com/scholar?q=LLM+in+a+Flash%3A+Efficient+Large+Language+Model+Inference+with+Limited+Memory 6. HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference — Peng Tang, Jiacheng Liu, Xiaofeng Hou, et al., 2024 https://scholar.google.com/scholar?q=HOBBIT%3A+A+Mixed+Precision+Expert+Offloading+System+for+Fast+MoE+Inference 7. Mixture Compressor for Mixture-of-Experts LLMs Gains More (MC) — Wei Huang, Yue Liao, Jianhui Liu, et al., 2024 https://scholar.google.com/scholar?q=Mixture+Compressor+for+Mixture-of-Experts+LLMs+Gains+More+%28MC%29 8. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2022 https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers 9. Microscaling Data Formats for Deep Learning (OCP MX specification) — Bita Darvish Rouhani et al. (Microsoft, AMD, Arm, Intel, Meta, NVIDIA, Qualcomm), 2023 https://scholar.google.com/scholar?q=Microscaling+Data+Formats+for+Deep+Learning+%28OCP+MX+specification%29 10. Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs — Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, Jae W. Lee, 2024 https://scholar.google.com/scholar?q=Any-Precision+LLM%3A+Low-Cost+Deployment+of+Multiple%2C+Different-Sized+LLMs 11. Matryoshka Quantization — Pranav Nair, Puranjay Datta, Jeff Dean, Prateek Jain, Aditya Kusupati, 2025 https://scholar.google.com/scholar?q=Matryoshka+Quantization 12. QTIP: Quantization with Trellises and Incoherence Processing — Albert Tseng, Qingyao Sun, David Hou, Christopher De Sa, 2024 https://scholar.google.com/scholar?q=QTIP%3A+Quantization+with+Trellises+and+Incoherence+Processing 13. Stripes: Bit-Serial Deep Neural Network Computing — Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor Aamodt, Andreas Moshovos, 2016 https://scholar.google.com/scholar?q=Stripes%3A+Bit-Serial+Deep+Neural+Network+Computing 14. SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference — Yuseon Choi, Sangjin Kim, Jungjun Oh, Gwangtae Park, Byeongcheol Kim, Hoi-Jun Yoo, 2026 https://scholar.google.com/scholar?q=SliceMoE%3A+Bit-Sliced+Expert+Caching+under+Miss-Rate+Constraints+for+Efficient+MoE+Inference 15. D²MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving — Haodong Wang, Qihua Zhou, Zicong Hong, Song Guo, 2025 https://scholar.google.com/scholar?q=D%C2%B2MoE%3A+Dual+Routing+and+Dynamic+Scheduling+for+Efficient+On-Device+MoE-based+LLM+Serving 16. DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge — Yuegui Huang, Zhiyuan Fang, Weiqi Luo, Ruoyu Wu, Wuhui Chen, Zibin Zheng, 2026 https://scholar.google.com/scholar?q=DyMoE%3A+Dynamic+Expert+Orchestration+with+Mixed-Precision+Quantization+for+Efficient+MoE+Inference+on+Edge 17. MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design — Haojie Duanmu, Xiuhong Li, Zhihang Yuan, Size Zheng, Jiangfei Duan, Xingcheng Zhang, Dahua Lin, 2025 https://scholar.google.com/scholar?q=MxMoE%3A+Mixed-precision+Quantization+for+MoE+with+Accuracy+and+Performance+Co-Design 18. GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs — Jianing Deng, Song Wang, Dongwei Wang, Zijie Liu, Tianlong Chen, Huanrui Yang, Jingtong Hu, 2026 https://scholar.google.com/scholar?q=GEMQ%3A+Global+Expert-Level+Mixed-Precision+Quantization+for+MoE+LLMs 19. KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models — Hongtao Chen, Weiyu Xie, Boxin Zhang, et al., 2025 https://scholar.google.com/scholar?q=KTransformers%3A+Unleashing+the+Full+Potential+of+CPU%2FGPU+Hybrid+Inference+for+MoE+Models 20. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, Mao Yang, 2025 https://scholar.google.com/scholar?q=T-MAC%3A+CPU+Renaissance+via+Table+Lookup+for+Low-Bit+LLM+Deployment+on+Edge 21. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng et al., 2023 https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU 22. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 23. Speculative MoE / expert prefetching works (e.g., Pre-gated MoE, ProMoE, AdapMoE) — Various, 2024–2025 https://scholar.google.com/scholar?q=Speculative+MoE+%2F+expert+prefetching+works+%28e.g.%2C+Pre-gated+MoE%2C+ProMoE%2C+AdapMoE%29 Interactive Visualization: PagedMoE: Cross-Precision Expert Caching for SSD-Resident MoE Inference
LeanSparse: Shrinking GPU Memory for Sparse-Attention LLM Serving
2026/11/04
This episode examines LeanSparse, a serving system that cuts the GPU memory needed to hold many long-context conversations on models with native sparse attention, such as DeepSeek-V4-Flash. It explains why sparse attention reduces reads but not storage. Even with KV offloaded to host memory, the lightning indexer's keys must stay on the GPU, and at 512K tokens they take about 7.2 times the space of the hot KV buffer. It also covers why agent-style workloads, where context grows every turn, make this memory limit bite, and why prefill needs a large temporary workspace. That workspace forces a bad choice between reserving memory that sits idle and making new requests queue behind decoders, which hurts time to first token. The proposed fixes are to stream each layer's history through two alternating GPU buffers during prefill, and to search compact indexer keys during decode, fetching only a short list of originals to rank exactly. All of this runs on a shared instance, avoiding the duplicated weights of prefill-decode disaggregation. Listeners get a clear walkthrough of where memory goes in modern sparse-attention serving, plus the reasoning behind why the obvious fix of offloading the indexer keys fails. Sources: 1. LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs — Zaifeng Pan, Chris Wu, Xinwei Qiang, Zhengding Hu, Zhongkai Yu, Yufei Ding, 2026 http://arxiv.org/abs/2611.04671 2. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan, Huazuo Gao, Damai Dai, et al. (DeepSeek-AI, Peking University), 2025 https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention 3. DeepSeek-V3.2-Exp technical report (DeepSeek Sparse Attention, DSA) — DeepSeek-AI, 2025 https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp+technical+report+%28DeepSeek+Sparse+Attention%2C+DSA%29 4. MoBA: Mixture of Block Attention for Long-Context LLMs — Enzhe Lu, Zhejun Jiang, Jingyuan Liu, et al. (Moonshot AI), 2025 https://scholar.google.com/scholar?q=MoBA%3A+Mixture+of+Block+Attention+for+Long-Context+LLMs 5. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024 https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference 6. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024 https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management 7. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference — Hanshi Sun, Li-Wen Chang, Wenlei Bao, et al., 2024 https://scholar.google.com/scholar?q=ShadowKV%3A+KV+Cache+in+Shadows+for+High-Throughput+Long-Context+LLM+Inference 8. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval — Di Liu, Meng Chen, Baotong Lu, et al., 2024 https://scholar.google.com/scholar?q=RetrievalAttention%3A+Accelerating+Long-Context+LLM+Inference+via+Vector+Retrieval 9. Billion-scale similarity search with GPUs (FAISS) — Jeff Johnson, Matthijs Douze, Hervé Jégou, 2017 https://scholar.google.com/scholar?q=Billion-scale+similarity+search+with+GPUs+%28FAISS%29 10. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, et al. (Moonshot AI, Tsinghua), 2024 https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving 11. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, et al., 2024 https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving 12. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, et al., 2024 https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve 13. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, et al., 2023 https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU 14. PQCache: Product Quantization-based KVCache for Long Context LLM Inference — Hailin Zhang et al., 2025 https://scholar.google.com/scholar?q=PQCache%3A+Product+Quantization-based+KVCache+for+Long+Context+LLM+Inference 15. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (DeepSeek Sparse Attention) — DeepSeek-AI, 2025 https://scholar.google.com/scholar?q=DeepSeek-V3.2%3A+Pushing+the+Frontier+of+Open+Large+Language+Models+%28DeepSeek+Sparse+Attention%29 16. CachedAttention (AttentionStore): Cost-Efficient LLM Serving for Multi-Turn Conversations — Bin Gao et al., 2024 https://scholar.google.com/scholar?q=CachedAttention+%28AttentionStore%29%3A+Cost-Efficient+LLM+Serving+for+Multi-Turn+Conversations 17. SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving — Zhou et al., 2025 https://scholar.google.com/scholar?q=SparseServe%3A+Unlocking+Parallelism+for+Dynamic+Sparse+Attention+in+Long-Context+LLM+Serving 18. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — Agrawal et al., 2024 https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference 19. Product Quantization for Nearest Neighbor Search — Hervé Jégou, Matthijs Douze, Cordelia Schmid, 2011 https://scholar.google.com/scholar?q=Product+Quantization+for+Nearest+Neighbor+Search Interactive Visualization: LeanSparse: Shrinking GPU Memory for Sparse-Attention LLM Serving
Joules-to-GDP: Economic Value Produced Per Unit of AI Energy
2026/11/04
This episode examines "Joules-to-GDP," a Stanford-led paper that asks how much economic value AI systems produce per joule of energy when doing real professional work. It lays out the setup: 220 public GDPval tasks across 44 occupations, expert completion time multiplied by Bureau of Labor Statistics median wages as the dollar value, and a small LLM judge deciding whether the AI's deliverable beats the expert's. The central metrics are Net Value per Joule and Time Saving per Joule. Both are measured against an expert-only baseline and a tougher expert-with-AI baseline. The discussion also covers how energy is metered across a whole batch and split among concurrent tasks, and how running a small model across eight GPUs can waste power. It flags a confound: one model ran on H200s rather than H100s, so differences between models may reflect hardware rather than capability. Listeners get a sceptical look at whether ranking AI by energy efficiency gives a different answer from ranking by capability, cost or speed. Sources: 1. Joules-to-GDP: Economic Value Produced Per Unit of AI Energy https://arxiv.org/pdf/2610.30858 2. GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks — Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, et al. (OpenAI), 2025 https://scholar.google.com/scholar?q=GDPval%3A+Evaluating+AI+Model+Performance+on+Real-World+Economically+Valuable+Tasks 3. The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization — Jae-Won Chung, Jeff J. Ma, Ruofan Wu, Jiachen Liu, Oh Jun Kweon, Yuxuan Xia, Zhiyu Wu, Mosharaf Chowdhury, 2025 https://scholar.google.com/scholar?q=The+ML.ENERGY+Benchmark%3A+Toward+Automated+Inference+Energy+Measurement+and+Optimization 4. From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference — Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, Vijay Gadepally, 2023 https://scholar.google.com/scholar?q=From+Words+to+Watts%3A+Benchmarking+the+Energy+Costs+of+Large+Language+Model+Inference 5. Energy and Policy Considerations for Deep Learning in NLP — Emma Strubell, Ananya Ganesh, Andrew McCallum, 2019 https://scholar.google.com/scholar?q=Energy+and+Policy+Considerations+for+Deep+Learning+in+NLP 6. The Simple Macroeconomics of AI — Daron Acemoglu, 2024 https://scholar.google.com/scholar?q=The+Simple+Macroeconomics+of+AI 7. Cost-of-Pass: An Economic Framework for Evaluating Language Models — Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, James Zou, 2026 https://scholar.google.com/scholar?q=Cost-of-Pass%3A+An+Economic+Framework+for+Evaluating+Language+Models 8. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, et al., 2025 https://scholar.google.com/scholar?q=Intelligence+per+Watt%3A+Measuring+Intelligence+Efficiency+of+Local+AI 9. MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts — Arya Tschand et al., 2024 https://scholar.google.com/scholar?q=MLPerf+Power%3A+Benchmarking+the+Energy+Efficiency+of+Machine+Learning+Systems+from+Microwatts+to+Megawatts 10. Economic Evaluations of Language Models — Alexander Wan, Stephane Hatgis-Kessell, Tomás Aguirre, Percy Liang, Rishi Bommasani, 2026 https://scholar.google.com/scholar?q=Economic+Evaluations+of+Language+Models 11. Remote Labor Index: Measuring AI Automation of Remote Work — Mantas Mazeika et al., 2025 https://scholar.google.com/scholar?q=Remote+Labor+Index%3A+Measuring+AI+Automation+of+Remote+Work 12. Power Hungry Processing: Watts Driving the Cost of AI Deployment? — Alexandra Sasha Luccioni, Yacine Jernite, Emma Strubell, 2024 https://scholar.google.com/scholar?q=Power+Hungry+Processing%3A+Watts+Driving+the+Cost+of+AI+Deployment%3F 13. AI Agents That Matter — Sayash Kapoor et al., 2024 https://scholar.google.com/scholar?q=AI+Agents+That+Matter 14. Generative AI at Work — Erik Brynjolfsson, Danielle Li, Lindsey R. Raymond, 2025 https://scholar.google.com/scholar?q=Generative+AI+at+Work 15. Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms — Prabhu Vellaisamy et al., 2026 https://scholar.google.com/scholar?q=Characterization+of+Request+and+Token+Energy+Costs+for+LLM+Inference+Workloads+on+GPU+Platforms Interactive Visualization: Joules-to-GDP: Economic Value Produced Per Unit of AI Energy
Intelligence per Watt: Measuring Local AI Efficiency Against the Cloud
2026/11/04
This episode examines "Intelligence per Watt," a Stanford and Together AI paper that asks whether small local models on laptop-class hardware can take real query load off the cloud. It walks through the paper's definitions. "Local" means at most 20 billion active parameters per token, which includes mixture-of-experts models like gpt-oss-120B. The proposed metric is average task accuracy divided by average power draw, and the paper also reports an energy-per-query (joules) variant. The discussion points out that watts alone ignore latency, so a low-power device that runs slowly can still cost more energy per query. It also flags the measurement caveats: power is read from the accelerator only, not the wall; queries run at batch size 1; chat accuracy is judged against a single reference model's answers; and "best-of-local" coverage is an oracle router that couldn't be deployed. The study spans about a million single-turn queries, more than twenty models and eight accelerators, and the episode places it against earlier LLM energy work such as "From Words to Watts" and MLPerf Power. Listeners get a careful guide to what the metric does and doesn't capture before the headline efficiency numbers are discussed. Sources: 1. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, J. Wes Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rebecca Joseph, Shlok Natarajan, Etash Kumar Guha, Shang Zhu, Ben Athiwaratkun, John Hennessy, Azalia Mirhoseini, Christopher Ré, 2025 http://arxiv.org/abs/2511.07885 2. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, et al. (Stanford, Together AI; senior authors incl. John Hennessy, Azalia Mirhoseini, Christopher Ré), 2025 (v7 revised Oct 2026) https://scholar.google.com/scholar?q=Intelligence+per+Watt%3A+Measuring+Intelligence+Efficiency+of+Local+AI 3. MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI — Arya Tschand, Vijay Janapa Reddi, et al. (MLCommons), 2024 https://scholar.google.com/scholar?q=MLPerf+Power%3A+Benchmarking+the+Energy+Efficiency+of+Machine+Learning+Systems+from+Microwatts+to+Megawatts+for+Sustainable+AI 4. Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model — Alexandra Sasha Luccioni, Sylvain Viguier, Anne-Laure Ligozat, 2022 https://scholar.google.com/scholar?q=Estimating+the+Carbon+Footprint+of+BLOOM%2C+a+176B+Parameter+Language+Model 5. From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference — Siddharth Samsi, Dan Zhao, Joseph McDonald, et al. (MIT Lincoln Laboratory), 2023 https://scholar.google.com/scholar?q=From+Words+to+Watts%3A+Benchmarking+the+Energy+Costs+of+Large+Language+Model+Inference 6. Small Language Models are the Future of Agentic AI — Peter Belcak, Greg Heinrich, Shizhe Diao, et al. (NVIDIA Research), 2025 https://scholar.google.com/scholar?q=Small+Language+Models+are+the+Future+of+Agentic+AI 7. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 8. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration — Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Song Han, et al., 2023 (MLSys 2024) https://scholar.google.com/scholar?q=AWQ%3A+Activation-aware+Weight+Quantization+for+On-Device+LLM+Compression+and+Acceleration 9. Energy Considerations of Large Language Model Inference and Efficiency Optimizations — Jared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, Emma Strubell, 2025 https://scholar.google.com/scholar?q=Energy+Considerations+of+Large+Language+Model+Inference+and+Efficiency+Optimizations 10. Part-time Power Measurements: nvidia-smi's Lack of Attention — Zeyu Yang, Karel Adamek, Wesley Armour, 2023 https://scholar.google.com/scholar?q=Part-time+Power+Measurements%3A+nvidia-smi%27s+Lack+of+Attention 11. Energy Use of AI Inference: Efficiency Pathways and Test-Time Compute — Felipe Oviedo et al., 2025 https://scholar.google.com/scholar?q=Energy+Use+of+AI+Inference%3A+Efficiency+Pathways+and+Test-Time+Compute 12. Minions: Cost-Efficient Collaboration Between On-Device and Cloud Language Models — Avanika Narayan, Dan Biderman, Sabri Eyuboglu, Avner May, Scott Linderman, James Zou, Christopher Ré, 2025 https://scholar.google.com/scholar?q=Minions%3A+Cost-Efficient+Collaboration+Between+On-Device+and+Cloud+Language+Models 13. RouteLLM: Learning to Route LLMs with Preference Data — Isaac Ong et al., 2025 https://scholar.google.com/scholar?q=RouteLLM%3A+Learning+to+Route+LLMs+with+Preference+Data 14. PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU — Yixin Song, Zeyu Mi, Haotong Xie, Haibo Chen, 2024 https://scholar.google.com/scholar?q=PowerInfer%3A+Fast+Large+Language+Model+Serving+with+a+Consumer-Grade+GPU 15. Green AI — Roy Schwartz, Jesse Dodge, Noah A. Smith, Oren Etzioni, 2020 https://scholar.google.com/scholar?q=Green+AI 16. Implications of Historical Trends in the Electrical Efficiency of Computing — Jonathan Koomey, Stephen Berard, Marla Sanchez, Henry Wong, 2010 https://scholar.google.com/scholar?q=Implications+of+Historical+Trends+in+the+Electrical+Efficiency+of+Computing Interactive Visualization: Intelligence per Watt: Measuring Local AI Efficiency Against the Cloud
ECHO: Lossless KV Cache Offloading for Sparse Attention LLMs
2026/11/04
This episode examines ECHO, a system from Shanghai Jiao Tong University and collaborators (OSDI 2026) that offloads the KV cache to host memory for serving sparse-attention models like DeepSeek-V3.2. It explains why sparse attention cuts compute but not storage. Cache size still grows linearly with context, so GPU memory limits batch size and leaves the sparse kernels underused. It then covers why naive recall over PCIe is slow: the top-k token set is known only after the lightning indexer has scored every earlier token. Prior prefetchers such as InfiniGen and FreeKV guess which tokens to fetch, which is lossy. ECHO instead guesses only when to fetch and then guarantees a recall of any missed token. To make this work under CUDA graph execution, it keeps its cache-management state in fixed-size integer tensors on the GPU. The discussion also stresses that the predictability ECHO relies on comes from one trained indexer, not from attention in general. It ends by questioning whether "lossless" is only a design property or something the paper actually measures. Sources: 1. ECHO: Lossless KV Cache Offloading for Sparse Attention LLMs https://www.usenix.org/system/files/osdi26-liu-guangda.pdf 2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 3. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024 https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management 4. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, et al., 2024 https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs 5. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, et al., 2023 https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU 6. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022 https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness 7. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024 https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision 8. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention — DeepSeek-AI, 2025 https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp%3A+Boosting+Long-Context+Efficiency+with+DeepSeek+Sparse+Attention 9. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, et al. (DeepSeek-AI and collaborators), 2025 https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention 10. FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference — Liu et al., 2025 https://scholar.google.com/scholar?q=FreeKV%3A+Boosting+KV+Cache+Retrieval+for+Efficient+LLM+Inference 11. ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction — Chen et al., 2024 https://scholar.google.com/scholar?q=ArkVale%3A+Efficient+Generative+LLM+Inference+with+Recallable+Key-Value+Eviction 12. SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving — Zhou et al., 2025 https://scholar.google.com/scholar?q=SparseServe%3A+Unlocking+Parallelism+for+Dynamic+Sparse+Attention+in+Long-Context+LLM+Serving 13. ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp — Chen et al., 2025 https://scholar.google.com/scholar?q=ESS%3A+An+Offload-Centric+Latent-Cache+Management+Architecture+for+DeepSeek-V3.2-Exp 14. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Tang et al., 2024 https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference 15. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (NSA) — Yuan et al. (DeepSeek-AI), 2025 https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention+%28NSA%29 16. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Cheng et al., 2025 https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference 17. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — Agrawal et al., 2024 https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference Interactive Visualization: ECHO: Lossless KV Cache Offloading for Sparse Attention LLMs

Podcast reviews

Read AI: post transformers podcast reviews


3.7 out of 5
3 reviews

Podcast sponsorship advertising

Start advertising on AI: post transformers relevant audience podcasts


What do you want to promote?