Advertise on podcast: Recsperts - Recommender Systems Experts

In episode 34 of Recsperts, I'm joined by Raghav Saboo, Staff Machine Learning Engineer at DoorDash and Tech Lead for Personalization and Search for New Verticals — groceries, convenience, retail, alcohol, pet supplies and more, beyond the original restaurant vertical. We discuss the particular challenges of personalized recommendation, ranking and search in quick commerce, recent trends in generative recommendations and the application of Semantic IDs to item ranking and query reformulation. Raghav's path into recommender systems started in chemical engineering before he moved into ML consulting, a Master's in Statistics, Machine Learning and Econometrics from Duke University, and building LLMs for new language launches on Amazon's Alexa AI, ahead of joining DoorDash. We start with the marketplace itself: DoorDash connects consumers, merchants and couriers, and growing it means balancing the interests of all three so that the platform stays healthy for everyone on it. Raghav walks me through how his team frames the consumer side around three pillars — familiarity (surfacing what a consumer already trusts), affordability (matching price sensitivity and timely deals) and novelty (introducing new items and categories without adding friction). From there we get into how DoorDash uses LLMs to build "memory blocks," structured natural-language representations of a consumer organized around semantic domains like dietary preference, pet ownership or trusted brands, and how these feed LLM-generated collections that get resolved into real items through embedding-based retrieval. We then turn to DoorDash's move to generative approaches, centered on Semantic IDs: hierarchical product identifiers learned through recursive clustering of item content embeddings, forming a taxonomy that captures attributes a human-built catalog structure might miss — as Raghav puts it, "within e-commerce, items really carry a lot of meaning." He walks me through two production use cases: replacing dozens of taxonomy-based dense features in the ranking model with Semantic ID n-gram aggregations while improving online metrics, and using Semantic IDs for query reformulation in search, letting the system traverse a learned hierarchy to refine or diversify a query. This connects to DoorDash's own paper on the topic and to a broader conversation about why search, recommendation and agentic ordering — DoorDash's own "Ask DoorDash" — are converging on a shared substrate of Semantic IDs and consumer memory, while today's app surfaces still need to grow more flexible for that convergence to feel seamless. We close with a preview of the RecSys 2026 tutorial "Recommender Systems in Delivery Platforms: Challenges, Solutions and Learnings," which Raghav is co-presenting with Wolt's Paavo Camps and myself, and his advice for navigating a field that reinvents itself every quarter: be honest about whether that pace suits you, use AI agents to filter what's worth your attention, and build the judgment to recognize dead ends early. Enjoy this enriching episode of RECSPERTS – Recommender Systems Experts.Don't forget to follow the podcast and please leave a review. (00:00) - Introduction (02:19) - RecSys 2026 Tutorial Preview (03:23) - About Raghav Saboo (10:47) - Working on Amazon Alexa AI (14:13) - About DoorDash (16:31) - Operating Model of a Multi-Sided Marketplace (20:45) - Affordability, Familiarity and Novelty (34:01) - Advantages of LLM-based Consumer and Item Profiles (46:53) - Generative Recommendations (59:21) - Semantic IDs for Item Ranking and Query Reformulation (01:19:49) - Agentic Shopping vs. Conversational RecSys (01:29:39) - Tutorial on Recommender Systems in Delivery Platforms (01:34:05) - Closing Remarks Click here to view the episode transcript. Links from the Episode: Raghav Saboo on LinkedInRaghav Saboo's SubstackUsing LLMs to Infer Grocery Preferences from Restaurant OrdersBuilding Ask DoorDash (Part 2): IntelligenceBridging Affordability, Familiarity, and Novelty (KDD 2025)Building a Unified Consumer Memory for Personalization at ScaleOffline LLMs, Online Personalization: Generating Carousels at DoorDashRecSys 2026 TutorialsWorkshop on Unified Search and Recommendation (USRW) 2026Papers: Xu et al. (2026): One Hierarchy, Two Systems - Semantic Product IDs for Discovery-Surface Ranking and Search-Page Query Reformulation (RecSys 2026, USRW Workshop)Xi et al. (2026): Mine and Refine - Optimizing Graded Relevance in E-commerce Semantic Search Retrieval (CIKM 2026, Applied Research Track)Chen et al. (2026): Joint Optimization of Relevance and Engagement in Multi-Task Ranking for E-Commerce with Efficient LLM Supervision (SIGIR Industry Track)Sinha et al. (2025): Mind the Gap - Bridging Behavioral Silos with LLMs in Multi-Vertical Recommendations (RecSys 2025 GenAI Workshop Talk)Das et al. (2024): Applications of LLMs in E-Commerce Search and Product Knowledge Graph - The DoorDash Case Study (WSDM 2024)General Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 33 of Recsperts, I speak with Joseph A. Konstan, Distinguished McKnight University Professor and Distinguished University Teaching Professor at the University of Minnesota, co-founder of GroupLens, and the very first General Chair of the ACM RecSys conference back in 2007. This year he returns in that role as General Co-Chair of the 20th RecSys in Minneapolis. We talk about what actually makes a recommendation useful, why the field is more than a machine learning application, and how the community around RecSys came into being. We start with what Joe cares about most: usefulness. He recalls the supermarket thought experiment of printing "buy bananas and bread" on every shopping cart and explains why the early systems were valuable because they were wrong a lot. A recommender that took ten people like you and often got it wrong was also often surprising when it got it right, whereas today's systems are wrong far less and useless far more. As Joe puts it, "I don't care about prediction at all. I care about changing people's behavior." We discuss why optimizing for click-through in news reliably produces clickbait, why leave-one-out evaluation only makes sense if you assume the user already knew the right answer and had simply forgotten it, and why usefulness can never be read off a single metric but depends on the task, the context and the breadth of the user's intent. Sometimes the most useful thing is not the recommendation itself, but the stars, the reviews or the comparison table you put around it. From there we turn to the community itself. Joe traces it back to the spring of 1996 and the Berkeley Collaborative Filtering Workshop organized by Hal Varian and Paul Resnick, through a decade of scattered workshops at SIGIR, CSCW and CHI, to the first RecSys in 2007: a room on the Minnesota campus, over 120 people, and a substantial delegation from industry including Amazon. We discuss why the conference has always been heavily international and always at the intersection of research and practice, what surprised him most in 20 years (that we are still here and thriving), and the vision he pushed against the pull of becoming just another application of machine learning: a highly constrained, highly contextualized, multi-measure and often multi-stakeholder problem spanning algorithms, interfaces, data and business. This year the 20th RecSys comes home to Minneapolis, with Joe as General Co-Chair alongside George Karypis and Gediminas Adomavicius. We close on where things are heading. Joe's advice for newcomers is to immerse yourself in an application and find the real problems and opportunities there, rather than arriving enamored with a tool and treating every nail as something to hit. He picks up Karl Higley's point from the POPROX team that the research community keeps focusing on the model when the real object is the system: your algorithm is useless if it is not embedded in something that can deliver its results in a useful way. And he names the work he is most excited about — human decision-making and consumer psychology in the context of choice, the business school perspective that brings marketing and pricing into the picture, multi-sided marketplaces with their often invisible market maker, and the ethics of these systems as a practical lens on the long-term value our metrics still fail to capture. Enjoy this enriching episode of RECSPERTS – Recommender Systems Experts.Don’t forget to follow the podcast and please leave a review. (00:00) - Introduction (04:02) - About Joseph Konstan (10:57) - Loving and Hating Machine Learning (29:05) - What Makes Recommendations Useful (35:11) - Three Decades of GroupLens (40:29) - POPROX and Open Online Experimentation (51:41) - From the Berkeley Workshop to the First RecSys in 2007 (01:11:00) - 20 Years of RecSys and the Vision for the Field (01:17:38) - Further Challenges and Closing Remarks Links from the Episode:Joseph Konstan on LinkedInWebsite of Joseph KonstanGroupLens ResearchPOPROX: Platform for OPen Recommendation and Online eXperimentationNet PerceptionsLensKit: Python Recommendation ToolkitRecommender Systems Specialization on Coursera (Konstan & Ekstrand)ACM TechTalk by Joseph Konstan: Recommender Systems - Beyond Machine LearningACM TechTalk by Joseph Konstan: Recommender Systems - The Power of PersonalizationRecSys: The ACM Conference on Recommender SystemsTLDR AI NewsletterAssociated PressPapers: Resnick et al. (1994): GroupLens - An Open Architecture for Collaborative Filtering of NetnewsHill et al. (1995): Recommending and Evaluating Choices in a Virtual Community of UseShardanand & Maes (1995): Social Information Filtering - Algorithms for Automating "Word of Mouth"Resnick & Varian (1997): Recommender Systems (CACM Special Issue)Sarwar et al. (2001): Item-Based Collaborative Filtering Recommendation AlgorithmsDas et al. (2007): Google News Personalization - Scalable Online Collaborative FilteringSu et al. (2023): Long-Term Value of Exploration - Measurements, Findings and AlgorithmsTran et al. (2024): Transformers Meet ACT-R - Repeat-Aware and Sequential Listening Session Recommendation (Deezer)Ekstrand et al. (2024): Conducting Recommender Systems User Studies Using POPROXHigley et al. (2025): What News Recommendation Research Did (But Mostly Didn't) Teach Us About Building A News RecommenderGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 32 of Recsperts, I’m joined by my colleague Sasha Fedintsev, Staff Applied Scientist at Wolt (DoorDash), working across personalization and ads, to unpack the realities of building large-scale recommender systems in food, grocery, and retail delivery. Together, we discuss the specifics of personalization in the delivery domain, and the models and ideas that power Wolt’s recommender system across 30+ markets - where theory quickly meets messy, high-stakes practice. We explore what makes this domain fundamentally different from traditional e-commerce: strong locality constraints, real-time context, and a heavy skew toward repurchasing behavior. Sasha explains how these factors break many textbook approaches - like standard collaborative filtering - and require creative adaptations such as clustering strategies and multi-stage ranking systems optimized for latency, all while respecting locality constraints. We also discuss the evolution of recommendation approaches over time - from classical collaborative filtering with ALS, to Neural Collaborative Filtering with BPR, and ultimately to transformer-based models for user sequence modeling and next-purchase prediction powering today’s venue ranking systems. We also touch on practical challenges such as evaluation in real-world systems, including A/B testing pitfalls and biases in logged data, as well as the complexity introduced by multi-surface experiences like discovery pages, vertical lists, and search. Beyond venues, we discuss why item-level recommendation is an order of magnitude harder - due to scale, context dependence, and availability constraints - and what this implies for future system design. Throughout the episode, Sasha provides a candid view on the evolving role of a Staff Applied Scientist - bridging research and production, setting scientific standards, and driving cross-team impact. Enjoy this enriching episode of RECSPERTS – Recommender Systems Experts.Don’t forget to follow the podcast and please leave a review. (00:00) - Introduction (05:10) - About Sasha Fedintsev (15:26) - The Role of a Staff Applied Scientist (25:50) - Challenges and Specifics of the Delivery Industry (47:24) - Ranking and Recommendation Problems at Wolt (51:31) - NCF with BPR for Wolt's First DNN Recommendation Model (01:16:43) - User Sequence Transformers for Next Purchase Prediction (01:26:51) - Explore vs. Exploit or New vs. Recurring Purchases (01:31:29) - Ads Personalization at Wolt (01:36:16) - Further Challenges in RecSys (01:37:58) - A Final Note on Radical Longevity (01:46:30) - Closing Remarks Links from the Episode:Alexander "Sasha" Fedintsev on LinkedInAlexander on XWoltAlexander Fedintsev at Wolt Tech Talks: Restaurant discovery with Wolt: Deep Neural Networks to power recommendationsH3 Geospatial Indexing SystemRecommenders RepositoryTanja Reilly: The Staff Engineer's PathWill Larson: Staff Engineer: Leadership beyond the management trackCoupon collector's problemAlexander Fedintsev (2026): Longevity Bottlenecks: Part I — DementiaPapers: Rendle et al. (2009): BPR: Bayesian personalized ranking from implicit feedbackHe et al. (2017): Neural Collaborative FilteringDacrema et al. (2019): Are we really making much progress? A worrying analysis of recent neural recommendation approachesRendle et al (2020): Neural Collaborative Filtering vs. Matrix Factorization RevisitedHu et al. (2008): Collaborative Filtering for Implicit Feedback DatasetsGrbovic et al. (2015): E-commerce in Your Inbox: Product Recommendations at ScaleQuadrana et al. (2018): Sequence-Aware Recommender SystemsSu et al. (2024): Long-Term Value of Exploration: Measurements, Findings and AlgorithmsTran et al. (2024): Transformers Meet ACT-R: Repeat-Aware and Sequential Listening Session RecommendationLichtenberg et al. (2024): Ranking Across Different Content Types: The Robust Beauty of Multinomial BlendingGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 31 of Recsperts, I sit down with Elisabeth Lex, Full Professor of Human-Computer Interfaces and Inclusive Technologies at Graz University of Technology and a leading researcher at the intersection of recommender systems, psychology, and human–computer interaction. Together, we explore how recommender systems can become truly human-centric by integrating cognitive, emotional, and personality-aware models into their design. Elisabeth begins by addressing a common reductionism in the field: treating users primarily as data points rather than as humans with goals, emotions, memories, and cognitive boundaries. We revisit the origins of psychology-informed recommendation, including the Grundy system -the first recommender system, built nearly 50 years ago - which framed book recommendation through stereotype modeling. From there, we discuss how the community’s focus shifted toward solving recommendation mainly as an algorithmic optimization problem, often sidelining richer models of human decision-making. We then map out the three major branches of psychology-informed RecSys - cognition-inspired, affect-aware, and personality-aware - and dive into practical examples. Elisabeth walks us through her work on modeling music re-listening behavior using cognitive architectures such as ACT-R (Adaptive Control of Thought–Rational) and shows how cognitive constructs like memory decay, attention, and familiarity can meaningfully augment standard approaches like collaborative filtering. We also explore how hybrid systems that combine cognitive models with collaborative filtering can yield not just higher accuracy but also more novelty, diversity, and clearer explanations. Our conversation also turns to user-centric evaluation. Elisabeth argues that accuracy metrics alone cannot tell us whether a system is genuinely helpful. Instead, we must measure attitudes, perceptions, motivations, and emotional responses - while carefully accounting for cognitive biases, UI effects, and users’ lived experiences. Towards the end, Elisabeth discusses emerging research directions such as hybrid AI (symbolic + sub-symbolic methods), the role of LLMs and agents, the risks of replacing human studies with automated evaluations, and the responsibility our community has to understand users beyond their clicks. Enjoy this enriching episode of RECSPERTS – Recommender Systems Experts. Don’t forget to follow the podcast and please leave a review. (00:00) - Introduction (03:15) - About Elisabeth Lex (07:55) - Grundy, the first Recommender System (09:03) - Bridging the Gap between Psychology and Modern RecSys (17:21) - On how and when Elisabeth became a Researcher (21:39) - Survey on Psychology-Informed RecSys (39:29) - Personality-Aware Recommendation (49:43) - Affect- and Emotion-Aware Recommendation (01:01:37) - Cognition-Inspired Recommendation and the ACT-R Framework (01:14:39) - Combining Collaborative Filtering and ACT-R for Explainability (01:21:26) - Human-Centered Design (01:26:15) - Further Challenges and Closing Remarks Links from the Episode:Elisabeth Lex on LinkedInWebsite of ElisabethAI for Society LabFirst International Workshop on Recommender Systems for Sustainability and Social Good | co-located with RecSys 2024Second International Workshop on Recommender Systems for Sustainability and Social Good | co-located with RecSys 2025HyPer Workshop: Hybrid AI for Human-Centric PersonalizationTutorial on Psychology-Informed RecSysACT-R: Adaptive Control of Thought-RationalPOPROX: Platform for OPen Recommendation and Online eXperimentationPapers: Elaine Rich (1979): User Modeling via StereotypesLex et al. (2021): Psychology-informed Recommender SystemsReiter-Haas et al. (2021): Predicting Music Relistening Behavior Using the ACT-R FrameworkMoscati et al. (2023): Integrating the ACT-R Framework with Collaborative Filtering for Explainable Sequential Music RecommendationTran et al. (2024): Transformers Meet ACT-R: Repeat-Aware and Sequential Listening Session RecommendationGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 30 of Recsperts, I speak with Annelien Smets, Professor at Vrije Universiteit Brussel and Senior Researcher at imec-SMIT, about the value, perception, and practical design of serendipity in recommender systems. Annelien introduces her framework for understanding serendipity through intention, experience, and affordances, and explains the paradox of artificial serendipity - why it cannot be engineered, but only designed for. We start by unpacking the paradox of serendipity: while serendipity cannot be engineered or planned, systems and environments can be designed to increase the likelihood that serendipitous experiences occur. Annelien explains why randomness alone is not enough and why serendipity always emerges from an interplay between an unexpected encounter and a user’s ability to recognize its relevance and value. A central part of our discussion focuses on Annelien’s recent framework that distinguishes between intended, experienced, and afforded serendipity. We explore why organizations first need to clarify why they want serendipity - whether as an ideal, a common good, a mediator to achieve other goals (such as long-term retention or long-tail exposure), or even as a product feature in itself. From there, we dive into how users actually experience serendipity, drawing on qualitative interview research that identifies three core components: encounters must feel fortuitous, refreshing, and enriching. These components can manifest in different “flavors,” such as taste broadening, taste deepening, or rediscovering forgotten interests. We then move beyond algorithms to discuss affordances for serendipity - design principles that span content, user interfaces, and information access. Using examples from libraries, urban spaces, and digital platforms, Annelien shows why serendipity is a system-level property rather than a single metric or model tweak. We also discuss where serendipity can go wrong, including the Netflix “Surprise Me” feature, and why mismatched expectations can actually harm user experience. To close, we reflect on open research questions, from measuring different types of serendipity to understanding how content types, business models, and platform economics shape what is possible. Annelien also challenges a common myth: serendipity does not automatically burst filter bubbles—and should not be treated as a silver bullet. Enjoy this enriching episode of RECSPERTS – Recommender Systems Experts. Don’t forget to follow the podcast and please leave a review. (00:00) - Introduction (03:57) - About Annelien Smets (14:42) - Paradox and Definition of (Artificial) Serendipity (27:04) - Intended Serendipity (43:01) - Experienced Serendipity (01:01:18) - Afforded Serendipity (01:13:49) - Examples of Serendipity Going Wrong (01:17:40) - Framework for Serendipity (01:22:41) - Further Challenges and Closing Remarks Links from the Episode:Annelien Smets on LinkedInWebsite of AnnelienLinkedIn Article by Annelien Smets (2025): Overcoming the Paradox of Artificial SerendipityThe Serendipity SocietySerendipity EnginePapers: Smets (2025): Intended, afforded, and experienced serendipity: overcoming the paradox of artificial serendipitySmets et al. (2022): Serendipity in Recommender Systems Beyond the Algorithm: A Feature Repository and Experimental DesignBinst et al. (2025): What Is Serendipity? An Interview Study to Conceptualize Experienced Serendipity in Recommender SystemsZiarani et al. (2021): Serendipity in Recommender Systems: A Systematic Literature ReviewChen et al. (2021): Values of User Exploration in Recommender SystemsSmets et al. (2025): Why Do Recommenders Recommend? Three Waves of Research Perspectives on Recommender SystemsSmets (2023): Designing for Serendipity, a Means or an End?General Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 29 of Recsperts, I welcome Craig Macdonald, Professor of Information Retrieval at the University of Glasgow, and Aleksandr “Sasha” Petrov, PhD researcher and former applied scientist at Amazon. Together, we dive deep into sequential recommender systems and the growing role of transformer models such as SASRec and BERT4Rec. Our conversation begins with their influential replicability study of BERT4Rec, which revealed inconsistencies in reported results and highlighted the importance of training objectives over architecture tweaks. From there, Craig and Sasha guide us through their award-winning research on making transformers for sequential recommendation with large corpora both more effective and more efficient. We discuss how recency sampling (RSS) reduces training times dramatically, and how gSASRec overcomes the problem of overconfidence in models trained with negative sampling. By generalizing the sigmoid function (gBCE), they were able to reconcile cross-entropy–based optimization results with negative sampling, matching the effectiveness of softmax approaches while keeping training scalable for large corpora. We also explore RecJPQ, their recent work on joint product quantization for item embeddings. This approach makes transformer-based sequential recommenders substantially faster at inference and far more memory-efficient for embeddings—while sometimes even improving effectiveness thanks to regularization effects. Towards the end, Craig and Sasha share their perspective on generative approaches like GPTRec, the promises and limits of large language models in recommendation, and what challenges remain for the future of sequential recommender systems. Enjoy this enriching episode of RECSPERTS – Recommender Systems Experts. Don’t forget to follow the podcast and please leave a review. (00:00) - Introduction (04:09) - About Craig Macdonald (04:46) - About Sasha Petrov (13:48) - Tutorial on Transformers for Sequential Recommendations (19:24) - SASRec vs. BERT4Rec (21:25) - Replicability Study of BERT4Rec for Sequential Recommendation (32:52) - Training Sequential RecSys using Recency Sampling (40:01) - gSASRec for Reducing Overconfidence by Negative Sampling (01:00:51) - RecJPQ: Training Large-Catalogue Sequential Recommenders (01:21:37) - Generative Sequential Recommendation with GPTRec (01:29:12) - Further Challenges and Closing Remarks Links from the Episode:Craig Macdonald on LinkedInSasha Petrov on LinkedInSasha's WebsiteTutorial: Transformers for Sequential Recommendation (ECIR 2024)Tutorial Recording from ACM European Summer School in Bari (2024)Talk: Neural Recommender Systems (European Summer School in Information Retrieval 2024)Papers: Kang et al. (2018): Self-Attentive Sequential RecommendationSun et al. (2019): BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from TransformerPetrov et al. (2022): A Systematic Review and Replicability Study of BERT4Rec for Sequential RecommendationPetrov et al. (2022): Effective and Efficient Training for Sequential Recommendation using Recency SamplingPetrov et al. (2024): RSS: Effective and Efficient Training for Sequential Recommendation Using Recency Sampling (extended version)Petrov et al. (2023): gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative SamplingPetrov et al. (2025): Improving Effectiveness by Reducing Overconfidence in Large Catalogue Sequential Recommendation with gBCE lossPetrov et al. (2024): RecJPQ: Training Large-Catalogue Sequential RecommendersPetrov et al. (2024): Efficient Inference of Sub-Item Id-based Sequential Recommendation Models with Millions of ItemsRajput et al. (2023): Recommender Systems with Generative RetrievalPetrov et al. (2023): Generative Sequential Recommendation with GPTRecPetrov et al. (2024): Aligning GPTRec with Beyond-Accuracy Goals with Reinforcement LearningGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] WebsiteDisclaimer:Craig holds concurrent appointments as a Professor of Information Retrieval at University of Glasgow and as an Amazon Scholar. This podcast describes work performed at the University of Glasgow and is not associated with Amazon.

In episode 28 of Recsperts, I sit down with Robin Burke, professor of information science at the University of Colorado Boulder and a leading expert with over 30 years of experience in recommender systems. Together, we explore multistakeholder recommender systems, fairness, transparency, and the role of recommender systems in the age of evolving generative AI. We begin by tracing the origins of recommender systems, traditionally built around user-centric models. However, Robin challenges this perspective, arguing that all recommender systems are inherently multistakeholder—serving not just consumers as the recipients of recommendations, but also content providers, platform operators, and other key players with partially competing interests. He explains why the common “Recommended for You” label is, at best, an oversimplification and how greater transparency is needed to show how stakeholder interests are balanced. Our conversation also delves into practical approaches for handling multiple objectives, including reranking strategies versus integrated optimization. While embedding multistakeholder concerns directly into models may be ideal, reranking offers a more flexible and efficient alternative, reducing the need for frequent retraining. Towards the end of our discussion, we explore post-userism and the impact of generative AI on recommendation systems. With AI-generated content on the rise, Robin raises a critical concern: if recommendation systems remain overly user-centric, generative content could marginalize human creators, diminishing their revenue streams.  Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.Don't forget to follow the podcast and please leave a review (00:00) - Introduction (03:24) - About Robin Burke and First Recommender Systems (26:07) - From Fairness and Advertising to Multistakeholder RecSys (34:10) - Multistakeholder RecSys Terminology (40:16) - Multistakeholder vs. Multiobjective (42:43) - Reciprocal and Value-Aware RecSys (59:14) - Objective Integration vs. Reranking (01:06:31) - Social Choice for Recommendations under Fairness (01:17:40) - Post-Userist Recommender Systems (01:26:34) - Further Challenges and Closing Remarks Links from the Episode:Robin Burke on LinkedInRobin's WebsiteThat Recommender Systems LabReference to Broder's Keynote on Computational Advertising and Recommender Systems from RecSys 2008Multistakeholder Recommender Systems (from Recommender Systems Handbook), chapter by Himan Abdollahpouri & Robin BurkePOPROX: The Platform for OPen Recommendation and Online eXperimentationAltRecSys 2024 (Workshop at RecSys 2024)Papers: Burke et al. (1996): Knowledge-Based Navigation of Complex Information SpacesBurke (2002): Hybrid Recommender Systems: Survey and ExperimentsResnick et al. (1997): Recommender SystemsGoldberg et al. (1992): Using collaborative filtering to weave an information tapestryLinden et al. (2003): Amazon.com Recommendations - Item-to-Item Collaborative FilteringAird et al. (2024): Social Choice for Heterogeneous Fairness in RecommendationAird et al. (2024): Dynamic Fairness-aware Recommendation Through Multi-agent Social ChoiceBurke et al. (2024): Post-Userist Recommender Systems : A ManifestoBaumer et al. (2017): Post-userismBurke et al. (2024): Conducting Recommender Systems User Studies Using POPROXGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 27 of Recsperts, we meet Alessandro Piscopo, Lead Data Scientist in Personalization and Search, and Duncan Walker, Principal Data Scientist in the iPlayer Recommendations Team, both from the BBC. We discuss how the BBC personalizes recommendations across different offerings like news or video and audio content recommendations. We learn about the core values for the oldest public service media organization and the collaboration with editors in that process. The BBC once started with short video recommendations for BBC+ and nowadays has to consider recommendations across multiple domains: news, the iPlayer, BBC Sounds, BBC Bytesize, and more. With a reach of about 500M+ users who access services every week there is a huge potential. My guests discuss the challenges of aligning recommendations with public service values and the role of editors and constant exchange, alignment, and learning between the algorithmic and editorial lines of recommender systems.We also discuss the potential of cross-domain recommendations to leverage the content across different products as well as the organizational setup of teams working on recommender systems at the BBC. We learn about skews in the data due to the nature of an online service that also has a linear offering with TV and radio services. Towards the end, we also touch a bit on QUARE @ RecSys, which is the Workshop on Measuring the Quality of Explanations in Recommender Systems. Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.Don't forget to follow the podcast and please leave a review (00:00) - Introduction (03:10) - About Alessandro Piscopo and Duncan Walker (14:53) - RecSys Applications at the BBC (20:22) - Journey of Building Public Service Recommendations (28:02) - Role and Implementation of Public Service Values (36:52) - Algorithmic and Editorial Recommendation (01:01:54) - Further RecSys Challenges at the BBC (01:15:53) - Quare Workshop (01:23:27) - Closing Remarks Links from the Episode:Alessandro Piscopo on LinkedInDuncan Walker on LinkedInBBCQUARE @ RecSys 2023 (2nd Workshop on Measuring the Quality of Explanations in Recommender Systems)Papers: Clarke et al. (2023): Personalised Recommendations for the BBC iPlayer: Initial approach and current challengesBoididou et al. (2021): Building Public Service Recommenders: Logbook of a JourneyPiscopo et al. (2019): Data-Driven Recommendations in a Public Service OrganisationGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 26 of Recsperts, I speak with Sanne Vrijenhoek, a PhD candidate at the University of Amsterdam’s Institute for Information Law and the AI, Media & Democracy Lab. Sanne’s research explores diversity in recommender systems, particularly in the news domain, and its connection to democratic values and goals. We dive into four of her papers, which focus on how diversity is conceptualized in news recommender systems. Sanne introduces us to five rank-aware divergence metrics for measuring normative diversity and explains why diversity evaluation shouldn’t be approached blindly—first, we need to clarify the underlying values. She also presents a normative framework for these metrics, linking them to different democratic theory perspectives. Beyond evaluation, we discuss how to optimize diversity in recommender systems and reflect on missed opportunities—such as the RecSys Challenge 2024, which could have gone beyond accuracy-chasing. Sanne also shares her recommendations for improving the challenge by incorporating objectives such as diversity. During our conversation, Sanne shares insights on effectively communicating recommender systems research to non-technical audiences. To wrap up, we explore ideas for fostering a more diverse RecSys research community, integrating perspectives from multiple disciplines. Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.Don't forget to follow the podcast and please leave a review (00:00) - Introduction (03:24) - About Sanne Vrijenhoek (14:49) - What Does Diversity in RecSys Mean? (26:32) - Assessing Diversity in News Recommendations (34:54) - Rank-Aware Divergence Metrics to Measure Normative Diversity (01:01:37) - RecSys Challenge 2024 - Recommendations for the Recommenders (01:11:23) - RecSys Workshops - NORMalize and AltRecSys (01:15:39) - On the Different Conceptualizations of Diversity in RecSys (01:28:38) - Closing Remarks Links from the Episode:Sanne Vrijenhoek on LinkedInInformfullyMIND: MIcrosoft News DatasetRecSys Challenge 2024NORMalize 2023: The First Workshop on the Normative Design and Evaluation of Recommender SystemsNORMalize 2024: The Second Workshop on the Normative Design and Evaluation of Recommender SystemsAltRecSys 2024: The AltRecSys Workshop on Alternative, Unexpected, and Critical Ideas in RecommendationPapers: Vrijenhoek et al. (2021): Recommenders with a Mission: Assessing Diversity in News RecommendationsVrijenhoek et al. (2022): RADio – Rank-Aware Divergence Metrics to Measure Normative Diversity in News RecommendationsHeitz et al. (2024): Recommendations for the Recommenders: Reflections on Prioritizing Diversity in the RecSys ChallengeVrijenhoek et al. (2024): Diversity of What? On the Different Conceptualizations of Diversity in Recommender SystemsHelberger (2019): On the Democratic Role of News RecommendersSteck (2018): Calibrated RecommendationsGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 25, we talk about the upcoming ACM Conference on Recommender Systems 2024 (RecSys) and welcome a former guest to geek about the conference. Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.Don't forget to follow the podcast and please leave a review (00:00) - Introduction (01:56) - Overview RecSys 2024 (07:01) - Contribution Stats (09:37) - Interview Links from the Episode:RecSys 2024 Conference WebsitePapers: RecSys '24: Proceedings of the 18th ACM Conference on Recommender SystemsGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 24 of Recsperts, I sit down with Amey Dharwadker, Machine Learning Engineering Manager at Facebook, to dive into the complexities of large-scale video recommendations. Amey, who leads the Video Recommendations Quality Ranking team at Facebook, sheds light on the intricate challenges of delivering personalized video feeds at scale. Our conversation covers content understanding, user interaction data, real-time signals, exploration, and evaluation techniques. We kick off the episode by reflecting on the inaugural VideoRecSys workshop at RecSys 2023, setting the stage for a deeper discussion on Facebook’s approach to video recommendations. Amey walks us through the critical challenges they face, such as gathering reliable user feedback signals to avoid pitfalls like watchbait. With a vast and ever-growing corpus of billions of videos—millions of which are added each month—the cold start problem looms large. We explore how content understanding, user feedback aggregation, and exploration techniques help address this issue. Amey explains how engagement metrics like watch time, comments, and reactions are used to rank content, ensuring users receive meaningful and diverse video feeds. A key highlight of the conversation is the importance of real-time personalization in fast-paced environments, such as short-form video platforms, where user preferences change quickly. Amey also emphasizes the value of cross-domain data in enriching user profiles and improving recommendations. Towards the end, Amey shares his insights on leadership in machine learning teams, pointing out the characteristics of a great ML team. Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.Don't forget to follow the podcast and please leave a review (00:00) - Introduction (02:32) - About Amey Dharwadker (08:39) - Video Recommendation Use Cases on Facebook (16:18) - Recommendation Teams and Collaboration (25:04) - Challenges of Video Recommendations (31:07) - Video Content Understanding and Metadata (33:18) - Multi-Stage RecSys and Models (42:42) - Goals and Objectives (49:04) - User Behavior Signals (59:38) - Evaluation (01:06:33) - Cross-Domain User Representation (01:08:49) - Leadership and What Makes a Great Recommendation Team (01:13:01) - Closing Remarks Links from the Episode:Amey Dharwadker on LinkedInAmey's WebsiteRecSys Challenge 2021VideoRecSys Workshop 2023VideoRecSys + LargeRecSys 2024Papers: Mahajan et al. (2023): CAViaR: Context Aware Video RecommendationsMahajan et al. (2023): PIE: Personalized Interest Exploration for Large-Scale Recommender SystemsRaul et al. (2023): CAM2: Conformity-Aware Multi-Task Ranking Model for Large-Scale Recommender SystemsZhai et al. (2024): Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative RecommendationsSaket et al. (2023): Formulating Video Watch Success Signals for Recommendations on Short Video PlatformsWang et al. (2022): Surrogate for Long-Term User Experience in Recommender SystemsSu et al. (2024): Long-Term Value of Exploration: Measurements, Findings and AlgorithmsGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 23 of Recsperts, we welcome Yashar Deldjoo, Assistant Professor at the Polytechnic University of Bari, Italy. Yashar's research on recommender systems includes multimodal approaches, multimedia recommender systems as well as trustworthiness and adversarial robustness, where he has published a lot of work. We discuss the evolution of generative models for recommender systems, modeling paradigms, scenarios as well as their evaluation, risks and harms. We begin our interview with a reflection of Yashar's areas of recommender systems research so far. Starting with multimedia recsys, particularly video recommendations, Yashar covers his work around adversarial robustness and trustworthiness leading to the main topic for this episode: generative models for recommender systems. We learn about their aspects for improving beyond the (partially saturated) state of traditional recommender systems: improve effectiveness and efficiency for top-n recommendations, introduce interactivity beyond classical conversational recsys, provide personalized zero- or few-shot recommendations.We learn about the modeling paradigms and as well about the scenarios for generative models which mainly differ by input and modelling approach: ID-based, text-based, and multimodal generative models. This is how we navigate the large field of acronyms leading us from VAEs and GANs to LLMs. Towards the end of the episode, we also touch on the evaluation, opportunities, risks and harms of generative models for recommender systems. Yashar also provides us with an ample amount of references and upcoming events where people get the chance to know more about GenRecSys. Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.Don't forget to follow the podcast and please leave a review (00:00) - Introduction (03:58) - About Yashar Deldjoo (09:34) - Motivation for RecSys (13:05) - Intro to Generative Models for Recommender Systems (44:27) - Modeling Paradigms for Generative Models (51:33) - Scenario 1: Interaction-Driven Recommendation (57:59) - Scenario 2: Text-based Recommendation (01:10:39) - Scenario 3: Multimodal Recommendation (01:24:59) - Evaluation of Impact and Harm (01:38:07) - Further Research Challenges (01:45:03) - References and Research Advice (01:49:39) - Closing Remarks Links from the Episode:Yashar Deldjoo on LinkedInYashar's WebsiteKDD 2024 Tutorial: Modern Recommender Systems Leveraging Generative AI: Fundamentals, Challenges and OpportunitiesRecSys 2024 Workshop: The 1st Workshop on Risks, Opportunities, and Evaluation of Generative Models in Recommender Systems (ROEGEN@RECSYS'24)Papers: Deldjoo et al. (2024): A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)Deldjoo et al. (2020): Recommender Systems Leveraging Multimedia ContentDeldjoo et al. (2021): A Survey on Adversarial Recommender Systems: From Attack/Defense Strategies to Generative Adversarial NetworksDeldjoo et al. (2020): How Dataset Characteristics Affect the Robustness of Collaborative Recommendation ModelsLiang et al. (2018): Variational Autoencoders for Collaborative FilteringHe et al. (2016): Visual Bayesian Personalized Ranking from Implicit FeedbackGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 22 of Recsperts, we welcome Prabhat Agarwal, Senior ML Engineer, and Aayush Mudgal, Staff ML Engineer, both from Pinterest, to the show. Prabhat works on recommendations and search systems at Pinterest, leading representation learning efforts. Aayush is responsible for ads ranking and privacy-aware conversion modeling. We discuss user and content modeling, short- vs. long-term objectives, evaluation as well as multi-task learning and touch on counterfactual evaluation as well. In our interview, Prabhat guides us through the journey of continuous improvements of Pinterest's Homefeed personalization starting with techniques such as gradient boosting over two-tower models to DCN and transformers. We discuss how to capture users' short- and long-term preferences through multiple embeddings and the role of candidate generators for content diversification. Prabhat shares some details about position debiasing and the challenges to facilitate exploration.With Aayush we get the chance to dive into the specifics of ads ranking at Pinterest and he helps us to better understand how multifaceted ads can be. We learn more about the pain of having too many models and the Pinterest's efforts to consolidate the model landscape to improve infrastructural costs, maintainability, and efficiency. Aayush also shares some insights about exploration and corresponding randomization in the context of ads and how user behavior is very different between different kinds of ads.Both guests highlight the role of counterfactual evaluation and its impact for faster experimentation. Towards the end of the episode, we also touch a bit on learnings from last year's RecSys challenge. Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.Don't forget to follow the podcast and please leave a review (00:00) - Introduction (03:51) - Guest Introductions (09:57) - Pinterest Introduction (21:57) - Homefeed Personalization (47:27) - Ads Ranking (01:14:58) - RecSys Challenge 2023 (01:20:26) - Closing Remarks Links from the Episode:Prabhat Agarwal on LinkedInAayush Mudgal on LinkedInRecSys Challenge 2023Pinterest Engineering BlogPinterest LabsPrabhat's Talk at GTC 2022: Evolution of web-scale engagement modeling at PinterestBlogpost: How we use AutoML, Multi-task learning and Multi-tower models for Pinterest AdsBlogpost: Pinterest Home Feed Unified Lightweight Scoring: A Two-tower ApproachBlogpost: Experiment without the wait: Speeding up the iteration cycle with Offline Replay ExperimentationBlogpost: MLEnv: Standardizing ML at Pinterest Under One ML Engine to Accelerate InnovationBlogpost: Handling Online-Offline Discrepancy in Pinterest Ads Ranking SystemPapers: Eksombatchai et al. (2018): Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-TimeYing et al. (2018): Graph Convolutional Neural Networks for Web-Scale Recommender SystemsPal et al. (2020): PinnerSage: Multi-Modal User Embedding Framework for Recommendations at PinterestPancha et al. (2022): PinnerFormer: Sequence Modeling for User Representation at PinterestZhao et al. (2019): Recommending what video to watch next: a multitask ranking systemGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 21 of Recsperts, we welcome Martijn Willemsen, Associate Professor at the Jheronimus Academy of Data Science and Eindhoven University of Technology. Martijn's researches on interactive recommender systems which includes aspects of decision psychology and user-centric evaluation. We discuss how users gain control over recommendations, how to support their goals and needs as well as how the user-centric evaluation framework fits into all of this. In our interview, Martijn outlines the reasons for providing users control over recommendations and how to holistically evaluate the satisfaction and usefulness of recommendations for users goals and needs. We discuss the psychology of decision making with respect to how well or not recommender systems support it. We also dive into music recommender systems and discuss how nudging users to explore new genres can work as well as how longitudinal studies in recommender systems research can advance insights. Towards the end of the episode, Martijn and I also discuss some examples and the usefulness of enabling users to provide negative explicit feedback to the system. Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.Don't forget to follow the podcast and please leave a review (00:00) - Introduction (03:03) - About Martijn Willemsen (15:14) - Waves of User-Centric Evaluation in RecSys (19:35) - Behaviorism is not Enough (46:21) - User-Centric Evaluation Framework (01:05:38) - Genre Exploration and Longitudinal Studies in Music RecSys (01:20:59) - User Control and Negative Explicit Feedback (01:31:50) - Closing Remarks Links from the Episode:Martijn Willemsen on LinkedInMartijn Willemsen's WebsiteUser-centric Evaluation FrameworkBehaviorism is not Enough (Talk at RecSys 2016)Neil Hunt: Quantifying the Value of Better Recommendations (Keynote at RecSys 2014)What recommender systems can learn from decision psychology about preference elicitation and behavioral change (Talk at Boise State (Idaho) and Grouplens at University of Minnesota)Eric J. Johnson: The Elements of ChoiceRasch ModelSpotify Web APIPapers: Ekstrand et al. (2016): Behaviorism is not Enough: Better Recommendations Through Listening to UsersKnijenburg et al. (2012): Explaining the user experience of recommender systemsEkstrand et al. (2014): User perception of differences in recommender algorithmsLiang et al. (2022): Exploring the longitudinal effects of nudging on users’ music genre exploration behavior and listening preferencesMcNee et al. (2006): Being accurate is not enough: how accuracy metrics have hurt recommender systemsGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website

In episode 20 of Recsperts, we welcome Bram van den Akker, Senior Machine Learning Scientist at Booking.com. Bram's work focuses on bandit algorithms and counterfactual learning. He was one of the creators of the Practical Bandits tutorial at the World Wide Web conference. We talk about the role of bandit feedback in decision making systems and in specific for recommendations in the travel industry. In our interview, Bram elaborates on bandit feedback and how it is used in practice. We discuss off-policy- and on-policy-bandits, and we learn that counterfactual evaluation is right for selecting the best model candidates for downstream A/B-testing, but not a replacement. We hear more about the practical challenges of bandit feedback, for example the difference between model scores and propensities, the role of stochasticity or the nitty-gritty details of reward signals. Bram also shares with us the challenges of recommendations in the travel domain, where he points out the sparsity of signals or the feedback delay. At the end of the episode, we can both agree on a good example for a clickbait-heavy news service in our phones. Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.Don't forget to follow the podcast and please leave a review (00:00) - Introduction (02:58) - About Bram van den Akker (09:16) - Motivation for Practical Bandits Tutorial (16:53) - Specifics and Challenges of Travel Recommendations (26:19) - Role of Bandit Feedback in Practice (49:13) - Motivation for Bandit Feedback (01:00:54) - Practical Start for Counterfactual Evaluation (01:06:33) - Role of Business Rules (01:11:26) - better cut this section coherently (01:17:48) - Rewards and More (01:32:45) - Closing Remarks Links from the Episode:Bram van den Akker on LinkedInPractical Bandits: An Industry Perspective (Website)Practical Bandits: An Industry Perspective (Recording)Tutorial at The Web Conference 2020: Unbiased Learning to Rank: Counterfactual and Online ApproachesTutorial at RecSys 2021: Counterfactual Learning and Evaluation for Recommender Systems: Foundations, Implementations, and Recent AdvancesGitHub: Open Bandit PipelinePapers: van den Akker et al. (2023): Practical Bandits: An Industry Perspectivevan den Akker et al. (2022): Extending Open Bandit Pipeline to Simulate Industry Challengesvan den Akker et al. (2019): ViTOR: Learning to Rank Webpages Based on Visual FeaturesGeneral Links: Follow me on LinkedInFollow me on XSend me your comments, questions and suggestions to [email protected] Website