Advertise on podcast: Experiment Nation: The Podcast

High-velocity experimentation can dramatically increase how quickly organizations learn, optimize their products, and drive revenue. But many teams still run A/B tests sequentially, waiting for one experiment to finish before starting the next. In this video, Varun, co-founder of Geeks of Digital, explains how overlapping A/B tests can remove experimentation bottlenecks while still maintaining rigorous randomization and measurement. You’ll learn how layered experimentation works, why the "isolation fallacy" can hold teams back, when experiments should still be isolated, and how organizations can build an operational system for running experiments at scale. The video also covers the "blue-by-blue" isolation trap, compounding gains from multiple winning experiments, fake door testing, experimentation culture, and how companies like Meta, Booking.com, Airbnb, and Spotify approach experimentation. --- Music: “Strong Female Pop Attitude” by JuicyLucyMusic, licensed via AudioJungle/Envato Market.https://audiojungle.net/item/strong-pop-attitude/19345362 This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Starting a career in Conversion Rate Optimization (CRO) can be surprisingly difficult. Getting your first experience, working with developers, prioritizing experiments, dealing with failed tests, and proving the value of experimentation all come with a learning curve.In this episode of Experiment Nation, Eline speaks with CRO and experimentation leader Edd about the lessons he learned during his first years in the industry, and what he wishes he had known sooner.They discuss:• How to get your first CRO experience when you don't have a CRO background• Why agency experience can be a powerful accelerator for learning CRO• Getting buy-in from developers and explaining experimentation to technical teams• Why traditional prioritization frameworks like ICE and P.I.E. don't always work• How to prioritize experiments using potential business impact and MDE• How to build a CRO career from digital marketing, analytics, UX, or other disciplines• Why your first experiments don't have to be perfect A/B tests• How to deal with failed and inconclusive experiments• Why CROs should spend more time in the problem space and less time generating test ideas• Using quantitative and qualitative research to identify customer problems• Jobs to Be Done, surveys, stakeholder interviews, and customer journey mapping• Why experimentation should generate customer intelligence, not just conversion lifts• How previous experiment results can inform your next hypothesis• Dealing with imposter syndrome and the fear of failure• Why CRO professionals should "have strong opinions, but hold them loosely"• The importance of learning from mistakes and communicating them effectivelyOne of the biggest themes of the conversation is the shift from "What should we test?" to "What problem are our customers actually experiencing?"For CRO practitioners, that's an important distinction. The goal isn't simply to generate more winning experiments. It's to understand customers better, validate problems, and build a growing body of evidence that makes future experimentation more effective.Whether you're starting your first CRO role or you've been running experiments for years, there are plenty of practical lessons here.Chapters:00:00 Introduction01:03 Meet Ed: 10+ Years in CRO & Experimentation02:26 An Unusual Path Into Digital Marketing05:22 Discovering Experimentation Through Email Testing08:34 How Previous Careers Can Prepare You for CRO10:20 How to Get Your First CRO Experience12:28 CRO Communities, Courses & Learning Resources15:42 Why Practical Experience Beats Theory16:00 The Hardest Parts of Starting a CRO Career16:24 Getting Technical Buy-In From Developers17:30 Why CRO Prioritization Frameworks Don't Always Work18:32 How to Prioritize Experiments Based on Business Impact21:38 Getting Your First CRO Client or Project23:01 How to Create CRO Opportunities In-House24:29 Why Agency Experience Can Accelerate Your CRO Career27:19 Turning Digital Marketing Work Into CRO Experience28:21 Experimenting Beyond A/B Tests30:08 Learning From Failed Experiments32:00 Overcoming Imposter Syndrome & Fear of Failure34:41 How to Communicate Mistakes at Work37:29 Get Out of the Solution Space38:43 Finding & Validating Customer Problems40:25 Stop Chasing Wins, Start Chasing Learnings45:14 Using Jobs to Be Done in CRO46:42 Surveys & Stakeholder Interviews for Customer Research47:24 Customer Journey Mapping for CRO50:40 Don't Forget the Human Behind the Data51:57 "Have Strong Opinions, But Hold Them Loosely"53:41 Keep Learning Beyond Traditional CRO Resources54:29 What CRO Can Learn From Radical Candor56:04 Final Thoughts#CRO #ConversionRateOptimization #Experimentation #ABTesting #GrowthMarketing #UX #DigitalMarketingBackground music:"Background Hip-Hop Urban" by CoffeeMusicLicensed via AudioJungle under the Music Standard License. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Jon Crowder joins Tracy Laranjo to explain why ethical experimentation is not just better for customers—it is better for long-term growth. They unpack fake urgency, misleading pricing and subscription tactics, short-term A/B test wins that erode trust, and practical ways to design experiments that improve retention, lifetime value, and revenue.Chapters:0:00 Welcome & introducing Jon Crowder1:29 Why ethical experimentation matters2:17 The pressure to drive results at any cost7:17 When winning an A/B test loses money long term8:34 Reframing urgency into trust-building experiences13:38 Choosing clients and setting ethical boundaries16:37 The ethics of price testing19:21 Fake sales, subscriptions & agency pressure20:46 How to steer unethical tests into ethical ones23:09 Have urgency and scarcity gone too far?24:02 Why legacy businesses cling to outdated tactics27:36 Gen Z, trust, and changing expectations29:48 What an ethical future for digital experiences looks like34:30 Brands that earn loyalty through values38:27 Why ethical experiences convert better40:15 A simple test for ethical experimentation41:37 Where to find Jon#CRO #Experimentation #ABTesting #GrowthMarketing #EthicalMarketing #ProductGrowth #ConversionOptimization #DigitalExperience #CustomerExperience #Retention #MarketingStrategy #GrowthStrategy This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode, we dive into CRO “best practices” that completely failed in real-world experimentation. From homepage carousels unexpectedly winning tests to why most experimenters still don’t trust AI-generated CRO workflows, we unpack the messy reality behind optimization advice.We also debate synthetic AI users for research, discuss the most useful survey questions in CRO, roast dark-mode websites, and explain why emotional branding can sometimes hurt usability.Join our slack: https://join.slack.com/t/experiment-nation/shared_invite/zt-3zowejvoj-ujS_tbqfXcPkS9O0rnWhTQTopics include:Why homepage carousels occasionally winAI-powered A/B testing platformsSynthetic users & fake research dataBest and worst CRO survey questionsDark mode website UXWhy qualitative context still mattersJoin the Experiment Nation Slack community to participate in future discussions and question threads.Chapters00:00 Intro & CRO community questions01:46 CRO best practices that failed in real life02:28 Do homepage carousels actually work?05:02 Hero videos, agencies & emotional design06:25 When carousels might make sense08:24 Should AI run the entire CRO workflow?11:48 Why experimenters still don’t trust AI14:49 Best & worst survey questions in CRO17:30 Emotional branding vs usable websites18:26 Dark mode websites & readability debate21:29 Synthetic AI users for testing & research23:56 Why fake users can’t mimic real humans26:32 Using AI as a research starting point28:41 Closing thoughts & Slack community invite This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

What do you do when an experiment tanks conversion rate by 8% in less than a week?In this episode, Gerda and Rommil break down real CRO and experimentation scenarios submitted by the community, including failed tests, executive pressure, guardrail metrics, knowledge management, developer friction, and why experimentation teams always seem to end up back in spreadsheets.Check out: http://katsed.com/We also discuss: * Root cause analysis for failed experiments * How to calm leadership during testing disasters * Guardrail metrics and control limits * Why experimentation knowledge banks become messy * Airtable vs spreadsheets vs enterprise tools * Why developers and designers struggle with experimentation culture * How to onboard technical teams into testing workflows A practical conversation for experimentation leads, growth teams, product managers, CRO specialists, and anyone building an experimentation culture.Chapters00:00 Burger Menu Debate: Left vs Right01:28 Mystery Meat Navigation & UX History03:03 Why Modern Phones Break UX05:08 Community CRO Scenarios Begin06:18 Experiment Tanks Conversion by 8%07:33 Root Cause Analysis & Leadership Pressure09:17 QA, Tracking Errors & Failed Tests10:03 Building a CRO Analysis Tool12:18 Probabilistic Root Cause Detection14:26 Managing Panic During Bad Experiments15:21 Guardrail Metrics Explained16:40 Using Control Limits in Experimentation18:18 Building an Experimentation Knowledge Bank20:05 Why You Don’t Need AI for Everything21:08 Why Process Changes Create Admin Pain23:08 Migrating Between Experimentation Tools24:31 Why Teams Always Go Back to Spreadsheets26:50 CRO Teams vs Developers & Designers29:09 Why Early Experiments Should Be Simple30:41 Why Developers Hate “Throwaway Code”32:16 Designers vs Experiment Constraints33:02 Outro & Next Community Questions This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Most teams say they “do research”… but very few actually do it well.In this episode, we break down what real customer research looks like in CRO and growth. From analyzing 200+ survey responses manually, to why AI still struggles with qualitative data, to the politics that slow everything down.We also get into:Why AI fails at nuanced research (and what it misses)How to actually analyze open-ended survey responsesThe biggest mistakes teams make with surveysWhy companies avoid talking to customers (and why that’s a red flag)How research becomes a political nightmare in larger orgsWhen you should sample vs analyze everythingHow research fuels better marketing and personalizationAnd yes… we somehow end up talking about billion dollars and laundry.If you’re in CRO, growth, product, or marketing, this is the stuff people don’t usually say out loud.👇 Timestamps00:00 Why content actually drives opportunities03:15 What “research” really means04:00 How many responses you actually need06:00 Why AI fails at qualitative analysis07:30 When to sample vs analyze everything09:00 Using research for marketing copy10:00 Why not talking to customers is a red flag11:00 Survey politics and internal resistance13:30 Bad survey questions and bias16:00 Personalization vs CRO (what’s the difference?)22:30 Personalizing outreach properly26:30 Why research projects spiral out of control29:00 How to actually make research a habit31:00 How to ship research faster (timeboxing)34:00 “Business jiu-jitsu” and fake urgency35:30 Final thoughts (and laundry) This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode, we dive into the realities of CRO (Conversion Rate Optimization) today — from AI anxiety to experimentation culture and how teams should actually be structured. We start with a surprisingly philosophical take on photography and overthinking, then quickly shift into what’s really happening in the CRO and AI space right now. We discuss: * Why AI might be making everything look the same * The growing fear (and noise) around AI in CRO * How to structure CRO teams: centralized vs hybrid vs embedded * Why experimentation still struggles with a “nice-to-have” perception * What actually builds a strong experimentation culture * How individual contributors can influence culture without authority * And how to position CRO as a true growth engine, not just testing If you work in growth, product, or experimentation, this is a candid look at what’s changing and what actually matters. Chapters 0:00 – Photography, overthinking, and creative process1:26 – Back to CRO: what’s happening in the industry2:07 – AI concerns and job market anxiety2:45 – The “everything looks the same” AI problem4:00 – Too many AI tools, no real differentiation5:38 – What people are really asking about AI6:23 – AI fear vs reality (and layoffs)7:08 – Fear-mongering vs signal8:00 – Fear and decision-making9:00 – CRO team structures: centralized, hybrid, embedded9:31 – Why centralized works early10:08 – When to move to hybrid10:29 – Why embedded is hard11:04 – The role of leadership buy-in12:13 – Why experimentation gets cut first13:00 – CRO’s “nice-to-have” branding problem14:00 – Why CRO is easy to underestimate15:45 – Future-proofing: combine CRO with other skills16:54 – Building experimentation culture (the “laundry” analogy)18:19 – How to destroy experimentation culture19:23 – Why authority and buy-in matter20:25 – How ICs can influence culture21:34 – Reframing experimentation as empowerment23:09 – Becoming a catalyst across teams24:46 – Confidence and presence in CRO roles25:59 – CRO vs Growth: what’s the difference?27:24 – Where CRO fits inside growth teams28:02 – Moving CRO upstream into systems thinking29:00 – Closing thoughts This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode, we answer real, unfiltered questions from the CRO and experimentation community. We dig into how far AI can realistically help with experimentation, why revenue metrics like ARPU are a nightmare for MDE calculations, how to handle sketchy interview “audits,” and why ROI conversations never seem to end.No hype. No magic bullets. Just honest perspectives from people actually doing the work.Topics include: * Where AI helps (and quietly breaks) CRO workflows * How to think about MDEs for continuous metrics like ARPU * Making time for research when you’re already drowning * When “auditing a site” crosses into spec work * Volume hiring for CRO roles * Escaping (or not) the endless ROI loop * Why some CRO agencies don’t have designers Drop your own questions in the comments or check the links below to submit anonymously.- Ask us a question here: https://docs.google.com/forms/d/e/1FAIpQLSd8P5yCsO5NB6WPe204k1WExhH1Bs8w6nZxPRRn-BiAGozpaQ/viewform- Visit Koalatative here: https://www.youtube.com/@koalatative⏱️ Chapters00:00 – Intro & warm-up04:34 – How far can AI actually help in delivering CRO?11:19 – How do experimentation experts estimate MDE for continuous metrics like ARPU?15:28 – How do you make the most of research time when there’s no time to do anything?19:29 – How do you handle being asked to audit a site as part of a CRO interview?24:20 – How do you approach CRO in volume recruitment?28:21 – How do you get out of the endless ROI loop with stakeholders (and stop saying “CRO”)?31:46 – Why do some CRO agencies not have designers on the team?33:47 – Outro & announcements#cro #experimentation #abtesting #aiincro #growth #productmanagement #conversionrateoptimization #experimentationculture #careerintech #experimentationnation This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

A real conversation about why A/B testing doesn’t magically fix broken businesses.We break down the biggest CRO myths, why copying case studies fails, how experimentation tools really get chosen, and what actually causes friction between CRO, UX, and research teams.For CROs, experimenters, product, growth, UX, and analytics leaders.1:38 The biggest myth in A/B testing3:00 The one-person CRO team fallacy4:34 Why copying tests never works10:44 Choosing an experimentation platform14:39 Working with “difficult” clients This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode, we go off-script and cover everything from CRO strategy to existential questions.We talk through:How to recover after being shaken at workWhy long-form content still matters in experimentationCRO differences between B2B, SaaS, and ecommerceWhy website redesigns without A/B testing backfireHow beginners can gain real CRO experienceWhat it actually takes to get people to listen to dataAnd yes… the meaning of life (besides 42)This episode is less about “perfect answers” and more about how CRO, experimentation, and work actually play out in the real world.If you work in experimentation, product, growth, or analytics, this one will feel uncomfortably familiar.Chapters00:00 – How to recover after being shaken at workReflection, time, and physical reset after tough moments02:10 – End-of-year reflection and career resetsBurnout, misfit roles, and rediscovering intrinsic motivation04:25 – Why short-form content fails complex industriesAttention, doomscrolling, and why long-form still matters07:18 – Listener Q&A begins: CRO across business modelsB2B vs SaaS vs ecommerce testing realities12:02 – The hidden buyer problem in B2B CROResearchers vs decision-makers and misaligned incentives13:44 – Website redesigns without A/B testingPolitics, risk, and how to introduce testing anyway18:16 – How beginners can get real CRO experienceSide projects, free tools, and hands-on learning23:46 – The meaning of life (other than 42)Creating your own meaning rather than finding one26:58 – Getting people to listen to data at workInfluence, likability, and organizational dynamics31:24 – Culture fit, authority, and when to move onWhy sometimes it’s not you, it’s the environment32:23 – Wrap-up and next episode teaser This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Koalatative's Gerda Thomas and Experiment Nation's Rommil Santiago answer questions from the CRO community.Chapters3:35 – Is there room in CRO for AI skeptics anymore?4:58 – How can I feel like I'm using my CRO skills for good?11:29 – AI is boring and speeding up the decline of civilization. Agree or disagree?15:22 – How do I tell my micromanaging manager to shut up?18:43 – Why are you the way that you are?23:48 – How do you share a complete story with clients so they actually hear both sides of the research?26:39 – How do you calculate ROI for my CRO program for a client?29:17 – When you have no ideas on what test to run where do you get ideas? This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Koalatative's Gerda Thomas and Experiment Nation's Rommil Santiago answer CRO questions from the community.Chapters:1:51 – What's the one thing people on internet still refuse to automate or use AI for that you think they should?”5:07 – How do you measure the impact of your CRO program over time?10:37 – What to do when leadership has a win at all cost mentality, choosing short-term wins over long-term ones. How can a proper experimentation program thrive?15:27 – How should checks and balances happen in an organization on data integrity? Who should own those roles?20:10 – Should a testing tool have a warning that says, 'Your win rate is too high. Check your data type warning.'?22:49 – It seems like every day it gets harder to land a quality job in CRO with some of my colleagues taking up to a year to bounce back from a layoff. Is it worth joining the field anymore? This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Koalatative's Gerda Thomas and Experiment Nation's Rommil Santiago answer your CRO questionsPart 1 of 3Chapters:8:05 – What is the right balance between big tests versus small and medium tests?12:47 – How many iterations per test will you run knowing the first test was negative but still see some relevant data toward the hypothesis?14:32 – In small traffic sites, is it best to run one A/B test or launch a test under pre-post test assumption?17:40 – What’s the one thing most people in experimentation still refuse to automate or use AI for that you think they should do it?-----Catch Gerda on Koalatative's channel here: This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode of Experiment Nation, host Jason Gossit sits down with Ameliano Blanca, a senior growth and product leader with 10+ years of experience driving revenue through data-driven experimentation.They dive deep into:How to build experimentation programs from the ground upThe balance between qualitative and quantitative insightsExperimentation in startups vs. mature organizationsCRO frameworks and the power of iterationHow to avoid "random testing mode" and adopt structured frameworksIf you’re a growth, product, or CRO professional, this episode is packed with actionable insights to help you scale experimentation programs, improve conversion rates, and create sustainable growth strategies.👉 Get our new book: Prove It or Lose It – The (Mostly) No-Nonsense Guide to Surviving Experimentation Program Drama at experimentnation.com and all major bookstores.📌 Chapters:0:00 – Why tests should be rerun after 12 months0:19 – Welcome & introduction with Jason Gossit0:35 – Meet Ameliano Blanca: growth & product leader1:32 – First steps in building experimentation programs2:42 – Measuring conversion funnels & revenue impact3:46 – Tools for gathering quantitative & qualitative insights5:05 – Heatmaps, surveys, and finding hidden gems6:00 – Drafting test ideas & prioritization7:19 – Using and adjusting the ICE framework9:50 – The “three strikes” rule for failed experiments11:16 – Experimentation in startups vs. mature companies13:00 – Why rerunning startup tests is critical14:57 – When to pause or rerun losing experiments15:29 – Complementing quantitative data with customer interviews17:16 – Running experiments at scale in larger organizations18:18 – Balancing speed vs. rigor in CRO testing20:04 – Defaulting experiments & tracking long-term results20:11 – CRO frameworks and the power of iteration22:26 – Why iteration is underrated in experimentation23:44 – Adapting frameworks to your company & industry25:18 – Advice for teams stuck in random testing mode27:39 – How CRO perspectives have evolved over 10 years28:39 – Growth, insights, and the rise of experimentation in strategy30:38 – Real-world example: financing simulator boosts conversions This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode of Experiment Nation, host Jason sits down with Namata Agarwal, a seasoned product designer with over 17 years of experience building digital products, websites, and apps.Together, they explore:- Ethical testing: how to balance user needs with business goals.- Leaks vs. puddles framework: a practical way to identify urgent vs. non-urgent product problems.- The role of design in CRO and experimentation.- Frameworks, guardrails, and collaboration strategies for building an experimentation culture.Whether you’re a CRO specialist, product manager, or experimentation nerd, this episode is packed with actionable insights you can bring to your own team.👉 Don’t forget to like, comment, and subscribe to stay updated on the latest in experimentation and growth!📖 Check out Prove It or Lose It: The (Mostly) No-Nonsense Guide to Surviving Experimentation Program Drama .⏱️ Chapters with Timestamps0:00 – Why design must be involved early0:22 – Welcome & intro to Jason0:43 – Guest intro: Namata Agarwal0:55 – Topics: Ethical testing & Leaks vs. Puddles1:18 – What is ethical testing?2:22 – Balancing business needs vs. user needs2:54 – Handling stakeholder pushback4:01 – Misaligned metrics and real solutions5:17 – The Leaks vs. Puddles analogy explained6:32 – Why better hypothesis building matters7:48 – Using Airtable & frameworks for test prioritization8:40 – Why experimentation must be collaborative9:24 – Reframing experiments around the user journey10:00 – Unpredictable user journeys & design challenges11:12 – Involving guest collaborators in experimentation11:50 – Mapping user journeys with Miro & qualitative research12:50 – Jobs To Be Done & other qualitative frameworks14:44 – Turning qualitative insights into hypotheses15:32 – How product design elevates CRO strategies16:11 – The importance of forms in experimentation17:36 – Why design should be included from day one18:20 – Design is not just visuals—it’s about impact19:11 – Governance, responsibility, and inclusivity in testing19:46 – Leak vs. Puddle framework deep dive22:58 – Running collaborative workshops25:45 – The “Stinky Fish” workshop format27:21 – Micro vs. macro frameworks for problem-solving28:05 – Balancing bold ideas vs. safe bets29:19 – Planning tests like an investment portfolio30:13 – Testing across 13 sites at scale30:26 – What Namata would A/B test in daily life31:57 – Art, painting, and creative balance32:42 – Martial arts & keeping calm32:56 – Wrap up & final thoughts33:14 – How to connect with Namata This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com