Advertise on podcast: The Experimentation Edge

SummaryMary Alfheim, VP Data and Analytics at Headspace, joins host Ashley Stirrup to explain why more engagement is not always better when your members come to you with real symptoms. She walks through how Headspace experiments on Ebb, its AI companion, by defining inputs, outputs and guardrails for every test, with clinicians in the loop and stop rules that let teams self-serve safely.     Mary also shares two hard-earned lessons from earlier in her career: an 18 month redesign that taught her to test early and in small pieces, and a payment processor test where the randomization was not what it seemed. She closes with how Headspace is scaling experimentation and why every portfolio needs a learning bucket. For data, product and growth leaders experimenting on AI features and long term outcomes.     Chapters00:00 Cold open01:02 Headspace and Mary's path from Amazon and HBO03:26 Why more engagement is not always better09:25 Predicting long term outcomes with composite metrics13:29 Testing Ebb: inputs, outputs and guardrails19:28 A/B testing LLM features22:35 Office hours and a culture of experimentation28:46 The redesign that waited too long to test32:03 Know your treatment: the payment processor test38:11 Faster tests and the learning bucket     Takeaways- At Headspace the target is health outcomes, not time in the app, so engagement metrics sit alongside self-reported measures of how members feel- Every experiment on Ebb, Headspace's AI companion, is defined by inputs, outputs and guardrails, with clinicians doing clinical QA- Stop rules written before launch let teams self-serve: lift with no guardrail harm ships, guardrail harm rolls back- Test early and in discrete pieces; an 18 month redesign could not show which change was underperforming- Keep a learning bucket of experiments that never have to report a win, alongside small loop tests and big bets     Connect with the GuestCompany Website: https://www.headspace.com     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

SummaryArv Ramchandani, Senior Product Manager for Experimentation & Analytics at Walmart Connect, runs ad experiments on pages owned by other teams, from search to the conversational AI shopping agents, each with its own primary metric. He explains how his team negotiates a degradation threshold with every page owner, why relevance and latency sit next to revenue as guardrails, and how months-long ad-free holdouts measure what ads cost in customer trust. Host Ashley Stirrup also hears the story of a cross-sell test everyone expected to win: showing complementary items on searches like banana encouraged exploration, but that engagement did not consistently translate into the conversion outcomes the team had hoped for. That highlighted an opportunity to better align the experience with shopper intent, so the team rebuilt around the customer's mission. The rebuilt version delivered better CTR and better conversion, and the team submitted a white paper on the work to the AAAI 2027 Innovative Applications of AI conference for consideration. Arv closes with how to design experiments for learnings instead of launches and why self-serve experimentation is the next step beyond the analyst. For product, analytics and retail media teams who run experiments on shared pages.     Chapters00:00 Intro01:42 Balancing metrics across search, shopping agents and advertisers03:08 Guardrails, degradation thresholds and long-term holdouts05:58 The banana problem: rethinking a cross-sell idea08:20 Rebuilding around the customer's mission11:16 Designing experiments for learnings, not launches13:48 Trustworthy experiments and where experimentation goes next15:56 Wrap     Takeaways- Every page owner has their own primary metric; ads teams negotiate a degradation threshold per page and stay inside it- Revenue is not the only guardrail: relevance and latency are long-term metrics, and ad-free holdouts running for months measure what ads cost in trust- Customers engaged with complementary recommendations, but that engagement did not always translate into purchases; the fix was reducing diversity and building themes around the customer's mission- Start every experiment by asking how to make the product more usable, not how to launch it; the hypothesis changes and so do the metrics- A written process (test plan, no early peeking, end-of-test checks) catches bugs early and builds trust; self-serve tooling is what lets PMs experiment without an analyst     Connect with the GuestArv Ramchandani LinkedIn: https://www.linkedin.com/in/arv-ramchandaniCompany Website: https://www.walmartconnect.com     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

SummaryWhat do you do when your big launch increases engagement and tanks conversion? On this episode of The Experimentation Edge, host Ashley Stirrup talks with Lina Blackman, Director of Product Analytics at Squarespace, about the blank template launch that flopped — and how its learnings became Blueprint, Squarespace's AI-guided website builder. Lina explains how her embedded analyst team runs 150–200 experiments a year for 3 million customers, the two questions she asks every time a test loses, why teams only need one or two big wins a quarter, how Squarespace calibrates statistical certainty to business stakes, and where AI belongs (and doesn't) in the A/B testing workflow. For product managers, data scientists, and experimentation leaders who want to extract more learning from every test. Chapters 00:00 Introduction: Lina Blackman, Director of Product Analytics at Squarespace 01:45 Squarespace's business and 3 million website customers 02:30 Decentralized analysts, centralized experimentation program 04:15 150–200 experiments a year: onboarding, mobile, checkout, pricing 04:55 The blank template disaster that became Blueprint AI 07:45 Two questions for every losing test 09:30 Moving ship-first teams up the experimentation maturity curve 12:30 A/B test logs and insights rituals 13:30 North Star metrics and the KPI tree 16:35 AI in the A/B testing workflow — and what stays manual. Takeaways Stated preference lies: users asked for a blank canvas, but behavior demanded guided design — and only the experiment could referee.Close every losing test with two questions: did it work for a granular segment, and is the idea worth further investment?One or two big wins a quarter is a healthy hit rate when you run 150–200 experiments a year.Calibrate certainty to stakes — tight bounds on revenue and pricing tests, wider bounds on engagement tests so teams don't spin on noise.Hand AI the mundane parts of the workflow (tracking, assignment setup), but if AI runs the brief and the analysis, ask why you're running the test at all. Connect with the GuestLina Blackman LinkedIn: https://www.linkedin.com/in/linanguyenAbout the guest: https://www.growthbook.io/podcast/guests/lina-blackmanEpisode page: https://www.growthbook.io/podcast/episode/1-17Company Website: https://www.squarespace.com SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/

SummaryCraig Kistler, VP of Experience Design, Personalization, and Experimentation at Signet Jewelers (the parent company of Kay, Jared, Zales, Peoples, and Banter), joins host Ashley Stirrup on The Experimentation Edge to unpack how a hybrid online-and-in-store jewelry retailer runs experimentation at scale. Craig shares the counterintuitive "view all" experiment where his team blocked the product grid, added friction on purpose, and grew revenue; why he optimizes for revenue per visitor instead of conversion rate; and how Signet deliberately traded a high-volume testing program for fewer, higher-value experiments. A practical listen for product managers, designers, and experimentation leaders building programs that compound. Chapters00:45 What Signet Jewelers actually is (Kay, Jared, Zales, and more)01:45 From art school to UX to experimentation: Craig's background04:45 How experimentation is organized: a centralized model across brands06:45 From 40–50 tests a quarter to 15–25 value-driven experiments08:45 The "view all" experiment: adding friction to grow revenue12:45 One product page, many stakeholders: financing, warranties, chat15:45 Extracting learnings when an experiment loses18:45 Why revenue per visitor beats conversion as the north star20:15 Intent-based personalization and "engagement season is every day"24:45 Bringing the whole org along by tying insights to dollars. Takeaways- Friction can increase revenue. Blocking the "view all" grid and forcing a style choice sent shoppers deeper and lifted conversion and revenue, because the extra click added value.- The three-click rule is conditional. Clicks only hurt when they're empty; a click that narrows thousands of options to dozens is a feature, not a cost.- Revenue per visitor is the honest north star. Conversion rate can be gamed to 100% by making everything free or cutting bounce-heavy traffic; revenue per visitor can't.- Fewer, bigger experiments beat high volume. Signet went from 40–50 tests a quarter to 15–25 because complex, value-driven tests produce reusable insights that small tweaks don't.- Tie every result to dollars. Translating experiment outcomes into revenue is how Craig keeps financing, warranty, and chat stakeholders aligned and gets executives to act. Connect with the GuestCraig Kistler LinkedIn: https://www.linkedin.com/in/craigkistler/About the guest: https://www.growthbook.io/podcast/guests/craig-kistlerEpisode page: https://www.growthbook.io/podcast/episode/1-16Company Website: https://www.signetjewelers.com SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/

SummaryRingCentral's Director of Product Management for AI Products, Mayank Agarwal, joins host Ashley Stirrup to dismantle the metrics most teams use to judge AI agents. Drawing on his background founding an AI-first quantitative trading firm and scaling Groupon's bookable marketplace, Mayank explains why accuracy and thumbs-up/down feedback both mislead, and introduces DART — a four-metric behavioral framework (decay, acceptance, relevance, task completion) ported from how he measured trading strategies. He also breaks down a Groupon flash-discount experiment that backfired and the scarcity pivot that fixed it. Essential listening for product managers, engineers, and data scientists building or measuring AI features. Chapters00:00 Welcome and Mayank's path from quant trading to RingCentral AI02:45 Why experimentation has to be owned cross-functionally04:55 Small experiments that compounded to a 12% lift at Groupon06:45 Why accuracy and thumbs-up/down fail for AI agents08:15 The DART framework, metric by metric12:45 Applying DART to AI-generated smart notes14:55 The Groupon flash-sale that dropped conversion16:45 Swapping price urgency for scarcity and social proof19:45 North Star metrics, guardrails, and Goodhart's law26:45 The future: experimenting on — and for — AI agents Takeaways- Accuracy is a comfortable lie. It grades a narrow test set and can stay high while the agent fails real users.- Thumbs-up/down feedback is sparse and skewed. Unhappy users rarely rate — they just quietly stop using the product.- DART measures behavior, not opinions. Four signals read off logs and transcripts: decay, acceptance, relevance, and task completion.- Acceptance rate is the trust metric. The share of output users keep without editing is the strongest available proxy for trust.- A losing experiment is paid-for information. Groupon's flash-sale flop revealed the lever was wrong, not the goal — scarcity beat price-based urgency. Connect with the GuestMayank Agarwal LinkedIn: https://www.linkedin.com/in/mayank-agarwal-6223b04a/About the guest: https://www.growthbook.io/podcast/guests/mayank-agarwalEpisode page: https://www.growthbook.io/podcast/episode/1-15Company Website: https://www.ringcentral.com SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/

SummaryOn this edition of The Experimentation Edge, Ashley Stirrup talks with Geoffrey Bell, Experimentation Product Specialist at Ford Credit, about building an experimentation practice inside a captive auto lender. Geoffrey shares the losing test that earned his program credibility, the "experimentation piggy bank" he picked up at Microsoft, and the breakthrough of connecting online experiments to offline dealership receivables. The throughline: a program proves its worth not just by the wins it ships, but by the expensive mistakes it prevents and the revenue it can finally trace. It's for product managers, data scientists, and growth leaders who want experimentation taken seriously by the business. Chapters00:00 Intro01:15 Geoffrey's path: Lowe's, Microsoft, Ford Credit07:15 How Ford Credit fits with Ford Motor10:15 The teams behind every Ford Credit page15:15 The vehicle selector test that lost on purpose19:15 Why feature placement beats feature ideas21:15 Personalization and the shrinking-audience problem25:15 Telling the story when a test loses30:45 Connecting an online test to an offline car sale33:55 The experimentation piggy bank Takeaways- 1. Losing tests often create more value than winners because they stop expensive mistakes before they ship.- 2. Measure experimentation two ways: the revenue you earn from wins and the revenue you save by killing bad experiences.- 3. A feature that fails early in a flow can succeed later; placement and timing often matter more than the idea itself.- 4. Connecting online experiments to offline outcomes like receivables turns a small lift into a number leadership cares about.- 5. When you struggle to land a result, lead with the story of what the customer did, then bring the numbers. Connect with the GuestGeoffrey Bell LinkedIn: https://www.linkedin.com/in/geoffrey-bell-62a03617/About the guest: https://www.growthbook.io/podcast/guests/geoffrey-bellEpisode page: https://www.growthbook.io/podcast/episode/1-14Company Website: https://www.ford.com/finance/ SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/ (00:00) - Intro (01:15) - Geoffrey's path: Lowe's, Microsoft, Ford Credit (07:15) - How Ford Credit fits with Ford Motor (10:15) - The teams behind every Ford Credit page (15:15) - The vehicle selector test that lost on purpose (19:15) - Why feature placement beats feature ideas (21:15) - Personalization and the shrinking-audience problem (25:15) - Telling the story when a test loses (30:45) - Connecting an online test to an offline car sale (33:55) - The experimentation piggy bank

This episode of The Experimentation Edge explores how A/B testing, feature flags, and user research transformed Atlassian's talent product after it failed with its first users. Andrew Willingham — 11 years at Amazon, now Head of Legal and People Products at Atlassian — shares how product experimentation works when you can't test at scale, why your customer and your user are not the same person, and how the metrics you choose decide which experiments you can even run. SummaryAndrew Willingham, Head of Legal and People Products at Atlassian, spent 11 years at Amazon before joining Atlassian a year ago. His path from running A/B tests on millions of Amazon shoppers to building talent management software for a few hundred thousand employees forced a fundamental shift: when you can't run tests at scale, you have to sit with your actual users and watch them fail. He shares how building a talent review product for Amazon's HR specialists completely flopped when handed to HRBPs — and why that failure taught him more than any winning experiment. Now at Atlassian, he's applying that same rigor to reimagining hiring processes with AI, testing everything from recruiter screens to interview sequences that the industry has run the same way for decades. Timestamps03:09 From marketing Amazon's mobile app to building HR software for 1.5 million associates08:19 Why a talent review product loved by IO psych experts flopped with actual HRBPs11:11 How A/B testing helps product managers escape opinion-based politics15:25 Testing copy that changes behavior: "We'll generate that status report for you"17:20 The two North Star metrics Andrew optimizes: efficiency and quality19:05 Khan Academy's metric trap: measuring cognitive engagement, not just completion21:10 Why product managers resist experimentation — and what changes when you admit you don't know Takeaways- Your customer and your user may not be the same person — building for HR specialists instead of the HRBPs who actually run talent reviews resulted in a feature nobody could use.- When you can't test at scale, desk rides replace A/B tests — sitting with users and watching them struggle reveals failures faster than any dashboard.- Experimentation short-circuits political debates by removing opinion from product decisions.- Test metrics before you test features — usage time could signal engagement or just mean your product takes too long to do its job.- The experiments that fail deliver the most valuable learnings, especially when you expected a slam dunk. Connect with the GuestAndrew Willingham LinkedIn: https://www.linkedin.com/in/andrewwillingham/About the guest: https://www.growthbook.io/podcast/guests/andrew-willinghamEpisode page: https://www.growthbook.io/podcast/episode/1-13Learn more about Atlassian: https://www.atlassian.com/ SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/ Topics: A/B testing, product experimentation, feature flags, user research, talent management, qualitative research, metric design, experimentation at scale, growth experimentation.

This episode of The Experimentation Edge unpacks how DoorDash's experimentation platform runs 12,000+ A/B tests per year across 42 million monthly active users — and now powers merchant-led testing on menu pricing and promotions. Ilya Izrailevsky, Senior Engineering Manager leading the platform, explains how feature flags, marketplace experimentation, and CEO-level experiment reviews built a multi-million-dollar experimentation culture across consumers, dashers, and merchants. SummaryMost companies struggle to scale experimentation beyond engineering teams. DoorDash runs over 12,000 experiments per year across 42 million monthly active users — and now they're enabling restaurant owners to run their own A/B tests on menu pricing and promotions. Ilya Izrailevsky, Senior Engineering Manager leading DoorDash's experimentation platform, shares how the company built a three-sided marketplace testing program that balances consumers, dashers, and merchants across 40+ countries. From his time scaling search at Amazon (where offline model evaluation narrowed hundreds of candidates down to 10 for live testing) to preventing DashPass churn at DoorDash, Ilya reveals what happens when experimentation scales beyond product teams — and why CEO-level experiment review emails drive cultural change faster than any training program. One standout learning: expanding delivery radius to 11+ miles increased grocery orders but tanked retail conversions. The lesson wasn't about distance — it was that one metric approach breaks in multi-dimensional marketplaces. DoorDash now segments experimentation by vertical, behavior pattern, and regional market, using AI agents to mine institutional knowledge from past tests and auto-generate experiment summaries that ship company-wide within hours of readout. Timestamps00:40 From building Wasabi (Intuit's open-source platform) to running ML at Amazon and Uber03:04 Why product velocity without experimentation creates feature bloat, not impact05:32 Scaling search at Amazon: billions of products, 10 visible results, 25% win rate08:22 Offline evaluation as a filter — golden data sets cut model candidates before live traffic10:23 DoorDash's three-sided marketplace: 300 million feature flag evaluations per second12:38 CEO Tony Xu reads every experiment email and replies with alternative hypotheses13:33 Democratization at scale: enabling merchants to A/B test menu pricing and promotions17:05 DashPass churn experiment uncovered value perception gap — became a full product area22:03 Why expanding delivery radius killed retail orders but boosted grocery conversions24:16 No single North Star metric — balancing consumer quality, dasher earnings, merchant mix27:29 Four-dimensional scale: democratization, global expansion, new verticals, AI agents31:03 Agentic experimentation: AI mines past tests to generate hypotheses and debug imbalance Takeaways- Win rate matters less than learnings per test — DoorDash ships company-wide experiment summaries (win or lose) that the CEO actively reads and responds to, creating cultural accountability around testing rigor.- Offline evaluation acts as a pre-filter for model velocity — Amazon's search team used golden data sets to cut hundreds of ML candidates down to 10 for live A/B testing, preventing wasted experiment slots.- One-size metrics break in multi-dimensional marketplaces — DoorDash balances consumer retention, dasher utilization, and merchant inventory mix across verticals because optimizing one side degrades the ecosystem.- Democratization requires opinionated templates, not open-ended tools — enabling non-technical users to run tests means embedding success metrics and guardrails into pre-built experiment configs.- AI scales institutional knowledge, not just analysis speed — mining past experiment readouts to auto-generate new hypotheses turns your testing history into a compounding advantage. Connect with the GuestIlya Izrailevsky LinkedIn: https://www.linkedin.com/in/ilyaizrailevsky/About the guest: https://www.growthbook.io/podcast/guests/ilya-izrailevskyEpisode page: https://www.growthbook.io/podcast/episode/1-12Learn more about DoorDash: https://www.doordash.com/ SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/ Topics: A/B testing, experimentation platform, feature flags, marketplace experimentation, machine learning, growth experimentation, statistical significance, experimentation culture, agentic AI workflows.

This episode of The Experimentation Edge shows how UPS's A/B testing program drove $500M+ in incremental revenue across 80+ customer-facing applications. Dave Massey — head of the J.E.D.I. team (Journey Experience and Design Innovation) — walks through the first test that proved UX could move revenue, how he defended counterintuitive results to skeptical execs, and how a small experimentation team can override opinion with data at enterprise scale. SummaryDave Massey walked into UPS in 2016 and immediately got pulled into a meeting about AB testing tools. By the end of the day, he owned the platform—and the problem: UPS hadn't run a single meaningful experiment. Three years later, senior leadership gave him a hard number to hit. Prove UX could move revenue, or the pilot dies. His first test—removing navigation from the checkout flow—delivered $35 million in incremental revenue. Senior leaders didn't believe it. They made him defend the results upside down and sideways. When the dust settled, the data held. Today, Massey's team has driven over half a billion dollars in incremental revenue by treating UPS.com like the e-commerce business it actually is. Massey's approach is simple: test everything, especially what senior leaders think will work. His team, Journey Experience and Design Innovation (nicknamed J.E.D.I.), has built a reputation for saying no with data, not opinion. When a business unit demanded required recipient emails to capture customer data, J.E.D.I. ran the test in 24 hours and killed it. Conversion tanked. Two years later, the international team asked for the same feature—but framed it as a customs solution. That test passed. Same feature, different reason, different outcome. That's the edge Massey's team delivers: rigorous hypothesis design, a UX research team embedded in the experimentation workflow, and zero tolerance for untested ideas. Timestamps03:09 Dave's first day at UPS: inheriting an AB testing tool with no program05:59 Senior leadership's ultimatum: prove UX ROI or kill the pilot08:38 First test result: $35M from removing navigation in checkout09:48 Defending the numbers: how Massey's team survived scrutiny11:07 Why a data-driven engineering culture made experimentation inevitable16:12 Team size: 80 people supporting almost 80 customer-facing applications19:08 The 24-hour test: when required email fields killed conversion22:28 Why Massey embeds UX research inside the experimentation team24:41 AI at UPS: treating it as a tool, not a replacement Takeaways- - Massey's first test removed navigation from UPS's shipping checkout flow and delivered $35 million in incremental revenue—proving e-commerce best practices apply even when customers think "this is just a tool, not e-commerce."- J.E.D.I.'s win rate stays high because UX research and experimentation teams operate under the same leader, giving the program both behavioral metrics and voice-of-customer insight before tests ever launch.- When senior leaders push ideas, Massey's team tests them instead of arguing—then delivers results that either validate the idea or identify three better alternatives the data actually supports.- The same feature (required recipient email) failed for customer data capture but passed for international customs—proof that framing and customer benefit matter more than the feature itself.- UPS runs everything centrally now, but the real win is that demand for testing has decentralized—business units across the company now come to J.E.D.I. asking to test their ideas. Connect with the GuestDave Massey LinkedIn: https://www.linkedin.com/in/masseycreates/ About the guest: https://www.growthbook.io/podcast/guests/dave-masseyEpisode page: https://www.growthbook.io/podcast/episode/1-11Learn more about UPS: https://www.ups.com SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/ Topics: A/B testing, experimentation, conversion rate optimization, feature flags, UX research, e-commerce experimentation, statistical significance, experimentation team building, growth experimentation, sequential testing.

SummaryMost e-commerce companies test a handful of features each month. Fanatics runs nearly 100 experiments monthly and delivers a big portion of the company's total annual growth through experimentation alone. Medha Umarji, VP of Growth and Experimentation at the multi-billion dollar sports merchandising retailer, explains how she built a program that scales from 10 tests per month to 100—and maintains enough rigor to spot false positives before they become costly decisions. The difference isn't tooling or headcount. It's culture. When your CEO reads Excel spreadsheets for fun and actively wants data to prove him wrong, you stop debating whether to test and start debating how to test smarter. Medha shares the frameworks Fanatics uses to balance speed with rigor: a "do no harm" track for brand plays that won't show up in conversion metrics, a small-sample framework for teams that can't hit statistical significance thresholds, and an experimentation Wiki that feeds a continuous iteration flywheel. One surprising test on ad removal initially showed 95% statistical significance—until they replicated it and found the result was a false positive. The lesson: even at scale, you need to double-click on causality. Timestamps03:09 How Fanatics scaled from 10 to 100 experiments per month over 10 years05:25 Why some leadership teams embrace experimentation and others resist it07:06 How experimentation consistently delivers a big portion of Fanatics' annual growth08:20 What happens when your CEO consumes Excel spreadsheets and questions everything10:35 How top-down humility shapes an entire company's testing culture12:10 The ad removal test that looked like a 95% win—then failed replication15:55 How Fanatics built an experimentation Wiki that powers their growth engine22:45 The "do no harm" framework for features that don't measure cleanly in A/B tests25:20 Why lowering barriers to adoption matters more than statistical perfection early on26:27 Your odds of winning at experimentation are worse than roulette Takeaways- Replication catches false positives: A 95% confidence level still means 1 in 20 results are noise—if a critical test outcome can't be explained through micro-metrics, run it again before committing resources.- Top-down buy-in shifts the conversation from "why test?" to "how do we test?": When leadership treats data as the tiebreaker, teams stop defending opinions and start building better experiments.- Frameworks like "do no harm" and "small sample" expand who can test: Not every initiative needs 30,000 orders to ship value—lower the barrier for teams that can't hit statistical thresholds while protecting core KPIs.- Documenting experiments in a centralized Wiki creates a growth flywheel: Fanatics' Wiki feeds their roadmap with iterations on already-built features, reducing tech dependency and accelerating velocity.- Micro-metrics establish causality beyond top-line KPIs: If revenue moves but scroll depth, cart adds, and product views don't follow the same pattern, question the result before declaring a win. Connect with the GuestMedha Umarji LinkedIn: https://www.linkedin.com/in/medhaumarji/About the guest: https://www.growthbook.io/podcast/guests/medha-umarjiEpisode page: https://www.growthbook.io/podcast/episode/1-10Learn more about Fanaticshttps://www.fanatics.com/

SummaryChess.com ran its first A/B test in 2023. Two years later, the team is on track to run 1,000 experiments in a single year—and they've already shipped 195 in Q1. In this episode, Ashley Stirrup sits down with Nafis Shaikh, Director of Product Management at Chess.com, to get inside the experimentation engine powering one of the world's most beloved gaming products. Nafis brings experience from Zynga and Prodigy and a refreshingly honest take on what changes when a product built on passion suddenly has to serve a 10-million-DAU user base that spans absolute beginners to rated FIDE players. He and Ashley get into why one-size-fits-all doesn't actually fit anyone, how to measure an AI coach when you can't tell whether users have their volume on, and a game review experiment that completely upended the team's assumptions about how players want to learn. Nafis also shares practical advice for product managers trying to introduce experimentation culture to organizations that have never done it before—starting with a simple pre/post test rather than a fancy platform. If you lead product, care about experimentation maturity, or just want to hear how a classic product is scaling its learning loop, this one's worth your time. Timestamps- [00:35] – Chess.com's experimentation origin story and the 1,000-test goal- [05:01] – Designing for a user base that spans beginners to FIDE-rated players- [07:30] – The four metrics dimensions Nafis uses to evaluate tests- [12:03] – How do you A/B test an AI coach when you can't tell who's listening?- [15:49] – Embracing humility and the shift away from "we know what works"- [20:50] – The game review test that surprised everyone: 80% of users review wins- [24:06] – Advice for PMs introducing experimentation at a new company- [29:15] – The onboarding debate and personalization from session zero Takeaways- Scale test volume to learning speed, not just shipping speed- Build hypotheses around user psychology, not just KPI movement- Accept that being wrong is the point—experimentation only works when leadership embraces humility- Start simple if you're new to experimentation; a clean pre/post comparison beats a fancy platform you don't use- Reposition features around how users actually feel, not how you assume they should feel- Design onboarding around the shortest path to value, not the longest path to personalization Connect with the GuestNafis Shaikh LinkedIn: https://www.linkedin.com/in/nafis-shaikh-20161916/About the guest: https://www.growthbook.io/podcast/guests/nafis-shaikhEpisode page: https://www.growthbook.io/podcast/episode/1-9Company Website: https://www.chess.com

SummaryWhat happens when A/B testing stops being a tool and becomes your operating system? Suresh Teckchandani, VP of Product & Technology at Ancestry (formerly PayPal and eBay), shares how the team scaled experimentation from isolated tests to capability-building that drives roadmap and revenue. He details the “growth metering” and paywall experiments that unlocked a 5.3% lift in key engagement and improved conversions—then became platform features. Suresh explains Ancestry’s centralized experimentation platform with self-serve access for PMs and engineers, why “obvious” UX changes can be the riskiest, and how removing friction actually hurt engagement by 20–25% due to user mental models. He also breaks down a major growth lever: AI-powered storytelling that turns raw records into narratives, delivering 30%+ CTR lift and a 5x increase in story views. You’ll learn how Ancestry balances input vs. output metrics, when not to test, and why the best leaders optimize for decision quality over win counts—with clean baselines, right audiences, adequate sample sizes, and true statistical significance. Timestamps[00:45] – Ancestry’s experimentation maturity: metering, paywalls, and the 5.3% lift that unlocked capabilities[03:48] – From isolated tests to a capability mindset: experimentation as an operating system[05:34] – Balancing wins with learning: zooming out for subscription engagement and NPS[08:49] – Operating model: centralized platform, self-serve dashboards, and baseline resets[11:59] – Counterintuitive UX lesson: removing friction backfired (–20–25% CTR); respect mental models[15:10] – AI storytelling as a growth lever: record comparisons into narratives, 30%+ CTR and 5x views[18:27] – Input vs. output metrics: when to roll back and how to link short- and long-term outcomes[30:00] – Parting advice: test what changes CX, avoid vanity testing, and optimize decision quality Takeaways- Build capabilities, not just tests—use experiments to unlock platform features (e.g., metering, paywalls).- Democratize experimentation with a centralized platform and self-serve tooling; reset baselines regularly.- Test “obvious” UX changes; preserve helpful friction and align with user mental models.- Turn data into narratives with AI to deepen engagement and increase discovery.- Define input and output metrics; ship only what improves core outcomes (retention, sign-ups), and roll back fast if not.- Optimize for decision quality: right audience, sufficient sample sizes, clean baselines, and true statistical significance. Connect with the GuestSuresh Teckchandani LinkedIn: https://www.linkedin.com/in/teckchandani/About the guest: https://www.growthbook.io/podcast/guests/suresh-teckchandaniEpisode page: https://www.growthbook.io/podcast/episode/1-8Company Website: https://www.ancestry.com SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/

SummaryHow do you drive hypergrowth without guessing? Kameron Tanseli, Head of Growth Engineering at Fyxer—an AI assistant for your email—breaks down the experimentation playbook that helped the company scale from $1M to $35M ARR, with sights set on $100–$150M. Kameron explains how startups should think about A/B testing differently: de-risk big bets, not just button colors. He shares a risk-based approach to when to run rigorous tests vs. ship-and-measure, why a 25% win rate is a sign you’re testing ambitiously, and how PLG features should be shipped first, then rapidly iterated to drive usage. You’ll hear how Fyxer uses AI to speed the entire lifecycle—Claude, Cursor desktop cloud agents, GrowthBook, and BigQuery—plus how a Slack-first changelog and an internal “AI data scientist” democratize insights. Kameron also details turning everyday product usage into growth loops, personalizing signup paths, and measuring success by movement in global ARR, not just local metrics. He closes with candid advice for new growth engineers: expect to struggle early, be T-shaped, and adopt your customer’s language. Timestamps[00:34] – Startup A/B testing mindset: de-risking big bets with only a 25% win rate[02:45] – When to A/B test vs. ship: risk appetite, funnel stage, and non-inferiority tests[04:43] – 360 experiments with 4 people: scaling to 1,000 using AI and Cursor cloud agents[08:22] – Separating feature impact from momentum: PLG and trial model moves ARR[10:29] – Ship PLG features, then iterate to drive usage; measuring DAU and revenue impact[11:40] – Habit loops to growth loops: turning product features into PLG (scheduling case study)[16:47] – Building an experimentation culture: founder buy-in, Slack changelog, shared data[26:50] – The modern growth stack: Claude, Cursor, GrowthBook, BigQuery, and DOT in Slack Takeaways- Prioritize by risk: run rigorous A/B tests where you have volume; use before/after or non-inferiority for low-risk in-product changes.- Test big levers—not just UI: pricing models, usage limits, onboarding pathways—and judge success by ARR movement, not micro-metrics.- Ship first, then optimize: launch PLG features and immediately run experiments to increase adoption; track daily active usage per feature.- Build growth loops from habits: design shareable artifacts and personalized signup paths; drive users back to your domain to capture value.- Scale experimentation with AI: use Cursor desktop/cloud agents for parallel builds and visual QA; orchestrate docs/analysis via Claude; automate cleanups and reporting.- Make experimentation company-wide: centralize data (BigQuery), broadcast wins/losses in Slack via GrowthBook, and auto-correlate metric dips to releases. Connect with the GuestKameron Tanseli LinkedIn: https://www.linkedin.com/in/kameron-tanseli/About the guest: https://www.growthbook.io/podcast/guests/kameron-tanseliEpisode page: https://www.growthbook.io/podcast/episode/1-7Company Website: https://www.fyxer.com

SummaryHow do you build a culture where nothing ships without evidence—and leaders actually act on the data? Makram Mansour, Head of Marketplace at ID.me and former experimentation leader at LinkedIn and Intuit, shares the systems, mindsets, and guardrails behind “experimenting everywhere.” At LinkedIn, he helped support 10,000+ annual experiments with 2,000 weekly platform users, and he explains the hard-earned lessons (like a 5px UI tweak causing a million-dollar ad loss) that led to a “test before release” mandate. At Intuit, he operationalized “fail forward,” partnering with HR to rewrite OKRs so teams are rewarded for learning, not just launching. Makram breaks down why to shift from MVP to MVT (minimum viable test), how to surface leap-of-faith assumptions with PRFAQs and “unit of one” prototypes, and where AI now unlocks faster, safer front-end testing. He also details critical guardrails—cost visibility for AI infrastructure, ethical and inclusion metrics, and the people-process-technology triad—plus practical ways to remove bottlenecks via a center of excellence. If you’re starting from scratch or scaling your program, you’ll learn how to personalize responsibly at the top of the funnel, define your North Star and signposts, and stack early wins while building influence across the org. Timestamps[00:45] – Makram’s path: running experimentation at LinkedIn and Intuit, and why nothing ships without an A/B test[02:15] – Costly lessons: 5px banner change, algorithm tweaks, and the case for rigorous guardrails[06:40] – Leadership discipline: killing features (voice meetups, LinkedIn Stories) and changing OKRs to reward learning[11:05] – People, process, technology: top-down and bottom-up tracks, and embedding “fail forward”[13:40] – From MVP to MVT: validating leap-of-faith assumptions, PRFAQ, and rapid “unit of one” prototypes[15:55] – Bottlenecks and unlocks: engineering/data science capacity, centers of excellence, and AI for fast front-end tests[22:45] – Personalization at the top of funnel: avoid waste, design reviews, and right-size testing before building[25:45] – Guardrail metrics that matter: AI infra costs, ethics/compliance, and fairness-by-design[29:45] – ID.me now: zero-to-one builds, vision-to-values, North Star and leading indicators[33:30] – How to start at a new org: crawl-walk-run, small wins, relationships, and over-communication Takeaways- Shift from MVP to MVT: list leap-of-faith assumptions and design minimum viable tests before you build.- Institutionalize learning: align OKRs with “fail forward,” and be willing to kill low-performing features quickly.- Build the triad: pair an easy-to-use platform with training, top-down sponsorship, and clear launch processes.- Add real guardrails: track AI infrastructure costs, ethics/compliance, and inclusion metrics alongside growth KPIs.- Unblock teams: create a center of excellence for data science and enable rapid variants with AI-powered tooling.- Start small and visible: rack up quick wins, over-communicate progress, and grow influence through relationships. Connect with the GuestMakram Mansour LinkedIn: https://www.linkedin.com/in/ilmansourAbout the guest: https://www.growthbook.io/podcast/guests/makram-mansourEpisode page: https://www.growthbook.io/podcast/episode/1-6Company Website: https://www.id.me

How do you boost developer velocity in a highly regulated industry—without sacrificing safety or customer trust? Charles Williams, Senior Vice President and Software Engineering Director at Truist (formed from the BB&T and SunTrust merger), shares how his team elevates developer experience to ship faster and more reliably. Charles breaks down shifting quality “left” with automation, measuring success with both DORA metrics and developer sentiment, and why human-in-the-loop is non-negotiable for AI in finance. He details Truist’s governance model—steering committees, enterprise architecture, and clear guardrails—to avoid tool sprawl while building a purpose-built AI ecosystem: Microsoft Copilot for productivity, GitLab’s AI-enabled DevSecOps platform for engineering, and separate consumer-facing capabilities. Expect practical insights on starting with low-risk, high-yield use cases (unit tests, docs, security triage), tracking AI utilization, and upskilling teams in prompt engineering so developers can “manage” AI agents effectively. Charles also explores the path to personalized experiences balanced with privacy, why branches should be enhanced—not reduced—by AI, and the cultural skills leaders need now: empathy, neurodiversity awareness, and change management. He closes with where AI is driving ROI first—developer onboarding and pipeline productivity—with code quality gains following close behind. Timestamps[00:02] – Truist overview and Charles’s mandate: improving developer experience at scale[00:56] – AI as a strategic priority; shifting quality left with automation to remove bottlenecks[02:19] – Measuring success: DORA metrics plus sentiment—eliminating toil to drive happiness[04:33] – Human-in-the-loop AI for high-stakes finance; customer and internal use cases[07:25] – How Truist evaluates tools: personas, pain points, and starting with tests, docs, security[08:35] – The stack: Microsoft Copilot, GitLab’s AI gateway approach, and tracking utilization[10:36] – New skills and culture: prompt engineering, “managing” AI agents, and strong governance[20:45] – What’s next: personalization vs privacy, fintech agility + bank stability, and where AI pays off now Takeaways- Shift quality left with automated checks so developers catch issues early without human gatekeeping.- Measure DORA metrics and developer sentiment; remove mundane toil to increase speed and satisfaction.- Keep humans in the loop for AI-assisted coding and customer answers—trust but verify in regulated contexts.- Build an AI ecosystem with clear purposes (productivity, engineering, consumer) and a steering committee to avoid duplication.- Start with low-risk, high-yield AI use cases—unit tests, documentation, and security triage—to build confidence and momentum.- Upskill teams in prompt engineering and AI oversight so developers can effectively direct and review AI “agents.” Connect with the GuestAbout the guest: https://www.growthbook.io/podcast/guests/charles-williamsEpisode page: https://www.growthbook.io/podcast/episode/1-5Company Website: https://www.truist.com