1879839422
The Experimentation Edge

Advertise on podcast: The Experimentation Edge

Rating
★★★★★
5
from
10 reviews
This podcast has
62 episodes
Language
English
Publisher
Growthbook
Explicit
No
Date created
2026/02/24
Latest episode
2026/11/17
Average duration
31 min.
Release period
17 days

Description

How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale. Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams. Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.

Unlock The Experimentation Edge podcast Email contact info,
Listeners & Audience details

Email contact information

Direct podcast contact details

Listeners

Audience numbers & engagement insights

Audience details

Podcast Insights

Podcast episodes

Check latest episodes from The Experimentation Edge podcast


From Opinion to Evidence: Evaluating AI Ads at Walmart Connect
2026/11/17
Summary How do you measure the quality of an AI-generated ad when designers, legal teams, and advertisers may all define “good” differently? Nishanth Kadiyala, Product Leader at HelloFresh and formerly an ad tech product leader at Walmart Connect, joins Ashley Stirrup on The Experimentation Edge to explain how the team shifted from subjective creative preferences to measurable acceptability, compliance, and performance. Nishanth walks through the realities of building creative automation for smaller advertisers: segmenting users, validating AI output against a large evaluation dataset, running A/B tests to earn stakeholder buy-in, and letting real advertiser needs shape the product. He describes moving creative production from weeks to minutes as more than 250 advertisers tried the product, then shares how those lessons inform creative velocity and diversity in his current work at HelloFresh. Chapters 00:00 Meet Nishanth Kadiyala 01:16 Inside Walmart Connect’s retail media stack 05:30 Why B2B experimentation requires a different playbook 07:30 Moving from subjective quality to acceptability 09:45 Helping smaller advertisers create better ads 12:21 Iterating with user research and phased launches 14:47 Testing AI creative against human-designed ads 15:57 Building a golden dataset for AI evaluation 18:05 Scaling across products, advertisers, and categories 22:18 Defining baseline quality and performance 24:50 Unexpected lessons from advertiser feedback 26:35 Choosing the right level of AI control 28:33 Results: 250+ advertisers and minutes to create 29:44 From manual reviews to AI evaluation pipelines 31:35 Creative velocity and diversity at HelloFresh 34:57 AI’s role in the next generation of testing 36:40 Closing reflections Takeaways -An ad that looks better is not necessarily an ad that performs better; A/B testing turns aesthetic debates into learning. -AI creative must first clear acceptability guardrails for product fidelity, context, and brand compliance, even while models keep improving. -Start with the advertisers and product categories that current technology can support, then expand with evidence from real users. -A representative golden dataset and ongoing evaluations are as important as the generative model itself. -Nishanth says the Walmart Connect effort reached more than 250 advertisers and reduced creative production from multiple weeks to minutes. -Creative velocity, diversity, and excellence are now central to Nishanth’s paid-social work at HelloFresh. Connect with the Guest Nishanth Kadiyala on LinkedIn: linkedin.com/in/nishkadi For more of Nishanth’s perspective on Walmart Connect and AI-powered ads, listen to his Rethink Retail conversation. SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
What Wealthsimple decides to test and what to just ship
2026/11/12
SummaryDelina Ivanova, Director of Data Science at Wealthsimple, joins host Ashley Stirrup to share how Canada's fast-growing financial services company decides what deserves an experiment and what should simply ship. With 4 million clients and a steady stream of new brokerage and banking features, Wealthsimple has spent the last 2 years formalizing experimentation across 3 data science teams. Delina explains her rules of thumb for skipping a test, why lagging metrics like deposits and revenue force teams to pick the right early signals, and what a margin promotion taught her team about messaging, targeting and combined offers. She also shares how to diagnose a losing test, her plan for meta-learning and uplift modeling, and how AI makes it cheaper to test ideas before investing in a full build. For product, data and growth leaders who want experiments that measure the thing they are trying to drive.     Chapters00:00 Wealthsimple, 4 million clients and a never-ending experimentation world02:46 Two years of formalizing experimentation03:42 What Misplay taught her about when you need a test06:00 Rules of thumb: when to test and when to just ship08:21 Ronny Kohavi, Bing and the value of a millisecond09:26 How Wealthsimple's 3 data science teams run experiments12:15 Engagement features, lagging metrics and early signals14:49 Launching margin and finding an education gap16:57 Why surveys need experiments19:10 Sharing results and building systemic learning21:56 The margin promotion that taught a messaging lesson26:23 Experiments as a way to learn about people27:39 Messaging, targeting or experience: diagnosing a losing test30:35 Meta-learning and uplift modeling33:42 Using AI to test ideas faster and cheaper     Takeaways- Skip the test when you are removing a clear blocker for clients, and test when the change is optional or the outcome is uncertain- Deposits and revenue can lag 60 to 90 days in financial services, so pick early metrics that are proven to lead to them- Combining 2 incentives in one message hides what you are trying to learn; decouple offers so each test isolates one outcome- When a feature you believe in loses, check whether it is a messaging, targeting or experience problem, and design variants to find out- Pooling every past experiment into one view turns tests into a mega learning system for pricing, messaging and personalization     Connect with the GuestDelina Ivanova LinkedIn: https://www.linkedin.com/in/delina-ivanova/Company Website: https://www.wealthsimple.com     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
How Andy Denning decides which A/B tests are worth running
2026/11/11
SummaryAndy Denning, Product Leader, joins host Ashley Stirrup to talk about the judgment that comes before any A/B test. Drawing on his experience with a two-sided jobs marketplace serving job seekers and recruiters across the UK and US, Andy explains why underpowered, badly chosen tests can create false confidence, how experimentation brings organizational honesty, and why his teams start with the riskiest assumption instead of the most comfortable one. He breaks down a US branding test that barely moved applications, how traffic and market shape test design, and why AI makes judgment, not test volume, the real bottleneck. For product and growth leaders who want fewer wasted tests and more learning from the ones they run.     Chapters00:00 Test the riskiest assumption, not the comfortable one00:55 A two-sided jobs marketplace in the UK and US01:53 Andy's product role and AI-powered job matching02:52 Deciding what's worth testing at a smaller company04:21 Underpowered tests and false confidence04:45 Experimentation as organizational honesty05:49 From prototyping software to Optimizely06:58 Cross-functional teams that own experiments end to end08:13 Sharing learnings and popcorn sessions10:31 Guardrail metrics in regulated industries11:53 Experimenting across the UK and US markets13:38 The branding test that didn't move applications17:04 The judgment layer: is the test worth running?19:03 Experiment design starts with the riskiest assumption20:39 What to do when a test will take 6 weeks21:27 B2C vs B2B: what your traffic can support22:27 Pre-mortems, non-goals and trade-offs23:49 AI and the flood of low-conviction tests27:36 Where to spend human judgment28:26 The AI apply tsunami in hiring     Takeaways- Test the riskiest assumption, not the most comfortable one; it is the one the bet rests on- Underpowered or badly chosen tests can manufacture false confidence that is worse than not testing- Before you test, ask whether the outcome is uncertain and whether you would ship it anyway- Run the numbers up front; if a test needs 6 weeks, find a faster or cheaper way to learn- AI makes variants cheap, so the bottleneck is now the quality of the hypothesis     Connect with the GuestAndy Denning LinkedIn: https://www.linkedin.com/in/andy-denning-547a0862/ SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
How GE HealthCare tests for quality leads, not just more clicks
2026/11/10
SummaryTom Gores, Director, Product - Digital Experience & E-commerce at GE HealthCare, joins host Ashley Stirrup to share what changes when you take an experimentation program from B2C retail to B2B healthcare. Tom built his first A/B testing program at Kohl's 12 years ago and now runs a centralized program at the center of GE HealthCare's digital ecosystem, where buying cycles can take a year and each qualified lead carries much higher stakes.   Tom explains why a Contact Us test that lifted clicks also lifted junk leads, how his team turns one-off page ideas into bigger, broader tests across templates, and why planning for failure before launch keeps experimentation framed as learning. He also covers the 3 areas where he sees experimentation growing, people and culture, data, and technology, and the metrics he watches, from CSAT and conversion rate to visibility in AI search. For product, digital and growth leaders running experiments in long, high-stakes B2B journeys.     Chapters00:00 Welcome Tom Gores of GE HealthCare00:51 From Kohl's to GE HealthCare: 12 years of A/B testing02:01 GE HealthCare's digital ecosystem02:51 B2B vs B2C: longer buying cycles and the long game03:49 A centralized program with lower traffic and higher stakes05:52 The Contact Us test: more clicks, junkier leads07:26 Funneling the right buyers to the right experience08:47 Designing experiments for the most learning10:33 Planning for failure and picking leading indicators13:07 Where experimentation grows next: people, data and technology15:22 CSAT, conversion rate and AI visibility17:34 How GE HealthCare measures CSAT     Takeaways- In B2B healthcare, buying cycles can run months to a year, so experiments need leading indicators and long-term tracking rather than instant conversions- Pushing a metric in front of people can lift clicks while lowering lead quality; check results with the sales teams who work the leads- Turn a single-page idea into a bigger, broader test by applying it to a whole template, so thousands of visitors are affected instead of hundreds- Map out what you will do if a test fails before launch, so results stay framed as learning and avoid wasted builds- Track CSAT, conversion rate and AI visibility together to measure an ecosystem that serves many audiences and journey stages     Connect with the GuestTom Gores LinkedIn: https://www.linkedin.com/in/tom-gores-6925a81aCompany Website - GE HealthCare: https://www.gehealthcare.com     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
How Glassdoor turned a handful of A/B tests a year into dozens a quarter
2026/11/09
SummaryJeremy Echols, Director of Digital Analytics at Glassdoor, part of Indeed, joins host Ashley Stirrup to talk about building an experimentation program around people, not sessions. Jeremy came to analytics from an unusual path, as a former pastor with a master of divinity, and he sees math and statistics as a way to do applied psychology.   Jeremy explains how Glassdoor went from a slow homegrown system and a handful of tests a year to dozens of tests a quarter, with a center of excellence, office hours, Amplitude templates and a shared data language. He breaks down an Indeed account-linking test that exposed variant jumpers across devices, why every test needs a hypothesis and a curious set of secondary metrics, and how some friction is worth keeping. For analytics, product and growth leaders who want faster iterations and deeper learning from every test.     Chapters00:00 Sessions don't buy anything, people do00:42 Glassdoor's mission: workplace transparency01:27 Glassdoor Recruiter: a recruiter that works for the job seeker02:50 From pastor to analytics: statistics as applied psychology03:46 Why synthetic users can't replace real people04:14 Centralizing experimentation at Glassdoor05:24 Templates, guardrails and an experimentation community06:49 The account-linking test that exposed variant jumpers08:10 Why users browse without logging in09:48 Guest checkout hides the customer journey11:01 Costco and the case for personalization11:40 What to learn when a new feature fails12:11 Write the hypothesis, then ask what else might happen14:00 Tying experiments to your analytics funnels14:51 Pre-mortems and testing the opposite15:56 When removing friction hurts the funnel17:04 Turning experimentation marathons into sprints18:19 One data language for every team19:02 Glassdoor Recruiter and SuperMatch20:35 Hiring managers and passive candidates     Takeaways- Measure people, not sessions; users move across devices, so incentivize login to see the whole journey- Bake best-practice settings and guardrail metrics into test templates so no one has to remember them- Write a hypothesis for every test, then add the secondary metrics that could teach you something unexpected- Some friction is good; more leads can mean fewer qualified leads and a weaker downstream funnel- One shared data language across marketing, email and product makes experiments faster and easier to compare     Connect with the GuestJeremy Echols LinkedIn: https://www.linkedin.com/in/jeremyechols/Company Website: https://www.glassdoor.com     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
How RealPage finds its biggest A/B test wins, one funnel at a time
2026/11/05
SummaryAndrew Miller, Director of Product Management, Growth and E-commerce at RealPage, joins host Ashley Stirrup to share how experimentation works inside Loft, RealPage's end-to-end renter app. Andrew has stood up experimentation programs at 3 companies, and today he runs a portfolio of moving services where a winning A/B test ties directly to revenue.   Andrew explains how a small, centralized optimization team ships tests in Optimizely faster than engineering release cycles allow, and how he builds a culture of experimentation with funnels, group ideation, the Opportunity Solution Tree and simple test templates. He also breaks down 2 electricity experiments: one that cut choices from 8 to 3 and lifted conversion 12%, and a price display test that lost. For product and growth leaders who want more ideas, faster launches and tests that actually teach them something.     Chapters00:00 RealPage, Loft and a B2B2C business01:17 Why every A/B test in Loft focuses on the renter03:02 The moving marketplace and how it makes money04:53 Standing up experimentation programs at 3 companies07:10 The T-shaped PM: depth in data and experimentation08:57 A centralized optimization team for all of Loft11:05 Why tests run in Optimizely instead of engineering12:42 Building a culture of experimentation with funnels15:34 The Opportunity Solution Tree and test templates17:06 How good is your gut at predicting winners?19:01 Cutting electricity options from 8 to 3 for a 12% lift21:13 The price display test that lost24:26 Minimizing variables when traffic is limited26:04 Why diversity of ideas matters27:38 PMs submitting PRs, not just hypotheses30:45 Prototyping tests with AI: from 2 weeks to 1 day     Takeaways- Start with telemetry and funnels; funnel drop-offs get stakeholders to the insight before you have to say it- Run the first ideation sessions as a group and organize ideas with an Opportunity Solution Tree; one session can fill a year of backlog- Require a hypothesis, a metric and a definition of success for every test idea so learning is the baseline outcome- Fewer choices can lift conversion: cutting electricity plans from 8 to 3 raised conversion 12%- An estimate shown with too much precision can backfire; round estimated prices so they read as estimates     Connect with the GuestAndrew Miller LinkedIn: https://www.linkedin.com/in/findandrew/Company Website: https://www.realpage.com     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
How Preply runs close to 1,000 experiments a year
2026/11/03
SummarySebastien Polis, Senior Experimentation Manager at Preply, joins host Ashley Stirrup to explain how a learning marketplace with 25 to 30 product teams runs close to 1,000 experiments a year. At Preply, no change reaches customers without a test, OKRs can only be claimed through experiments with an approved target metric, and a public Slack channel puts every launch, result and kill decision in front of the exec team.     Sebastien shares why roughly 80% of experiments lose and 30 to 40% are iterations of an earlier idea, how a global control group shows double-digit growth, and what Preply learned when a control group went negative and a simple wording win turned out to be Twyman's law in action. He also covers why marketplaces randomize tutors instead of learners, the clustered analysis that requires, and his plan to build a learning system that gives AI the context it needs. For product, data and growth leaders building an experimentation culture at scale.     Chapters00:00 Preply's learning marketplace and Sebastien's path into experimentation04:52 Tying OKRs to experiments across 25 to 30 product teams06:39 Why 80% of experiments lose and 30 to 40% are iterations08:35 Double-digit growth from a global control group09:46 The control group that went negative and Twyman's law15:03 How a new PM designs an experiment at Preply17:57 A public Slack channel where execs question every test20:48 When clicks go up but learners don't buy23:23 Building a learning system before feeding AI25:20 An experiment run entirely by AI, with a human approver28:13 Randomizing tutors, not learners, in a 2-sided marketplace     Takeaways- Preply ties OKRs to experimentation: teams can only claim results through experiments that use an approved target metric- Losing is the norm; about 80% of Preply's tests fail and 30 to 40% of each year's experiments are iterations of an earlier idea- A surprising win from a simple change deserves a rerun; Preply unscaled a wording test and the lift disappeared- A public Slack channel where execs question PMs about launches and kills keeps rigor and transparency high across 1,000 tests a year- In a marketplace, randomize the tutor and measure learner outcomes with a clustered analysis, and ship bug-free code because sample size is scarce     Connect with the GuestSebastien Polis LinkedIn: https://www.linkedin.com/in/sebastien-polis-b5aa8256/Company Website: https://preply.com     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
Building a Culture of Experimentation in Sports Analytics with New York Red Bulls
2026/10/29
SummaryVamsi Krishna Kunapuli, Director of Analytics at New York Red Bulls, leads a small team that supports marketing, ticket sales and partnerships for one of the original MLS clubs and its 25,000 seat stadium. He shares how an experiment across five to six seasons changed the club's North Star from average ticket price to gross margin. Host Ashley Stirrup and Vamsi also discuss why a losing test is not a failure, why segments can never be static, where good test ideas come from, and how identity resolution will open the door to machine learning and personalized fan experiences. For analytics, marketing and product leaders building experimentation into a business where every event is different.     Chapters00:00 Intro00:20 New York Red Bulls: an original MLS club01:32 Data-driven fan engagement 04:46 Single game buyers outspend season ticket holders07:21 Testing frameworks 09:29 Holistic fan value 11:11 Segments can never be static12:59 A losing test isn't a failure14:51 Loser or lever: judging tests against the North Star17:13 Building a culture with clear end goals20:25 Ideas populate when there is more data23:10 Identity resolution, AI and personalization26:06 Empowering more teams to experiment27:57 Every game is like hosting a wedding     Takeaways- We have seen instances where single game buyers at the New York Red Bulls outspend season ticket holders once concessions and the Hype House zone are counted- The club moved its North Star from average ticket price to gross margin, which changed couponing, fan profiles and retargeting- A losing test is not a failure; judged against the North Star, a negative result shows what not to do and that a lever exists- Segments should be dynamic, and the club is building toward a separate segmentation for every event- Ideas populate when there is more data, and the stakeholders who know the business best should drive the questions     Connect with the GuestVamsi Krishna Kunapuli LinkedIn: https://www.linkedin.com/in/vamsikunapuli/Company Website: https://www.newyorkredbulls.comVamsi is a panelist at DMWF North America 2026, "Connected Data: The Foundation of Intelligent Marketing"     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
Motive's Shiva Manjunath on CRO as a mindset, not a role
2026/10/27
SummaryShiva Manjunath, Lead Web Strategist at Motive and host of the From A to B podcast, joins Ashley Stirrup to explain why CRO works best as a mindset every team carries rather than a role one team owns. He shares how building bridges to sales and customer service turns everyday complaints into test ideas. The conversation covers why customers who ask for price are often asking for value, how emotional context earns a CRO credibility, how to make an impact without an executive sponsor, and why experimentation needs the kind of marketing AI gets. For CROs, product managers and growth leaders building an experimentation culture.     Chapters00:00 Cold open: gifts under $5001:18 What a true culture of experimentation looks like04:43 CRO as a mindset across every function11:27 Become the sales team's best advocate15:29 Fear, uncertainty and doubt: the window shade story21:51 Customers ask for price when they want value24:17 GTA 6 and why button colors cost CROs trust29:14 Making an impact without an executive sponsor34:27 Making experimentation as exciting as AI     Takeaways- CRO works best as a mindset shared by design, marketing and product, not as one team's job.- Sales and customer service teams hold free qualitative data that makes strong test ideas.- When customers ask for price, test value messaging such as benefits, ROI and time saved.- Understanding the emotional context around a purchase earns a CRO more trust than tactical button tests.- A culture of experimentation needs data-driven decision making first and a safe space to lose.     Connect with the GuestShiva Manjunath LinkedIn: https://www.linkedin.com/in/shiva-manjunath/From A to B podcast: https://open.spotify.com/show/5xDlubE3GZwUDDen64hefzNewsletter: https://fromatob.beehiiv.com/     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
Engagement with safety: Headspace's guardrails for AI tests
2026/10/20
SummaryMary Alfheim, VP Data and Analytics at Headspace, joins host Ashley Stirrup to explain why more engagement is not always better when your members come to you with real symptoms. She walks through how Headspace experiments on Ebb, its AI companion, by defining inputs, outputs and guardrails for every test, with clinicians in the loop and stop rules that let teams self-serve safely.     Mary also shares two hard-earned lessons from earlier in her career: an 18 month redesign that taught her to test early and in small pieces, and a payment processor test where the randomization was not what it seemed. She closes with how Headspace is scaling experimentation and why every portfolio needs a learning bucket. For data, product and growth leaders experimenting on AI features and long term outcomes.     Chapters00:00 Cold open01:02 Headspace and Mary's path from Amazon and HBO03:26 Why more engagement is not always better09:25 Predicting long term outcomes with composite metrics13:29 Testing Ebb: inputs, outputs and guardrails19:28 A/B testing LLM features22:35 Office hours and a culture of experimentation28:46 The redesign that waited too long to test32:03 Know your treatment: the payment processor test38:11 Faster tests and the learning bucket     Takeaways- At Headspace the target is health outcomes, not time in the app, so engagement metrics sit alongside self-reported measures of how members feel- Every experiment on Ebb, Headspace's AI companion, is defined by inputs, outputs and guardrails, with clinicians doing clinical QA- Stop rules written before launch let teams self-serve: lift with no guardrail harm ships, guardrail harm rolls back- Test early and in discrete pieces; an 18 month redesign could not show which change was underperforming- Keep a learning bucket of experiments that never have to report a win, alongside small loop tests and big bets     Connect with the GuestCompany Website: https://www.headspace.com     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
How Walmart Connect learned to align cross-sell with shopper intent
2026/10/15
SummaryArv Ramchandani, Senior Product Manager for Experimentation & Analytics at Walmart Connect, runs ad experiments on pages owned by other teams, from search to the conversational AI shopping agents, each with its own primary metric. He explains how his team negotiates a degradation threshold with every page owner, why relevance and latency sit next to revenue as guardrails, and how months-long ad-free holdouts measure what ads cost in customer trust. Host Ashley Stirrup also hears the story of a cross-sell test everyone expected to win: showing complementary items on searches like banana encouraged exploration, but that engagement did not consistently translate into the conversion outcomes the team had hoped for. That highlighted an opportunity to better align the experience with shopper intent, so the team rebuilt around the customer's mission. The rebuilt version delivered better CTR and better conversion, and the team submitted a white paper on the work to the AAAI 2027 Innovative Applications of AI conference for consideration. Arv closes with how to design experiments for learnings instead of launches and why self-serve experimentation is the next step beyond the analyst. For product, analytics and retail media teams who run experiments on shared pages.     Chapters00:00 Intro01:42 Balancing metrics across search, shopping agents and advertisers03:08 Guardrails, degradation thresholds and long-term holdouts05:58 The banana problem: rethinking a cross-sell idea08:20 Rebuilding around the customer's mission11:16 Designing experiments for learnings, not launches13:48 Trustworthy experiments and where experimentation goes next15:56 Wrap     Takeaways- Every page owner has their own primary metric; ads teams negotiate a degradation threshold per page and stay inside it- Revenue is not the only guardrail: relevance and latency are long-term metrics, and ad-free holdouts running for months measure what ads cost in trust- Customers engaged with complementary recommendations, but that engagement did not always translate into purchases; the fix was reducing diversity and building themes around the customer's mission- Start every experiment by asking how to make the product more usable, not how to launch it; the hypothesis changes and so do the metrics- A written process (test plan, no early peeking, end-of-test checks) catches bugs early and builds trust; self-serve tooling is what lets PMs experiment without an analyst     Connect with the GuestArv Ramchandani LinkedIn: https://www.linkedin.com/in/arv-ramchandaniCompany Website: https://www.walmartconnect.com     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
Ancestry on AI Storytelling, Paywalls, and Decision Quality
2026/04/14
SummaryWhat happens when A/B testing stops being a tool and becomes your operating system? Suresh Teckchandani, VP of Product & Technology at Ancestry (formerly PayPal and eBay), shares how the team scaled experimentation from isolated tests to capability-building that drives roadmap and revenue. He details the “growth metering” and paywall experiments that unlocked a 5.3% lift in key engagement and improved conversions—then became platform features. Suresh explains Ancestry’s centralized experimentation platform with self-serve access for PMs and engineers, why “obvious” UX changes can be the riskiest, and how removing friction actually hurt engagement by 20–25% due to user mental models. He also breaks down a major growth lever: AI-powered storytelling that turns raw records into narratives, delivering 30%+ CTR lift and a 5x increase in story views. You’ll learn how Ancestry balances input vs. output metrics, when not to test, and why the best leaders optimize for decision quality over win counts—with clean baselines, right audiences, adequate sample sizes, and true statistical significance. Timestamps[00:45] – Ancestry’s experimentation maturity: metering, paywalls, and the 5.3% lift that unlocked capabilities[03:48] – From isolated tests to a capability mindset: experimentation as an operating system[05:34] – Balancing wins with learning: zooming out for subscription engagement and NPS[08:49] – Operating model: centralized platform, self-serve dashboards, and baseline resets[11:59] – Counterintuitive UX lesson: removing friction backfired (–20–25% CTR); respect mental models[15:10] – AI storytelling as a growth lever: record comparisons into narratives, 30%+ CTR and 5x views[18:27] – Input vs. output metrics: when to roll back and how to link short- and long-term outcomes[30:00] – Parting advice: test what changes CX, avoid vanity testing, and optimize decision quality Takeaways- Build capabilities, not just tests—use experiments to unlock platform features (e.g., metering, paywalls).- Democratize experimentation with a centralized platform and self-serve tooling; reset baselines regularly.- Test “obvious” UX changes; preserve helpful friction and align with user mental models.- Turn data into narratives with AI to deepen engagement and increase discovery.- Define input and output metrics; ship only what improves core outcomes (retention, sign-ups), and roll back fast if not.- Optimize for decision quality: right audience, sufficient sample sizes, clean baselines, and true statistical significance. Connect with the GuestSuresh Teckchandani LinkedIn: https://www.linkedin.com/in/teckchandani/About the guest: https://www.growthbook.io/podcast/guests/suresh-teckchandaniEpisode page: https://www.growthbook.io/podcast/episode/1-8Company Website: https://www.ancestry.com SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/
Fyxer's engineering playbook to go from $1M to $35M ARR
2026/04/02
SummaryHow do you drive hypergrowth without guessing? Kameron Tanseli, Head of Growth Engineering at Fyxer—an AI assistant for your email—breaks down the experimentation playbook that helped the company scale from $1M to $35M ARR, with sights set on $100–$150M. Kameron explains how startups should think about A/B testing differently: de-risk big bets, not just button colors. He shares a risk-based approach to when to run rigorous tests vs. ship-and-measure, why a 25% win rate is a sign you’re testing ambitiously, and how PLG features should be shipped first, then rapidly iterated to drive usage. You’ll hear how Fyxer uses AI to speed the entire lifecycle—Claude, Cursor desktop cloud agents, GrowthBook, and BigQuery—plus how a Slack-first changelog and an internal “AI data scientist” democratize insights. Kameron also details turning everyday product usage into growth loops, personalizing signup paths, and measuring success by movement in global ARR, not just local metrics. He closes with candid advice for new growth engineers: expect to struggle early, be T-shaped, and adopt your customer’s language. Timestamps[00:34] – Startup A/B testing mindset: de-risking big bets with only a 25% win rate[02:45] – When to A/B test vs. ship: risk appetite, funnel stage, and non-inferiority tests[04:43] – 360 experiments with 4 people: scaling to 1,000 using AI and Cursor cloud agents[08:22] – Separating feature impact from momentum: PLG and trial model moves ARR[10:29] – Ship PLG features, then iterate to drive usage; measuring DAU and revenue impact[11:40] – Habit loops to growth loops: turning product features into PLG (scheduling case study)[16:47] – Building an experimentation culture: founder buy-in, Slack changelog, shared data[26:50] – The modern growth stack: Claude, Cursor, GrowthBook, BigQuery, and DOT in Slack Takeaways- Prioritize by risk: run rigorous A/B tests where you have volume; use before/after or non-inferiority for low-risk in-product changes.- Test big levers—not just UI: pricing models, usage limits, onboarding pathways—and judge success by ARR movement, not micro-metrics.- Ship first, then optimize: launch PLG features and immediately run experiments to increase adoption; track daily active usage per feature.- Build growth loops from habits: design shareable artifacts and personalized signup paths; drive users back to your domain to capture value.- Scale experimentation with AI: use Cursor desktop/cloud agents for parallel builds and visual QA; orchestrate docs/analysis via Claude; automate cleanups and reporting.- Make experimentation company-wide: centralize data (BigQuery), broadcast wins/losses in Slack via GrowthBook, and auto-correlate metric dips to releases. Connect with the GuestKameron Tanseli LinkedIn: https://www.linkedin.com/in/kameron-tanseli/About the guest: https://www.growthbook.io/podcast/guests/kameron-tanseliEpisode page: https://www.growthbook.io/podcast/episode/1-7Company Website: https://www.fyxer.com
The 5 pixels that cost LinkedIn a million dollars a month
2026/04/02
SummaryHow do you build a culture where nothing ships without evidence—and leaders actually act on the data? Makram Mansour, Head of Marketplace at ID.me and former experimentation leader at LinkedIn and Intuit, shares the systems, mindsets, and guardrails behind “experimenting everywhere.” At LinkedIn, he helped support 10,000+ annual experiments with 2,000 weekly platform users, and he explains the hard-earned lessons (like a 5px UI tweak causing a million-dollar ad loss) that led to a “test before release” mandate. At Intuit, he operationalized “fail forward,” partnering with HR to rewrite OKRs so teams are rewarded for learning, not just launching. Makram breaks down why to shift from MVP to MVT (minimum viable test), how to surface leap-of-faith assumptions with PRFAQs and “unit of one” prototypes, and where AI now unlocks faster, safer front-end testing. He also details critical guardrails—cost visibility for AI infrastructure, ethical and inclusion metrics, and the people-process-technology triad—plus practical ways to remove bottlenecks via a center of excellence. If you’re starting from scratch or scaling your program, you’ll learn how to personalize responsibly at the top of the funnel, define your North Star and signposts, and stack early wins while building influence across the org. Timestamps[00:45] – Makram’s path: running experimentation at LinkedIn and Intuit, and why nothing ships without an A/B test[02:15] – Costly lessons: 5px banner change, algorithm tweaks, and the case for rigorous guardrails[06:40] – Leadership discipline: killing features (voice meetups, LinkedIn Stories) and changing OKRs to reward learning[11:05] – People, process, technology: top-down and bottom-up tracks, and embedding “fail forward”[13:40] – From MVP to MVT: validating leap-of-faith assumptions, PRFAQ, and rapid “unit of one” prototypes[15:55] – Bottlenecks and unlocks: engineering/data science capacity, centers of excellence, and AI for fast front-end tests[22:45] – Personalization at the top of funnel: avoid waste, design reviews, and right-size testing before building[25:45] – Guardrail metrics that matter: AI infra costs, ethics/compliance, and fairness-by-design[29:45] – ID.me now: zero-to-one builds, vision-to-values, North Star and leading indicators[33:30] – How to start at a new org: crawl-walk-run, small wins, relationships, and over-communication Takeaways- Shift from MVP to MVT: list leap-of-faith assumptions and design minimum viable tests before you build.- Institutionalize learning: align OKRs with “fail forward,” and be willing to kill low-performing features quickly.- Build the triad: pair an easy-to-use platform with training, top-down sponsorship, and clear launch processes.- Add real guardrails: track AI infrastructure costs, ethics/compliance, and inclusion metrics alongside growth KPIs.- Unblock teams: create a center of excellence for data science and enable rapid variants with AI-powered tooling.- Start small and visible: rack up quick wins, over-communicate progress, and grow influence through relationships. Connect with the GuestMakram Mansour LinkedIn: https://www.linkedin.com/in/ilmansourAbout the guest: https://www.growthbook.io/podcast/guests/makram-mansourEpisode page: https://www.growthbook.io/podcast/episode/1-6Company Website: https://www.id.me
Truist on shipping faster and more safely with AI and human-in-the-loop banking
2026/04/02
How do you boost developer velocity in a highly regulated industry—without sacrificing safety or customer trust? Charles Williams, Senior Vice President and Software Engineering Director at Truist (formed from the BB&T and SunTrust merger), shares how his team elevates developer experience to ship faster and more reliably. Charles breaks down shifting quality “left” with automation, measuring success with both DORA metrics and developer sentiment, and why human-in-the-loop is non-negotiable for AI in finance. He details Truist’s governance model—steering committees, enterprise architecture, and clear guardrails—to avoid tool sprawl while building a purpose-built AI ecosystem: Microsoft Copilot for productivity, GitLab’s AI-enabled DevSecOps platform for engineering, and separate consumer-facing capabilities. Expect practical insights on starting with low-risk, high-yield use cases (unit tests, docs, security triage), tracking AI utilization, and upskilling teams in prompt engineering so developers can “manage” AI agents effectively. Charles also explores the path to personalized experiences balanced with privacy, why branches should be enhanced—not reduced—by AI, and the cultural skills leaders need now: empathy, neurodiversity awareness, and change management. He closes with where AI is driving ROI first—developer onboarding and pipeline productivity—with code quality gains following close behind. Timestamps[00:02] – Truist overview and Charles’s mandate: improving developer experience at scale[00:56] – AI as a strategic priority; shifting quality left with automation to remove bottlenecks[02:19] – Measuring success: DORA metrics plus sentiment—eliminating toil to drive happiness[04:33] – Human-in-the-loop AI for high-stakes finance; customer and internal use cases[07:25] – How Truist evaluates tools: personas, pain points, and starting with tests, docs, security[08:35] – The stack: Microsoft Copilot, GitLab’s AI gateway approach, and tracking utilization[10:36] – New skills and culture: prompt engineering, “managing” AI agents, and strong governance[20:45] – What’s next: personalization vs privacy, fintech agility + bank stability, and where AI pays off now Takeaways- Shift quality left with automated checks so developers catch issues early without human gatekeeping.- Measure DORA metrics and developer sentiment; remove mundane toil to increase speed and satisfaction.- Keep humans in the loop for AI-assisted coding and customer answers—trust but verify in regulated contexts.- Build an AI ecosystem with clear purposes (productivity, engineering, consumer) and a steering committee to avoid duplication.- Start with low-risk, high-yield AI use cases—unit tests, documentation, and security triage—to build confidence and momentum.- Upskill teams in prompt engineering and AI oversight so developers can effectively direct and review AI “agents.” Connect with the GuestAbout the guest: https://www.growthbook.io/podcast/guests/charles-williamsEpisode page: https://www.growthbook.io/podcast/episode/1-5Company Website: https://www.truist.com

Podcast reviews

Read The Experimentation Edge podcast reviews


5 out of 5
10 reviews
★★★★★
LightWave2 2026/09/17
Great podcast
Every guest shares at least one nugget that I learn from. Great show. The Disney episode is especially good.
★★★★★
Experimentation Enthusiast 2026/09/15
Great insights
It’s really interesting to hear how different companies are using testing and experimentation in practice.
check all reviews on apple podcasts

Podcast sponsorship advertising

Start advertising on The Experimentation Edge relevant audience podcasts


What do you want to promote?