Advertise on podcast: The Pragmatic Embedded Podcast

Most embedded teams run unit tests on the host and integration tests on real hardware (possibly manually), with very little in between. Marie Meyer and Joel Schulz-Andres founded OnMCU to fill that gap, and we talk with them about what belongs there. Joel's approach begins with "on-target unit tests" on bare dev boards, which catch clock misconfigurations and peripheral errata before your own board exists, and we look at what changes once it does. Fault injection is the kind of testing most teams never get to, because the rig for it is effectively a second embedded product built alongside the first. We talk about how much of that work you can avoid. Luca suggests injecting faults further up the software stack, wherever your architecture has a seam for it. Joel's rule is to move a test up the hierarchy wherever you can afford the latency. We also discuss the point at which cloud boards stop being enough and the test hardware has to sit next to your own boards. If you've ever built a test rig yourself, some of what went wrong for them will sound familiar. They bought USB hubs that claim per-port power switching and don't have it, rack drawers that waste half the space, and looked for a cable chain that doesn't exist. And quite apart from technology, there's the quintessentially German pushback OnMCU has to overcome when selling their cloud-based HILs: "If you don't want a dev board on your desk, you shouldn't have studied electrical engineering." Key Topics[00:00] Marie and Joel introduce themselves and OnMCU[01:58] An FFT that "needed an FPGA", and why you can't rent a dev board the way you rent a Mac[06:39] On-target unit tests: catching clock and peripheral bugs on bare dev boards[13:21] Moving from cloud boards to your own hardware, and tunnelling I2C/SPI to a remote sensor shield[16:19] Reusing the test rig for end-of-line testing in production[18:38] Why generic fault injection is hard: open-drain vs push-pull, 1.8/3.3/5 V, bus speeds[23:40] Cloud or on-prem: maintenance, cost, firmware confidentiality, network isolation via 5G[27:57] What they got wrong: cloud scepticism, USB hubs that lie about power switching, rack drawers and cable chains[35:55] The on-prem box and the interposer in detail[38:32] A gap between home-built HIL rigs and dSPACE-priced systems[41:05] Munich Embedded on 27 October: technical talks, no pitches[44:47] How they ran the first conference for under 10k with 80 peopleNotable Quotes"There's a blog post from 1999 about daily builds being essential - that was 27 years ago, and in embedded, some companies are not doing daily builds today. There's really this continuity of testing that's missing." — Joel Schulz-Andres "Whenever we had to build something from scratch, it felt great. Whenever we had to depend on others - we're standing on the shoulders of giants, but a lot of times you have to really work around it to get something working." — Joel Schulz-Andres "Testing in the cloud is really a mindset shift, especially in Germany. People told us: if you don't want to have a dev board on your table, you shouldn't study electrical engineering." — Marie Meyer Resources MentionedOnMCU - Cloud-based and on-premises testing platform for embedded systems with error injection capabilitiesMunich Embedded Conference - Technical conference in Munich on October 27, 2026, featuring talks on embedded engineering and networkingThe Joel Test - Joel Spolsky's influential 1999 blog post outlining 12 essential software development practicesGoertzel Algorithm - Frequency-selective FFT algorithm mentioned for efficient spectral analysis on resource-constrained MCUsAgile Embedded Podcast Slack - Community discussion channel where you can reach the hosts and connect with other listeners Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

We're back talking about Test-Driven Development, just 10 episodes after our last TDD discussion. This time, Jeff and Luca tackle the practical challenges that trip up developers trying to adopt TDD - especially that awkward moment staring at a blank screen wondering "what do I even type first?" Drawing from Luca's experience training embedded developers, we explore why TDD feels so unnatural at first and how to actually get started. We break down the process: from sketching test ideas as bullet points, to writing test outlines in plain English, to finally implementing actual test code. Along the way, we discuss how LLMs change the game - both helping overcome the fear of the empty screen and creating new traps when they generate overwhelming amounts of test code. Whether you're coding by hand or working with AI assistants, the key is maintaining control through deliberate, incremental steps and keeping your focus on module interfaces and test quality over implementation details. Key Topics[02:30] The common struggle: Why developers abandon TDD after trying it once[05:15] TDD is not a design methodology - it comes after decomposition[08:45] The mental leap: Calling functions that don't exist yet[12:00] Starting with test ideas as bullet points, not full test implementations[18:30] Two traps: Coding by hand (fear of empty screen) vs. using LLMs (overwhelming output)[25:00] Turning test statements into questions for effective coding spikes[32:15] Review test code more thoroughly than production code when using LLMs[36:00] Connecting TDD to higher-level specs and requirementsNotable Quotes"TDD is really just a Jedi mind trick of thinking about your tasks in a different way that makes it easier for you to come to good answers." — Luca Ingianni "TDD is not how you build the map. It's how you have confidence in a quality implementation and protection against future regressions." — Jeff Gable "I've gotten into the habit of reviewing my test code more thoroughly than the actual production code when using LLMs. Because the test code really tells me the shape of the production code." — Luca Ingianni Resources MentionedLuca Ingianni's website - Link site with information about Luca's trainings, coaching, and contact detailsBaseRef - Requirements, test, and documentation management tool for medical device startups founded by JeffEpisode 92 - Previous TDD episode discussing test-driven development in the age of AIJacob Beningo's work - Resources on writing simulators and making business logic data-driven rather than hardware-drivenAgile Embedded Podcast Slack - Community discussion channel where you can reach the hosts and connect with other listeners Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

After five years and 101 episodes, we're doing something we never thought we'd do: renaming the podcast. The Agile Embedded Podcast is now the Pragmatic Embedded Podcast: having the title reflect where we've been heading for quite a while. Jeff and Luca discuss why the word "agile" has become cringe-worthy, how the podcast has naturally evolved beyond its original scope, and what this means for future episodes. We talk about the reality of AI adoption in embedded development (spoiler: it's not as widespread as you might think), the importance of engineering fundamentals in an AI-assisted world, and why we're excited to explore topics without feeling constrained by our old name. This is still the same podcast you know, just with a name that finally fits what we actually do: exploring pragmatic approaches to building embedded systems. Key Topics[00:00] The big reveal: Agile Embedded becomes Pragmatic Embedded[02:30] Why "agile" has become cringe and why the podcast has outgrown its original name[04:15] How past episodes like the QP framework discussion already pointed beyond agile[06:00] The reality check: AI adoption in embedded development isn't as widespread as the hype suggests[09:45] Jeff's perspective: Every developer is becoming a lead developer managing AI-generated code[12:30] What's next: More freedom to explore what's actually useful for embedded developersNotable Quotes"I have literally been cringing internally for a long time saying Agile, Agile, Agile. And I'm just really looking forward to broadening the focus and not feeling bad about it." — Jeff "My customers never buy the curly brackets, do they? There's a lot of good old-fashioned engineering craft that we can talk about and that we'll continue to talk about." — Luca "Writing curly braces is going the way of being dead. But software engineering fundamentals, essentially, I see the trend... every developer is going to be essentially what is now a lead developer." — Jeff Resources MentionedEmbedded AI Podcast - Luca's other podcast with Ryan Torvik, focused on AI tools in embedded developmentLuca Ingianni's website - Training and consulting on AI and embedded systemsBaseRef - Jeff's requirements and test management tool for medical device startupsAgile Embedded Podcast Slack - Community discussion channel where you can reach the hosts and connect with other listeners Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

We flip the script for episode 100! Instead of hosting, Luca and Jeff become the guests as Joe Schneider and Jacob Beningo take over the interview chair. We look back at how two consultants from opposite sides of the world accidentally started a podcast, share war stories about the hardest bugs we've ever chased (spoiler: weeks of work for one-line fixes), and discuss why "Agile" might not be the buzzword it once was. The conversation touches on everything from helicopter avionics to the future of AI in embedded systems, plus some honest talk about what keeps us going after five and a half years and 100 episodes. Key Topics [02:30] Origin story: How a consulting group for solopreneurs brought together two embedded engineers from California and Munich [08:00] Favorite episodes and memorable guests: James Grenning, Miro Samek, Matthew Eshelman, and the value of reaching out to people you admire [15:00] Why "Agile Embedded"? Finding an unclaimed niche at the intersection of Agile practices and embedded systems [25:00] War stories: The hardest bugs to fix - from five-day hunts for one-line interrupt bugs to months tracking down temperature-dependent DDR RAM failures [35:00] The dynamic between hosts: Jeff keeps things concrete, Luca pulls back to the big picture - and why that balance matters [42:00] Looking ahead: AI as "the new Agile," requirements engineering's renewed importance, and navigating the hype cycle [50:00] The Gartner hype cycle and AI: Are we heading for the trough of disillusionment, or is this an exponential that never stops? Notable Quotes "No one gives you permission. You're listening to us because we started talking, and maybe some of you found it interesting. But you can put yourself out there, and you learn by speaking." — Jeff Gable "Testing is like vacuuming. It's only fun if it rattles in the hose. It's just no fun to test a system that has no bugs." — Luca Ingianni "AI really is the new Agile - the magical fairy dust that you can sprinkle over your team. It'll take a while for people to realize that this is also much harder than it looks." — Luca Ingianni Resources Mentioned Embedded Vibe - Joe Schneider's benchmarking site testing LLMs' ability to generate embedded code in one shot Beningo Embedded Group - Jacob Beningo's consulting and training services for embedded systems development Dojo Five - Joe Schneider's embedded systems consulting and development company Embedded Systems Summit - In-person conference organized by Jacob Beningo and Stefan, planned for mid-November in the Bay Area Agile Embedded Podcast Slack - Community discussion channel where you can reach the hosts and connect with other listeners Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

In the third installment of our requirements engineering series, Luca and Jeff shift focus from requirements themselves to the bigger picture: how do you actually turn ideas into working products? We explore the journey from product vision through roadmaps to backlogs, discussing what makes a good roadmap and—perhaps more importantly—the surprisingly common mistakes that even experienced teams make. We dig into practical challenges like the "progress by PowerPoint" trap (putting milestones like "specification complete" on your roadmap instead of actual value delivery), the dangers of over-committing to dates before you understand the problem space, and why most organizations struggle with prioritization. Luca shares insights from his agile requirements engineering training, including a clever trap he sets for participants around roadmap design. We also tackle the iron triangle of project management and why varying resources is often the least effective lever to pull when a project is running late. Whether you're a product manager, engineering lead, or startup founder, this episode offers a grounded perspective on planning that respects both ambition and reality. Key Topics [02:15] The hierarchy of planning artifacts: from product vision to roadmaps to backlogs [03:30] What makes a good product vision (Kennedy's moon landing speech as the gold standard) [06:45] Roadmaps as alignment and discovery tools, not static commitments [11:20] The prototyping phase and managing uncertainty in early-stage planning [15:40] Common mistake #1: Putting the wrong things on roadmaps ("specification complete" vs. actual value milestones) [21:30] The danger of over-committing to dates before understanding the problem (especially in VC-funded startups) [26:15] Tailoring roadmaps for different audiences (internal, customer-facing, investor-facing) [30:45] The failure to prioritize and its impact on roadmap execution [35:20] Finding the right level of granularity: the iceberg principle for roadmaps [40:10] The iron triangle of project management and why varying resources is the least effective lever [43:30] Who should own the roadmap (product management's role in synthesis and collaboration) Notable Quotes "If somebody has visions they should go see a doctor—but still, you know, it makes sense to have such a thing. A product vision should be nice and compact, like Kennedy's moon landing speech: what we're doing, the timeline, and the value." — Luca Ingianni "I don't care about the stupid presentation. Show me a product increment. Show me captured value. Then we're talking. Anything else is just smoke and mirrors." — Luca Ingianni "Plans are useless but planning is indispensable. That process of getting all the key players in the same room and trying to hash out this schedule is where you will uncover the biggest uncertainties and risks." — Jeff Gable Resources Mentioned Agile Embedded Podcast Episode 31 - Interview with John Odo on product roadmaps (May 2022) Luca's Agile Requirements Engineering Training - Training course covering requirements, roadmaps, and product planning BaseRef - Jeff's new product for requirements and test management for medical device startups The Mythical Man-Month - Classic book on software project management discussing why adding people to late projects makes them later Agile Embedded Podcast Slack - Community discussion channel where you can reach the hosts and connect with other listeners Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

We talk with Pete Staples, founder of Blue Clover Devices, about the often-overlooked challenge of flashing firmware in production. Pete shares insights from running a contract manufacturing operation in Shenzhen and explains why the handoff from engineering to manufacturing is more like "hucking it over a fence" than a smooth relay race. We explore the gap between engineers' assumptions about factory capabilities and the dusty reality of production floors. Pete discusses security challenges, the complexity of modern microcontroller programming, and how Blue Clover's Production Line Tool addresses the middle ground between expensive custom automation and ad-hoc bench setups. We also touch on provisioning, calibration workflows, and why the engineer who designs the product must also define how it's tested. Key Topics [02:30] The reality of factory firmware flashing - dusty PCs, hot glue, and cables everywhere [06:15] Security challenges: managing sensitive firmware and the "glass room" solution [09:45] The gap between engineer assumptions and factory reality - no, they don't have better equipment than you [14:20] In-circuit testing and bed-of-nails fixtures explained [22:30] The Production Line Tool: standardizing hardware and software across engineering and factory [28:00] Recording what matters: firmware versions, hardware serial numbers, and test results per device [31:45] Provisioning and security: webhooks, cloud databases, and managing secrets in production [38:20] The Test Agent: a companion device for running third-party software and complex programming workflows [43:00] Who should write the test plan? Why engineers must define "good enough" before production Notable Quotes "Engineers assume that the factories are a lot more sophisticated than they really are. In reality, it's a lot more like just hucking it over a fence and just hoping there's somebody there waiting." — Pete Staples "They show you their pick-and-place machine and 10-zone reflow oven, and you're like, 'wow, these guys are tipped off.' And then rarely do they say, 'oh, and here's where we do firmware flashing.' It's normally another floor of the building, dimly lit, dusty old PCs." — Pete Staples "The engineer responsible for the product has to not only engineer the product, but how it's tested. They can't just say, 'here's a bunch of design files, build it and let's see what happens.'" — Pete Staples Resources Mentioned Blue Clover Devices - Pete's company specializing in factory firmware flashing solutions Embedded World (Nuremberg) - Annual trade show in March where Blue Clover exhibits Embedded World North America (Anaheim) - North American version of Embedded World, September 22nd Kinetic (San Francisco) - Hardware-focused event put on by Hardware FYI Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

Requirements Engineering Part 2: A Practical Process for Safety-Critical Development In this second part of our requirements engineering series, Jeff walks us through his preferred process for developing safety-critical products, particularly medical devices. We explore the crucial distinction between prototyping and design-controlled development, discussing when to start formal requirements work and how to keep your first version minimal yet complete. Jeff emphasizes the importance of deeply fleshing out requirements before implementation—including error handling, which often comprises 70% of a product. We discuss tracer bullets as a development strategy, the value of writing test cases alongside requirements, and why tracking requirements completion gives you honest project status. Luca and Jeff also debate the finer points of MVPs versus prototypes, and Jeff announces his upcoming requirements management tool for medical device startups. Key Topics [00:00] Introduction and listener feedback on Part 1 [02:30] The prototyping phase: answering 'can we build it?' and 'should we build it?' before design controls [06:00] Luca vs. Jeff: The great MVP and prototype debate [12:00] Starting the V-model: minimal but deeply fleshed-out requirements for V1 [18:00] Error handling is 70% of your product—don't skip it in requirements [22:00] When to write test cases: early, alongside requirements [26:00] War story: the SATCOM system that needed a satellite slot in five years [32:00] Project management: tracking requirements completion for honest status updates [38:00] Tracer bullets: vertical slices through all layers, not horizontal completion [45:00] Jeff's upcoming requirements and test management tool for medical device startups Notable Quotes "Error handling is 70% of your product, if not more. If you only do the 30% of the requirements for the happy path, you're fooling yourself." — Jeff "If you don't get this right at the outset, it will haunt you through the entirety of your product development process." — Luca "Paper is the best place to figure it out. Actually think through the requirements rigorously before you start building." — Jeff Resources Mentioned Agile Embedded Slack Channel - Community discussion space for embedded development topics Matt Pocock's YouTube Channel - AI for serious engineers, discusses tracer bullets and AI-assisted development The Pragmatic Programmer - Classic software development book, source of the tracer bullet concept The Art of Unix Programming by Eric S. Raymond - Referenced for the chapter 'Don't just do something. Stand there.' Embedded.fm Episode 440: Condemned to Being Perfect - Crossover episode with Elecia White and Christopher White, where bootloaders and update strategies came up Jeff's Requirements Management Tool - Upcoming requirements, risk, and test management tool for medical device startups Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

Fuzzing and Dynamic Analysis for High-Integrity Software with Paul Butcher We sit down with Paul Butcher, Unit Director of Dynamic Analysis at AdaCore, to explore verification techniques beyond basic compliance in safety-critical software. Paul shares his experience from Eurofighter to automated trains, explaining how dynamic analysis—from unit testing to coverage analysis to fuzzing—helps find bugs that traditional testing misses. The conversation dives deep into fuzzing: how it works, why it's so effective at finding corner-case bugs (even in well-tested systems), and the challenges of applying it to embedded systems with timing constraints. Paul introduces an intriguing approach that combines static analysis with targeted fuzzing to automatically triage false positives and generate reproducers. We also touch on formal verification, the role of LLMs in verification workflows, and why the simplest software is often the safest. Whether you're working in aerospace, medical devices, or any safety-critical domain, this episode offers practical insights into building more robust systems. Key Topics [02:30] Paul's background in high-integrity embedded systems: Eurofighter, rail, drones, and AdaCore's dynamic analysis tools [05:00] Dynamic vs. static analysis: executing code to observe real behavior across different environments [08:15] How fuzzing works: mutation engines, anomaly detection, and finding bugs through negative testing [14:20] Challenges of fuzzing timing and concurrency bugs in embedded systems [17:45] Real-world success: fuzzing the NH90 avionics via the MIL bus uncovered numerous bugs [22:30] Safety standards (DO-178C, SIL levels) and objective-based approaches vs. checkbox compliance [28:00] Determining 'enough' fuzzing: coverage, input space complexity, and building certification arguments [32:15] Combining static analysis with targeted fuzzing to automatically triage false positives and generate reproducers [38:45] Symbolic execution and theorem provers: breaking through complex branch conditions in fuzzing campaigns [42:00] Shift-left philosophy: building verifiable software from the start with testing and analysis tools [47:30] Formal verification in practice: London Underground's Victoria line uses SPARK-proven emergency braking [51:00] LLMs in verification: cautious adoption for report analysis, but determinism remains critical for core tools [54:30] High Integrity Software Conference (HISC) in Birmingham, October 2026 Notable Quotes "Software testing is typically about, is it functionally correct? Fuzzing is like a negative testing technique. It's the inverse of that. It fires random inputs into your system with the intent of finding anomalies." — Paul Butcher "Every time I speak to someone who's tried fuzzing, even if it's a system that's considered high integrity with a high level of assurance, they always find something. It's really good at eking out those weird corner case scenarios." — Paul Butcher "With testing you would like to prove the absence of bugs, but unfortunately you can't. So you have to settle for a very distant second place of proving the presence of bugs." — Luca Ingianni Resources Mentioned Paul's paper on fuzzing in safety-critical contexts - Detailed discussion of how to argue 'enough' fuzzing for certification High Integrity Software Conference (HISC) - Annual conference in Birmingham, UK (October 2026) covering high-integrity software across industries AdaCore Dynamic Analysis Tools - Coverage, fuzzing, and unit testing solutions for high-integrity software SPARK formal verification - Formal proof technology used in London Underground's Victoria line emergency braking AFL++ - Successor to the discontinued AFL (American Fuzzy Lop): Fuzzing technology mentioned as capable of quickly finding the Heartbleed bug Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

Linux Profiling with Mohammed Billoo We sit down with Mohammed Billoo, founder of Mab Labs and author of the Embedded Linux Essentials Handbook, to explore the world of embedded Linux profiling and optimization. Mohammed shares hard-won lessons from the field, including debugging a scientific instrument that mysteriously crashed after 60-minute runs and optimizing a sophisticated MANET platform that took a 20% throughput hit. The conversation reveals a fundamental truth: in embedded Linux, the CPU is rarely the bottleneck. Mohammed walks us through his systematic approach to performance problems, starting with simple tools like HTOP before diving into specialized instrumentation. We discuss the critical difference between VM size and VM RSS for memory analysis, why dumping console output can kill boot times, and how to leverage kernel configurations for maximum diagnostic bang-for-buck. Mohammed emphasizes the importance of building instrumentation into systems from day one—not for premature optimization, but to give your future self the data needed when problems inevitably surface. The discussion also touches on how LLMs can accelerate the learning curve for complex tools like Valgrind and perf, while stressing that physical reality remains the ultimate arbiter of system performance. Key Topics [03:15] The surface area problem: why embedded Linux profiling requires a tool chest, not just a toolbox [06:30] Case study: debugging a scientific instrument that crashed after 60-minute runs [08:45] VM size vs. VM RSS: understanding the critical difference in memory analysis [14:20] Why the CPU is rarely the bottleneck: coprocessors, DMA, and crypto engines [18:50] Essential kernel configurations: function tracer, perf, and config kallsyms [24:10] File system bottlenecks: moving from CSV files to SQLite for data integrity [28:40] Boot time optimization: why console output is one of the biggest time sinks [32:15] Premature optimization vs. smart instrumentation: building in diagnostic capability from day one [38:25] Leveraging LLMs for visualization and analysis of perf data and Valgrind output [43:50] The first five commands: starting with HTOP and working down to specialized tools Notable Quotes "When you first get started, you have generally this arrogance that like, oh, it works fine. I've tested it. It's good to go. But then as you get more experience, as you become a more senior-level engineer, that arrogance, you start to kind of strip away a lot of that arrogance. You get humbled pretty quickly." — Mohammed Billoo "The CPU is very rarely the bottleneck because it's meant to, and the drivers are implemented in Linux in such a way that they're intelligent enough that they can hand off a lot of the things of the CPU to coprocessors so that the CPU is really idle." — Mohammed Billoo "I don't convince myself of a claim that I'm making until I have data to back it up. So I don't say, oh, you know, this is working fine. Like, well, again, what does fine mean? Or, you know, what does well mean? And what is the data to prove that?" — Mohammed Billoo Resources Mentioned HTOP - Interactive process viewer for Linux - Mohammed's first tool for getting a high-level view of system performance perf - Linux profiling tool with performance counters - requires kernel configuration to enable LTTng - Linux Trace Toolkit Next Generation - provides visibility across both user space and kernel space Valgrind - Memory debugging and profiling tool for detecting memory leaks iperf - Network throughput measurement tool with server and client components GStreamer - Multimedia framework with built-in tools for per-frame timestamp analysis Tracealyzer - Visualization tool for LTTng and other performance data SQLite - Embedded database recommended for data integrity over CSV files in embedded systems Embedded Linux Essentials Handbook - Mohammed Billoo's book published by Packt Mab Labs - Mohammed Billoo's embedded solutions consultancy with blog on embedded Linux topics Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

Requirements Engineering Fundamentals - Part 1 We kick off a multi-part series on requirements engineering by exploring what requirements actually are and why they matter - even for Agilists. Jeff shares his medical device expertise while Luca brings his automotive and aerospace background to discuss the different levels of requirements (from high-level user needs to testable system requirements), the importance of traceability, and why proper tooling beats Word and Excel every time. We dig into practical aspects like the EARS format for writing requirements, the crucial distinction between requirements and design choices, and why glossaries aren't as boring as they sound. Along the way, we tackle the tension between regulatory compliance and actual engineering value, emphasizing that documentation should be an artifact of diligent work - not the work itself. Whether you're in safety-critical industries or just want to build better products, understanding requirements engineering helps manage complexity and prevent costly mistakes. Key Topics [02:30] What is requirements engineering and why it matters beyond safety-critical industries [06:45] Don't Agilists hate requirements? Debunking the myth and discussing iteration vs. waterfall [11:20] The hierarchy of requirements: user needs, system requirements, and subsystem requirements [18:00] Requirements vs. design choices: where to draw the line and why it matters for testing [24:15] Writing good requirements: EARS format, must vs. shall vs. may, and the value of glossaries [32:40] Traceability: linking requirements across levels and to test cases [40:30] Why Word and Excel don't cut it: the case for proper requirements management tools [48:20] Risk analysis and mitigation in safety-critical development [52:00] Documentation as artifact of diligent work, not the work itself Notable Quotes "The whole agile movement was a reaction to the one time through the requirements specification build test loop that took several years. By the time you got to the end, the requirements no longer applied." — Jeff "Do not use an LLM to manage requirements. Do use the LLM to write tools that help you manage requirements." — Luca "I view any medical device that I work on as if it's going to be used on my child. What do I need to do to convince myself that it is safe and effective? Once I have done that, if there are remaining boxes to check to get it through FDA, I will check those boxes." — Jeff Resources Mentioned EARS (Easy Approach to Requirements Syntax) - A grammar format for writing clear, verifiable requirements that constrains how requirements are written to reduce ambiguity FDA Guidance on Agile Development - Regulatory guidance describing how to do Agile development in medical device context ISO 26262 - Automotive safety standard mentioned as having similar traceability requirements to medical devices DO-178B - Aerospace software safety standard with similar requirements engineering principles Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

We talk with Tobias Kästner, a physicist-turned-software-architect and technical consultant at Inovex, about his journey from painfully slow hardware-software integration cycles to achieving three-week hardware sprints. Tobias shares hard-won lessons from medical device development, where fuzzy requirements and constant feedback from life scientists forced his team to rethink traditional approaches. The conversation centers on practical techniques: breaking monolithic PCB designs into modular "feature boards" connected via shields (think Arduino-style), using Git for hardware version control with SHA-1s printed on silkscreens, and leveraging tools like Zephyr RTOS to enable plug-and-play firmware that matches the modularity of the hardware. Tobias explains how relaxing constraints like board size and using automation to merge schematics allowed his team to iterate rapidly while maintaining a clear path to final form-factor designs. We discuss how this approach scaled to projects with 120+ people across multiple teams, and why the interplay between system architecture, organizational structure, and information flow matters more than most realize. Key Topics [02:30] The painful reality of traditional hardware development: six-month wait for hardware, nine months of debugging [08:00] Breaking apart monolithic PCB designs into modular feature boards with shield connectors [12:45] Relaxing constraints: larger board areas, autorouting, and prioritizing testability over final form factor [18:20] Version control for hardware: putting schematics in Git and printing SHA-1s on silkscreens [22:00] Using automation to merge feature board schematics into final form-factor designs [26:15] Firmware architecture: NuttX, Zephyr, KConfig, and device trees for modular, plug-and-play software [35:40] Scaling agile hardware-software co-development to 120+ person projects across multiple teams [39:00] The interplay of system architecture, organizational architecture, and information architecture Notable Quotes "When the board arrived, not a single line of code had been written for it because no one had been able to touch it. It took us nine additional months to debug all the things out of it." — Tobias Kästner "I've never seen any board working the first time. I've never seen any prototype without thin wires patching things out, but that's maybe a different story." — Tobias Kästner "We cannot think these architectures as independent of one another. If we have limitations in two of these architectures, we will see these limitations in the third architecture as well." — Tobias Kästner Resources Mentioned Inovex - Tobias's company offering engineering consulting services, trainings, and expertise in embedded systems, IoT, and full-stack development Zephyr RTOS - Open-source real-time operating system with KConfig, device tree support, and extensive driver library that Tobias recommends for modular firmware development NuttX RTOS - Apache Foundation RTOS with clean device driver model and KConfig support that Tobias used in earlier projects KiCad - Open-source PCB design software with emerging Python API support for schematic automation Services and Contact Through Inovex, Tobias provides trainings for both Zephyr and Yocto Linux, as well as consultancy and engineering support for embedded projects -- from 1-2 day workshops evaluating architectural state and cost/benefit analysis, to first prototypes, to full-fledged software development. With partners such as alpha-board (Berlin) and Blunk electronic (Erfurt), they also offer agile hardware services and help teams get started with the methods discussed in this episode. Email: [email protected] Tobias Kästner on LinkedIn tobiaskaestner on the Zephyr Discord Channel Links Companies: Inovex -- Embedded Systems Blunk electronic alpha-board Navimatix Talks and publications: Modular and Agile HW Development (2018 talk) Leveraging Zephyr's HW Abstraction for Agile Systems Engineering (2023 talk) Whitepaper: Agile in der Hardware -- by Gregor Gross, Christoph Schmiedinger, and Tobias Kästner Leveraging Zephyr for Functional Architecture Decomposition (2025 talk) Books recommended by Tobias: Small Groups as Complex Systems -- Holly Arrow et al., SAGE Publications The Dao of Complexity -- Jean Boulton, DeGruyter Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

We explore how test-driven development (TDD) remains essential—perhaps more than ever—when working with AI coding tools. Luca shares his evolved workflow using Claude Code, breaking down how he structures tests in three phases: test ideas, test outlines, and test implementations. We discuss why TDD provides the necessary control and confidence when AI generates code, how it prevents technical debt accumulation, and why tests serve as precise specifications for AI rather than afterthoughts. The conversation covers practical challenges like AI's tendency toward "success theater" (overly generous assertions), the importance of maintaining tight control over code quality, and why the bottleneck in AI-assisted development isn't code generation—it's expressing clear intent. We also touch on code spikes, large-scale refactorings, and why treating AI development as pair programming keeps you in the driver's seat. If you're wondering whether TDD still matters when AI writes your code, this episode makes a compelling case that it matters more than ever. Key Topics [02:30] Why TDD still matters with AI: confidence and control over generated code [06:45] Tests as specifications: describing desired behavior to AI rather than writing prompts [09:20] The three-phase test workflow: test ideas, test outlines, and implementations [15:30] Pair programming with AI: staying at the conceptual level while AI handles implementation [20:15] Code spikes and exploration: using AI to answer questions before writing production tests [24:40] AI failure modes: over-mocking and "success theater" with weak assertions [28:50] Large-scale refactorings: how AI excels at updating hundreds of tests simultaneously [32:10] The real bottleneck: expressing intent and specifications, not code generation speed Notable Quotes "As far as I am concerned, test-driven development is just about writing prompts for the AI that it can then use to build what you want it to build." — Luca "If you expect that a five-line prompt resulting in 10,000 lines of code will not result in 9,995 lines of uncertainty, you're just deluding yourself." — Luca "You can be five times faster than you were before and still maintain a very high production level quality code, but you probably can't be a hundred times faster." — Jeff Resources Mentioned Claude Code - Terminal-based AI coding assistant that Luca uses for TDD workflows, keeping conceptual work separate from code-level work Embedded AI Podcast - Luca's separate podcast focusing on AI in embedded systems, co-hosted with Ryan Torvik Luca's AI Training Courses - Hands-on trainings for using AI in embedded systems development (and much more!) links to all of Luca's work - Training, consulting, podcasts, conference talks and everything else Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

In this second part of our series on engineering organizations, Jeff and Luca explore how companies that build products should focus their efforts differently depending on their stage and scope. We start with startups and early-stage companies desperately searching for product-market fit, where the brutal truth is: quality doesn't matter yet. Your MVP should embarrass you—if it doesn't, you waited too long. We discuss the critical mental shift from throwaway prototypes to proper engineering once validation arrives, and why technical founders often fail by solving the wrong problem brilliantly. Moving up the ladder, we examine narrow-focus companies that have found their niche—like the German firm that does nothing but maintain a 100-year-old anchor chain machine, or specialists in medium-power electrical switches. These companies win through efficiency and deep expertise, but face existential risk if the market shifts. Finally, we tackle wide-focus companies introducing multiple product lines, where the challenge becomes running internal startups while managing established products, each requiring radically different approaches. The key insight: your focus must match your product's lifecycle stage, whether that's ruthless speed, cost optimization, or high-level process learning. Key Topics [02:30] Startups and early-stage companies: the existential search for product-market fit [06:45] The MVP philosophy: if you're not embarrassed, you waited too long [11:20] Quality vs. speed vs. scope: why quality doesn't matter in early stages [15:40] The Potemkin village approach: building facades to validate demand [19:15] Embedded products and MVPs: when physical products need creative shortcuts [23:50] The critical switch: from prototypes to proper engineering after validation [28:30] Narrow-focus companies: German hidden champions and deep specialization [34:10] Wide-focus companies: running internal startups within established organizations [40:25] Product teams and parallel focuses: managing different lifecycle stages simultaneously [45:00] Large established companies: high-level process learning and avoiding organizational weight Notable Quotes "If you read the Lean Startup, they will explicitly say: if you weren't embarrassed by your MVP, you waited too long. It really has to be painfully flimsy because you cannot afford to do it well." — Luca "Quality doesn't even factor because you're very explicitly building mock-ups from chewing gum and paper mache. They are fully intended to be thrown away." — Luca "Getting that product-market fit is existential. You will die if you do not get it and get it relatively quickly." — Jeff Resources Mentioned The Lean Startup - Eric Ries' book discussing MVP philosophy and the importance of being embarrassed by your first product The Mom Test - Rob Fitzpatrick's book about getting real customer feedback and validation through financial commitment The Art of Innovation - Tom Kelley's book on IDEO's design process, including the clothespin switch story Luca's Website - Trainings on embedded agile, AI in embedded systems, and more Jeff's Website - Consulting services for medical device software development Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

In this first part of a two-part series, Jeff and Luca explore how different types of service-oriented engineering organizations should focus their learning and improvement efforts. Drawing from their consulting experience, they examine three distinct categories: product development firms that turn client ideas into reality, engineering development firms that sell specialized technical expertise, and solo engineers who package all necessary knowledge into one person. The core insight: what you should focus on learning depends entirely on what you're actually selling. Product development firms need to master the entire client journey and product design process, not just engineering excellence. Engineering development firms must become technical wizards in a specific domain that clients actually value. Solo engineers face the challenge of needing deep expertise while wearing every business hat. Across all three types, the common traps are the same: focusing too much on craft and too little on client experience, failing to specialize, and not investing enough in teaching as marketing. Throughout the discussion, Jeff and Luca emphasize that for service firms, you are the product - and that changes everything about where you should direct your improvement efforts. The conversation is grounded in real experiences, including some cautionary tales about firms that tried to be everything to everyone. Key Topics[00:00] Introduction: Two-part series on engineering organizations and their different focuses[02:30] Overview of the framework: Service firms vs. product-building companies[05:15] Product development firms: Why engineering excellence isn't enough[08:45] The critical importance of product design and client guidance over pure engineering[12:20] Process-level learning: Shortening cycle times and enabling rapid prototyping[15:40] The Irinos example: In-house board manufacturing to tighten feedback loops[18:30] Requirements will always change - designing for learning, not perfection[21:00] The danger of being a generalist: Why specialization matters for service firms[24:15] Engineering development firms: Selling technical expertise, not complete products[27:45] Technology-focused learning: Going deep on specific technical capabilities[30:20] The trap of becoming a commodity: Why domain expertise beats technology alone[33:40] The forklift invoice review example: You can't specialize too narrowly[35:30] Solo engineers: The complete package vs. temporary employee trap[39:00] Common failures across all service firms: Too much craft focus, too little client experience and marketing[41:30] Teaching as the best form of marketing for technical service firmsNotable Quotes"The customers don't actually hire them for their engineering skills. They are sort of a given. But what such a product development firm should offer the client is guiding them through the development process, which they don't have enough skills for to do it on their own." — Luca "Engineering is not the point. The unit of work is delivering a working product to the client that satisfies their business case, that has a reasonable cost to manufacture, and that you feel confident your own client has validated their market." — Jeff "It's not that engineering is irrelevant, but rather that it's table stakes. This is just taken for granted, but what such a product development firm should offer is guiding them through the development process." — Luca "You almost can't be narrow enough. I remember our friend Philip Morgan having this example of a company that specializes in reviewing invoices of forklift repairs. This is what they do. They review forklift repair invoices. And they're doing very well apparently." — Luca "Teaching and giving information and solving problems publicly is the best form of marketing. It's not advertising. It's building trust with an audience." — Jeff Resources MentionedIDEO - Prototypical design firm mentioned as an example of companies specializing in product designIRNAS - Product development firm with in-house board manufacturing capabilities, featured in previous episodes, exemplifying tight feedback loopsPhilip Morgan - Consultant and friend mentioned for his example about specialization (forklift invoice review company)Jeff Gable's website - Jeff's consulting services for medical device software development and advisoryLuca Ingianni's website - Luca's training products and resources for embedded systems, IoT, and AIConnect With UsStay tuned for Part 2, where we'll explore organizations that build products and what they should focus on when the market decidesIf you're in the medical device industry and need help with embedded software - either writing it or navigating the regulatory landscape - reach out to Jeff at jeffgable.comCheck out Luca's training products for embedded systems, IoT, and AI at luca.engineerReflect on your own organization: Are you focusing on the right things for the type of service firm you are? Are you specializing enough? Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

In this episode, we sit down with Milica Kostic, an embedded software architect from Belgrade, Serbia, to discuss her journey from C/C++ to Rust and what it means for embedded development. Milica shares her experience adopting Rust in production environments, starting with an embedded Linux project using a microservice architecture that allowed for clean isolation of Rust code. We explore the practical realities of learning Rust as an experienced C/C++ developer - yes, there's a learning curve, and yes, the compiler will slap you on the wrist frequently. But Milica explains how the development experience, with cargo as a package manager and built-in tooling for testing and static analysis, makes the journey worthwhile. She's candid about where Rust shines (embedded Linux, greenfield projects) and where challenges remain (microcontroller support, IDE tooling, vendor backing). The conversation touches on the bigger question facing our industry: with memory safety becoming critical in our connected world, what role should Rust play in new embedded projects? While Milica takes a measured stance - acknowledging that C and C++ aren't going anywhere - she's clearly excited about Rust's potential, especially in safety-critical domains like medical devices. Whether you're Rust-curious or still skeptical, this episode offers a grounded perspective from someone who's actually shipped production code in Rust. Key Topics[02:30] Milica's background in embedded systems and her journey from electrical engineering to embedded software development, with focus on safety-critical industries like medical devices[04:15] The path to adopting Rust: from first hearing about it in 2020 to finding a client project willing to embrace it, and the importance of having experienced Rust developers on the team[07:00] Choosing the right project for Rust adoption: embedded Linux with microservice architecture as an ideal starting point, avoiding complex C/C++ interoperability[10:45] The learning curve: getting used to the Rust compiler's strictness, discovering the ecosystem of unofficial but widely-used crates, and how learning Rust improved C++ skills[14:20] What makes Rust development pleasant: cargo as package manager, built-in testing and static analysis, cleaner code organization with modules, and writing unit tests alongside source code[17:30] Current limitations: lack of official vendor support for microcontrollers, community-driven development, potential gaps in certified stacks (like BLE), and IDE support challenges[20:15] Interfacing Rust with C and C++: C binding works well, C++ has limitations with inheritance and templates, and the safety considerations when using unsafe code blocks[25:40] Integrating Rust into legacy projects: when it makes sense (isolated new features requiring memory safety) and when it doesn't (just for experimentation), plus maintenance considerations[30:00] The big question: Is it irresponsible not to use Rust for new projects? Discussion of Philip Marcraff's strong stance and Milica's more nuanced view considering team knowledge, existing tooling, and project context[33:45] The influence between languages: how C++ is learning from Rust's memory safety features, and why the borrow checker is harder to retrofit than basic safety improvements[36:20] Rust in operating systems: adoption in the Linux kernel and Microsoft Windows, and major tech companies pushing C++/Rust interoperability forward[39:00] The future of Rust in embedded: Milica's view that C, C++, and Rust will coexist, each with their own use cases, advantages, and trade-offsNotable Quotes"Learning Rust has also made me a better C++ developer as well. Once you get used to those rules, you apply them in C++ as well." — Milica "Just like writing Rust code is pleasant. It flows much nicer than or easier than it would with C++, for example. The way you organize your code, in my opinion, is also cleaner." — Milica "If you are developing Rust for embedded systems on microcontrollers, you need to be aware that there is no official vendor support. Everything currently is open source and driven by the community." — Milica "You definitely do not lose benefits of using Rust for the rest of your codebase when using a C library. That C library is isolated, and if there are some memory issues, then you know where to look." — Milica "I think most of the benefits come from starting with Rust in the first place. So having a clean slate, starting a new product, new project with Rust. That's where you see the most benefits." — Milica Resources MentionedEmbassy - An async framework used in embedded Rust projects, mentioned as a good starting point for greenfield embedded developmentZephyr RTOS - Real-time operating system that is working on official Rust integration, though not fully there yetRust Rover - JetBrains' official IDE for Rust development, released about a year and a half ago, though with some limitations for embedded developmentZed - A new IDE written completely in Rust, mentioned as an emerging option for Rust developmentSlint - A Rust-based GUI framework for embedded systemsEmbedded Online Conference - Conference where Milica gave a talk on Rust for embedded systems - link to her presentation in show notesMilica's LinkedInMilica's talk on Rust at the Embedded Online Conference Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com. You can find Luca at https://luca.engineer. Want to join the Pragmatic Embedded Slack? Click here Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/ Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/