1889249600
The Databricks Data Engineer

Advertise on podcast: The Databricks Data Engineer

Rating
★★★★★
5
from
4 reviews
Categories
This podcast has
34 episodes
Language
English
Publisher
Jakub Lasak
Explicit
No
Date created
2026/03/30
Latest episode
2026/11/16
Average duration
12 min.
Release period
8 days

Description

Helping 18k+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.

Unlock The Databricks Data Engineer podcast Email contact info,
Listeners & Audience details

Email contact information

Direct podcast contact details

Listeners

Audience numbers & engagement insights

Audience details

Podcast Insights

Podcast episodes

Check latest episodes from The Databricks Data Engineer podcast


Leave or stay? The 4 questions Databricks engineers keep asking (and the 10-minute test that answers them)
2026/11/16
It's late. A recruiter message sits unanswered, and the question you keep circling is short: I'm bored. Is bored a good enough reason to leave? It almost never travels alone. Four questions about leaving keep coming up from Databricks data engineers, and only the wording changes. Is boredom enough? Is it too late to stay for the migration? What is a good stack worth against twenty thousand more? Am I running toward something, or just away? They sound like questions about salary, stack and timing. They're one question, and from the inside its two answers feel identical. In this episode: - Why boredom says more about the next six months than the last two years, and where to look before you act on it - How joining a Databricks migration late becomes either a strong senior interview story or an answer you can't give in a final round - What a good stack is actually worth against a bigger offer, and why the worse platform might have the higher ceiling - The question to ask in every interview that shows how high a company lets anyone go - A ten-minute test built on the decisions your company made in the last six months This episode is for Databricks data engineers with a background thread about whether to move. Whether you're bored in a stable seat or weighing an offer on a different stack, you'll walk away able to name which situation you're actually in, and what to do about it this week. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
Medallion architecture vs. reality - when your Databricks bronze, silver and gold are just folders
2026/11/09
Three schemas. Bronze, silver, gold. The diagram got nods in the architecture review. Then two dashboards land in the same leadership meeting with two different active customer counts, and the head of finance asks which one is right. The trail ends at bronze, where rows that failed parsing were quietly dropped. The vendor keeps thirty days. Every label was there. What the medallion was invented to fix was not, and the pipeline had rebuilt the old mess across three schemas. In this episode: - What pipelines looked like before bronze, silver and gold, and the three pains the diagram was drawn to fix - Why dropping bad rows at bronze feels clean and quietly decides how far back you can rebuild - When skipping a layer is the right call, and why seven layers is not a safer bet than three - The question to ask about every gold table before your next architecture review This episode is for Databricks data engineers who build or inherit a medallion architecture and have wondered what silver is actually for. Whether you're designing a new lakehouse or maintaining one that looks textbook, you'll walk away with a way to check that each layer does a job no other layer does, and to defend it with more than "it's best practice." --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
How the always-sure Databricks engineer caps their own promotion (and never sees it)
2026/11/02
Mark's manager asks the room one question: if we move the biggest fact table onto Liquid Clustering, will it fix the slow dashboards? Mark answers first: yes, problem solved. Sarah says what she'd bet on, what she doesn't know yet, and when she'll know. Sarah claimed less. The decision goes to Sarah. Nobody tells Mark why. The instant-answer reflex that got him to mid-level is the same one that quietly keeps him there, and there's a name for what he's paying. In this episode: - Why a fast, confident answer is right early in a Databricks career, and when it stops paying - What the room needs from you when nobody knows for sure - The difference between being decisive and sounding certain - One sentence, two parts and a date, to carry into your next design review This episode is for Databricks data engineers who answer fast and still watch the risky migration and the design review invite go to someone else. You'll walk away with a way to make your "I'm sure" mean something again. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
Two Databricks jobs, one Delta table, same moment: 3 endings (and the 1 that needs a human)
2026/10/26
Two in the morning. A nightly merge is folding yesterday's orders into a Delta table. A backfill someone kicked off before logging off is correcting March. Neither job knows the other exists, and nothing in Delta would tell them. A lot of design around this runs on superstition. Either "it's Delta, it's ACID, relax." Or "never let two jobs touch that table." Both are guesses, and the tidy middle version, same data collides, different data is fine, falls apart the first time the error shows up. In this episode: - What Delta actually does when the second job tries to commit to a table that moved under it - Why two jobs writing completely different months can still collide, and what that error message is really telling you - Why scheduling jobs hours apart is a lock you built by hand, and the cheaper fix - When a retry quietly does damage instead of fixing things This episode is for Databricks data engineers who own a Delta table with more than one writer: a nightly merge, a backfill, a plain append. Whether you're designing a new pipeline or defending one in a design review, you'll walk away with a way to say which endings each writer can produce, instead of hoping. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
Unity Catalog Can't Manufacture Trust: Your Most-Read Databricks Table Might Be the Least Trusted
2026/10/19
The rollout is done. Every production table migrated to Unity Catalog, every one owned by a team group, and the revenue table carries a certified check mark. The platform lead posts in Slack: fully governed. Monday morning, finance replies in the thread. The revenue on the finance dashboard and the revenue in the sales deck are about three percent apart. Which one's real? The catalog has something to show for every field: an owner, a check mark, a health indicator that says healthy, a beautiful lineage graph. None of them answers that question. Unity Catalog might be the best thing your platform team ever shipped, and it still can't make one person believe a number. The missing piece isn't a feature. It's the habit of treating every trust complaint like one. In this episode: - Why an owner, a certified check mark and a healthy indicator can all look right while the number is still wrong - Why the most-read table in your catalog might be the least trusted one - What a metric view settles when two teams define revenue differently, and what it leaves to humans - The one question to ask before you touch the platform for any governance request This episode is for Databricks data engineers who work in Unity Catalog and keep getting governance requests another tag won't fix. Whether you just finished the migration or you're the one answering "which number is real?" in Slack, you'll walk away with a way to sort every request before anyone opens a ticket. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
Your Databricks director: what they fear, why they fund the duller proposal, what gets yours through
2026/10/12
Two proposals sit on a head of data's desk, both about the nightly pipeline that feeds the finance dashboard. One swaps the full reload for incremental loads with Auto Loader: it finishes in a fraction of the time and it's cheaper every month after the first. The other keeps the full reload, puts it on a bigger job cluster and starts it an hour earlier. It costs more. Forever. She asks one question, "What happens if this breaks on quarter close?", and approves the second one. It's tempting to call that ignorance or politics. It's neither. The better proposal was arguing merit to someone who gets asked a completely different question upstairs, in a meeting the engineer who wrote it will never be in. In this episode: - The fear that filters every proposal on your director's desk, and why it isn't fear of change - The question a tech lead, an engineering manager and a director each get asked, and why pitching a director like a tech lead loses - Three kinds of data leader, and how to tell which one you work for from the first question they ask about any proposal - Three things to add to any proposal with a real spread, so it gets funded without getting duller - A two-sentence test to run before anything goes upstairs This episode is for Databricks data engineers who have watched a better proposal lose to a duller one and couldn't explain why. Whether you're the mid-level engineer writing the pitch or the senior trying to get a platform change funded, you'll walk away able to read your leader's decisions and write proposals they can say yes to. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
Data skew is a checkout line - how to unjam Databricks jobs without bigger clusters
2026/10/05
Your nightly Databricks job ran long, so you open the Spark UI. Two hundred tasks. One hundred ninety-nine finished in about a minute. One is still running, and it's been running for forty. You double the cluster and rerun it. Same forty minutes. Nearly twice the bill. The cluster was never the problem. The rule that keeps your joins and group-bys correct is the same rule that lets one task carry the whole job, and no amount of extra workers moves that work off it. Sometimes the stuck task turns out to be worse than slow. In this episode: - Why more workers and more shuffle partitions both leave the stuck task exactly where it was - Where heavy keys come from, and the one source that is usually a bug in a performance costume - Why plenty of skewed aggregations never slow down, and where adaptive query execution on Databricks does and doesn't split the load - When salting pays, when a different approach beats it, and when the right call is to leave skew alone This episode is for Databricks data engineers whose reflex for a slow Spark stage is a bigger cluster. Whether you're mid-level and tired of paying for compute that changes nothing, or senior and about to be asked "how do you handle data skew?" in an interview, you'll walk away able to tell a skew problem from a capacity problem before you touch the cluster settings. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
Can You Reach Senior Databricks Engineer Without Ever Leading an Architecture Review?
2026/09/28
Sam is four years into a Databricks role, up for senior this cycle. Merged pull requests on the ingestion layer. The nightly job he rescued at three in the morning when the orders table stopped landing. Then his manager asks for one call where the answer wasn't obvious, he framed it, and he lived with the result. Sam starts three different sentences. None of them finish. The packet is full of work and empty of decisions. Sam thinks what he's missing is a chair at the architecture review table. It isn't. In this episode: - What the architecture review was ever supposed to prove, and why the room was never the proof - The one test to run on any call you've made, with no meeting, no invite and no template required - The two paragraphs that turn a design doc from a build plan into a decision somebody can audit nine months later - How to run the whole thing in a Slack thread on a four-person team with no review process For Databricks data engineers building a case for senior, especially where architecture reviews are rare or don't exist. You'll walk away able to tell work you did apart from a decision you can prove you owned. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
Why Databricks serverless exists, and which of your jobs should stay on classic clusters
2026/09/21
Open the workspace. Start a cluster. Get coffee. Come back, and sometimes it's ready. If you've been doing this a few years you can feel that wait in your shoulders, and it's the least interesting part of the story. Before any of it, you had to answer a question you could not possibly answer: how big should this be? Start small and you pay in latency. Start big and you pay for machines that are barely breathing for most of the run. Wait, or waste, and there was never a third option. In this episode: - Why the cluster wait was only the receipt, and what classic Databricks compute never actually solved - Why Databricks can keep compute warm when your own team never could, and what that shows up as in the rate - What "serverless" means here, jobs and notebooks, and why a serverless SQL warehouse is a different conversation - The thing a fixed cluster was quietly doing for your budget that serverless stops doing, and why the policy you're thinking of doesn't cover it - Three tests to run on your own workloads this week, plus the ones that can't move at all no matter what the math says This episode is for Databricks data engineers who have serverless sitting in a dropdown and no real argument either way. Whether you're defending a migration in an architecture review or explaining next month's bill, you'll walk away able to say which of your jobs should move, which should stay on classic, and why. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
How Databricks engineers lose the final round (and the 4 places it happens)
2026/09/14
You cleared the recruiter screen. You cleared the Databricks technical round, and cleared it well. Then the final round, then the email that says strong candidate, tough decision. You go back looking for the mistake and there isn't one. You were good. Somebody was just easier to picture. That sentence never makes it into the feedback. And it isn't a bad panel or a bias against you. It's a belief you carry into the room: that the final round is the last exam, the hardest one. If your last stage is still a coding or SQL round, that one is an exam, go and study. But most of these emails come out of a different loop, where the technical question was settled and the form closed well before you walked in. In this episode: - What that last round is actually deciding, and why it rarely shows up on a rubric - The four places a strong candidate loses it, ordered from the most common to the one that stays invisible longest - How an unnarrated trade-off reads as taste instead of judgment, and the one sentence that fixes it - The single behavior to take into your next final round, and why trying to fix all four is the wrong move This episode is for Databricks data engineers who keep reaching final rounds and keep getting the same email. Whether you're three no-offers into a year of interviewing or prepping for a loop that starts next week, you'll walk away with one concrete thing to change in the room, not another weekend of Spark internals. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
The real reason your Databricks week never got simpler - it isn't the lakehouse
2026/09/07
Your org finished the migration. Two systems became one. The nightly copy nobody wanted to own is gone, and so is the quarterly meeting where finance and the data science team each showed up with a real revenue number from tables in different buildings. That pain was real, and it is genuinely dead. So why does your week feel heavier than the deck promised? Because most of what got sold as simplification was a relocation. The copy didn't die, it turned into an argument about which schema in which catalog is the real one. The sync job didn't die, it became a maintenance calendar that pages nobody as an alert and pages you months later as the job that got a little slower every week. In this episode: - Why every architecture in this field gets sold on removal, and what the slide never names - How to tell the work that genuinely died in the lakehouse from the work that only changed address - Why a heavier week after a clean migration is not a verdict on you - The two questions to run on the next architecture pitched to your org, and why the second one is about names, not tasks This episode is for Databricks data engineers who got the win the lakehouse promised and still can't explain why the calendar filled back up. Whether you ran the migration or inherited it, you'll walk away with a lens you can run on any architecture sold as the thing that finally makes data simple. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
The Databricks incident where the most senior engineer never touched the keyboard
2026/08/31
Thursday, just after two in the afternoon. The first message doesn't come from an alert, it comes from a person in revenue operations: the biggest region jumped overnight and nobody sold anything new. Nobody answers. The engineer who owns the nightly Lakeflow job finds it in the run history. An overnight failure, a restart before anyone was awake, a window of orders written a second time under every executive dashboard in the company. So the fixing starts immediately. Head down, no message, because typing in the channel feels like stealing time from the fix. Which is how a second engineer joins a silent channel, starts the same correction, and lands the same window twice again. Effort was never the scarce thing in that channel. In this episode: - Why every outage is really two problems running in parallel, and only one of them has an owner - What a staff engineer did in ninety seconds without opening a notebook, and why the slower fix was the point - How to state an impact boundary a non engineer can repeat, including the bucket everyone skips - The one sentence that takes the most underpriced role in an incident, at any level This episode is for Databricks data engineers who have sat in a chaotic incident channel watching questions pile up faster than anyone can answer them. Whether you own the broken job or you're wondering if it's your place to speak, you'll know which seat in that channel is empty and how to take it. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
9 Databricks decisions that reveal your real seniority (and one that only looks senior)
2026/08/24
Three in the morning. The nightly job that loads the orders table died halfway. Half the tasks are green, and the rerun button is right there. What you do in the next ten seconds says more about your level than anything on your resume. Nobody places you by your title or your years. They place you by a pattern of routine calls made fast, in the dark, and everyone around you can read it except you. In this episode: - The ten-second question to ask before you rerun anything that writes - Why the fastest fix for a slow Databricks job is often the one that renews its cost every month - How the flaky job you quietly restart every week stops being a defect and starts being weather - What it means that most data engineers cannot price their own job, and what changes when they can - The one call that reads as the most senior contribution in the room and almost never is This episode is for Databricks data engineers who have the title, or want it, and want to know how hiring managers actually place them. Whether you are prepping for a senior interview or wondering why your level has not moved in two review cycles, you will walk away with a lens for your very next decision. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
The Databricks job market in 2026: there are two of them, and only one answers
2026/08/17
One engineer tells you hiring is back and recruiters are warm again. The next one is sixty applications deep and hearing nothing. Same month, same platform, sometimes the same city. That sounds like noise. It isn't. It's a split, and what sorts you onto one side of it has almost nothing to do with how good you are. In this episode: - Why forty applications in a week returns two dead screens, while a smaller and pickier search gets answered - The three things that changed in the last year and turned the generic Databricks data engineer posting into a stack of a hundred before lunch - Why fewer postings can be the better market for you, and the one condition that has to hold for it - The honest case against specializing, plus the test for which second surface is actually safe to bet on - The one resume sentence to rewrite this week, using work you already did and never wrote down This episode is for Databricks data engineers hearing two contradictory versions of the 2026 market and unsure which one is theirs. Whether you're mid-level and stuck in the application pile, senior and getting less traction than your years should buy, or early career and facing the most crowded entry market in a while, you'll walk away knowing which market your resume is sitting in and what it costs to move. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
Databricks broadcast joins: when the memo beats the meeting (and when it kills a task)
2026/08/10
Two engineers on the same team join the same big orders table to the same small lookup table. Same cluster, same data, one line of code different. Sarah's finishes in the time it takes to get a coffee. Mike's dies, and the error isn't about the data at all. A task ran out of memory building a hash map. Broadcast joins get passed around as a tip instead of a model: small table equals fast, flip the switch when a join runs long. But the mechanism in almost every write-up is years out of date, and the size Spark checks is not the size that has to fit in memory. In this episode: - How to explain broadcast versus shuffle joins at standup, in one sentence, without a whiteboard - What actually happens on a current Databricks runtime when a join broadcasts, and why the popular explanation stopped being true - Why the size estimate Spark trusts is not the size that lands, and where to read the real payload - What adaptive query execution rescues you from, and the two places its hands are tied - The question to ask before you add a broadcast hint, and the failure signature that tells you a broadcast is what broke This episode is for Databricks data engineers who write joins every week and treat the broadcast hint as a speed switch. Whether you're mid-level and tired of guessing why one join flies and an identical one falls over, or senior and about to be asked in an interview why Spark chose a sort merge join, you'll walk away able to predict the call before you run the query. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake1

Podcast reviews

Read The Databricks Data Engineer podcast reviews


5 out of 5
4 reviews

Podcast sponsorship advertising

Start advertising on The Databricks Data Engineer relevant audience podcasts


What do you want to promote?