Advertise on podcast: The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

In this episode, we explore the newly released Apache Airflow common AI provider — what problem it solves, how it was built and what's coming next. Kaxil Naik, Senior Director of Engineering at Astronomer and Apache Airflow PMC member, and Pavan Kumar Gopidesu, Lead Data Engineer at Experian and Apache Airflow PMC member, join us to walk through the provider's first release and the technical decisions behind it. Key Takeaways: 00:00 Introduction. 04:05 The common AI provider was born from a real production problem. 07:10 Airflow already had the primitives needed for durable agent execution, making it the natural foundation for AI orchestration.  09:15 The LLM schema compare operator uses Apache DataFusion to fetch source schemas. 11:07 Apache DataFusion was chosen for its speed. 13:09 Hook tool sets expose Airflow's provider hooks to agents with an allowed methods list that blocks destructive operations. 15:20 Passing durable=True to an LLM operator caches tool calls and LLM outputs mid-task.  18:13 The provider offers three abstraction levels.  21:20 The provider currently requires Airflow 3 — the team is open to adding Airflow 2.11 support if demand is high enough.  24:10 MCP server configs can be stored as Airflow connections. Resources Mentioned: Kaxil Naik https://www.linkedin.com/in/kaxil/ Pavan Kumar Gopidesu https://www.linkedin.com/in/pavan-kumar-gopidesu/ Astronomer | LinkedIn https://www.linkedin.com/company/astronomer/ Astronomer | Website https://www.astronomer.io Experian https://www.linkedin.com/company/experian/ Apache Airflow https://www.linkedin.com/company/apache-airflow Apache Airflow common AI provider docs https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/commits.html Apache DataFusion https://datafusion.apache.org/ Pydantic AI https://pydantic.dev/docs/ai/overview/ Airflow Slack https://airflow.apache.org/docs/apache-airflow-providers-slack/stable/index.html Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow https://airflow.apache.org/blog/common-ai-provider/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #Automation #Airflow #MachineLearning

Aviation data pipelines run on strict 28-day publication cycles, and the margin for error is zero. In this episode, we're joined by Samantha Blaney Cuevas, Software Engineer at Jeppesen ForeFlight, to explore how her team orchestrates a complex, time-sensitive data pipeline with Airflow and where AI is starting to fit into that picture. Key Takeaways: 00:00 Introduction. 04:05 Airflow orchestrates almost all business logic and data transformations across the cycle, with custom timetables built to track busy and slow periods programmatically. 06:10 Cycle-aware sensing tasks handle irregular source deliveries, including duplicates and early or late arrivals, without disrupting the pipeline. 08:07 The two main AI use cases are pipeline debugging and cycle awareness — both designed to reduce the manual overhead of monitoring a complex DAG dependency graph. 09:03 The Data Port agent is a two-task DAG that routes Slack pipeline alerts to either a predefined command list or an AI token, depending on whether the fix is already known. 13:10 AI is still in development at Jeppesen ForeFlight — the team is focused on token efficiency and scoping how much autonomy to give agents across different environments. 15:04 Airflow setup and MCP configuration were straightforward — the harder design work was deciding which environments agents could access across QA staging and production. 17:06 Airflow's skills repo and agent tooling are helping onboard new developers and extend pipeline awareness to analysts who work alongside engineers on the cycle. 19:10 Samantha would like to see single-task retries with different parameters in Airflow — resetting one task without clearing the full pipeline run. 21:05 A future AI use case under consideration is live DAG editing and re-upload within Airflow to make one-off fixes without halting pipeline progress. Resources Mentioned: Samantha Blaney Cuevas https://www.linkedin.com/in/samantha-blaney/ Jeppesen ForeFlight | LinkedIn https://www.linkedin.com/company/jeppesen-foreflight/ Jeppesen ForeFlight | Website http://www.foreflight.com Astronomer Airflow Skills Repo http://www.github.com/astronomer/airflow-llm-providers-demo Apache Airflow  https://airflow.apache.org/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines. Astronomer’s Head of Customer Education, Marc Lamberti, and Senior Manager of Developer Relations, Kenten Danas, break down what’s new, from asset partitioning to Async Python tasks and DAG versioning. They explore how these updates improve scheduling, performance and observability in production workflows. Key Takeaways: 00:00 Introduction. 02:10 Airflow 3 architecture separates workers from the metadata database. 03:05 Plugin versioning and UI-based backfills simplify operations. 06:20 Asset partitioning enables granular, partition-level scheduling. 07:15 Triggering DAGs on partitions instead of full datasets. 11:05 Deferrable operators reduce worker slot usage. 12:00 Async operators reduce database pressure and overhead. 14:10 Async improves throughput, not single task speed. 22:20 Inlets and outlets improve asset lineage visibility. 23:00 DAG version markers show changes directly in the UI. Resources Mentioned: Marc Lamberti https://www.linkedin.com/in/marclamberti/ Apache Airflow  https://airflow.apache.org/ Astronomer | LinkedIn https://www.linkedin.com/company/astronomer/ Astronomer | Website https://www.astronomer.io/ 3.2 Webinar https://www.astronomer.io/events/webinars/introducing-airflow-3-2-video Asset Partitioning Guide https://www.astronomer.io/docs/learn/airflow-partitioned-runs Asynchronous Processes Guide https://www.astronomer.io/docs/learn/deferrable-operators Release Notes https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-3-2-0-2026-04-07 Provider Registry https://airflow.apache.org/registry/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning

Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent. In this episode, Benjamin Rogojan, Owner and Data Consultant at Seattle Data Guy, joins us to reflect on how the data engineering landscape has evolved alongside Apache Airflow. We explore when Airflow makes sense as an orchestrator, why batch processing is still dominant and how AI is reshaping the workflows and responsibilities of modern data engineers. Key Takeaways: 00:00 Introduction. 03:00 Airflow becomes valuable when workflows involve many pipelines, teams and dependencies. 05:00 Data engineers are still focused on making data accessible and aligning work with business needs. 05:30 Batch pipelines remain the most common approach even as real-time use cases grow. 07:45 Many “real-time” requests are actually event-driven batch workflows. 09:00 Airflow replaced many custom-built pipeline systems with built-in dependency management. 11:00 Modern orchestration tools often build on Airflow concepts or differentiate from them. 14:00 AI can assist with writing SQL and pipelines but still requires experienced engineers. 15:30 Organizations are collecting increasingly granular data creating more engineering demand. 19:00 The data stack has shifted rapidly from Hadoop-era systems to modern cloud platforms. Resources Mentioned: Benjamin Rogojan https://www.linkedin.com/in/benjaminrogojan/ Seattle Data Guy https://www.linkedin.com/company/seattle-data-guy/ Apache Airflow https://airflow.apache.org Airflow Summit / Airflow Conference https://airflowsummit.org Snowflake https://www.snowflake.com HubSpot Data Sharing / APIs https://developers.hubspot.com MLflow https://mlflow.org Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

Data quality is not optional when you manage credit data at scale. In this episode, Ashir Alam, Senior Data Engineer at Credit Karma, joins us to share how his team acts as the gatekeeper for credit data ingestion, how they standardize data quality with Airflow and DAG Factory and how they scale safely across thousands of DAGs. We explore how governance, PII protection and orchestration come together inside a modern data platform.  Key Takeaways: 00:00 Introduction. 01:00 Overview of Credit Karma’s products and financial data ecosystem. 02:00 The team acts as gatekeepers for ingesting data from TransUnion and Equifax. 03:00 Why PII handling and controlled downstream access led to adopting Airflow. 04:00 BigQuery as the warehouse and Airflow as the primary orchestrator. 05:00 Why data quality and governance are critical in financial systems. 07:00 Why Airflow was selected: ease of use and unified ETL plus data quality. 09:00 Introduction to DAG Factory and YAML-based DAG generation. 10:00 GitHub executor creates PR-driven DAG workflows with CI checks. 12:00 BigQuery operators, structured checks and custom Slack and PagerDuty alerts. 13:00 Failed checks stop ETL pipelines and trigger notifications. 17:00 Scaling DAG Factory across thousands of DAGs and runtime vs compile-time concerns. 19:00 Future improvements: better defaults, retries and GenAI workflows in Airflow. Resources Mentioned: Ashir Alam https://www.linkedin.com/in/ashir-alam/ Credit Karma https://www.linkedin.com/company/intuit-credit-karma/ Apache Airflow https://airflow.apache.org/ DAG Factory https://github.com/astronomer/dag-factory BigQuery (Google Cloud) https://cloud.google.com/bigquery GitHub https://github.com/ Slack https://slack.com/ PagerDuty https://www.pagerduty.com/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

Modern Airflow isn’t just orchestration. It's a contribution.  In this episode, we explore how open source investment drives real performance gains and deeper observability. We’re joined by Christos Bisias, Open Source Software Engineer, Apache Airflow at G-Research, to discuss how his team uses Airflow for large-scale data transformations, contributes upstream and improves scheduler throughput and OpenTelemetry support. From trace-level observability to CI-enforced metrics governance and a major scheduler optimization, this conversation spans strategy, engineering and community impact. Key Takeaways: 00:00 Introduction. 01:20 How G-Research applies machine learning and big data to predict financial market movements. 02:15 Contributing to open source is a business decision. 03:10 Maintaining a fork is costly. 04:30 OpenTelemetry collects metrics, logs and traces to provide deep system visibility. 06:10 Custom spans help identify bottlenecks inside tasks and enable performance optimization. 08:05 OpenTelemetry integration works properly in Airflow 3.0 and above. 10:00 A YAML-based metrics registry with CI enforcement ensures consistency between docs and exported metrics. 12:10 Scheduler throughput improved significantly by applying concurrency limits earlier in the database query.  15:20 Future Task SDK changes may enable language-agnostic DAG authoring beyond Python. Resources Mentioned: Christos Bisias https://www.linkedin.com/in/xbis/ G-Research https://www.linkedin.com/company/g-research/ Apache Airflow https://airflow.apache.org/ OpenTelemetry https://opentelemetry.io/ Prometheus https://prometheus.io/ Grafana https://grafana.com/ Jaeger https://www.jaegertracing.io/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

In this episode, Karan Alang, Principal Software Engineer at Versa Networks, joins the conversation to discuss how Airflow can be used to automate threat intelligence in modern cybersecurity environments. He explains the growing scale of cloud computing, the profitability of hacking and the shortage of SOC analysts. Karan also outlines a novel architecture that combines Airflow, XDR, graph databases and LLMs to orchestrate automated threat detection and response. Key Takeaways: 00:00 Introduction. 05:00 Organizations face massive log volumes and a shortage of SOC analysts. 07:00 The solution integrates Airflow, XDR, Neo4j graph databases and LLMs into one architecture. 08:00 MITRE ATT&CK provides a global framework for mapping tactics and techniques. 11:00 Airflow acts as the orchestration backbone for ingestion graph transformation and LLM workflows. 13:00 Graph databases provide a full relationship view of attackers’ systems and entities. 14:00 LLMs automate mapping activity to MITRE ATT&CK and assign explainable risk scores. 17:00 Traditional signature-based detection allows lateral movement and exfiltration before teams can react. 18:00 End-to-end automation is essential to mitigating modern cybersecurity threats. 20:00 Future opportunities include deeper LLM integration as first-class citizens within Airflow. Resources Mentioned: Karan Alang https://www.linkedin.com/in/karan-alang-4173437 Versa Networks | LinkedIn https://www.linkedin.com/company/versa-networks Versa Networks | Website https://versa-networks.com Google Cloud Composer (Managed Airflow on GCP) https://cloud.google.com/composer Microsoft Defender XDR  https://www.microsoft.com/es-es/security/business/siem-and-xdr/microsoft-defender-xdr Neo4j (Graph Database) https://neo4j.com MITRE ATT&CK Framework https://attack.mitre.org Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning

In this episode, we explore how teams scale Apache Airflow in complex environments and what it takes to make orchestration work across many stakeholders. We look at real-world challenges around visibility, ownership and predictability as data platforms grow. Egor Tarasenko, Data and AI Engineer at Ponder Labs, joins us to share how Ponder Labs customizes Airflow for education organizations using plugins, event-driven architectures and AI-powered tooling. He explains how his team supports large charter school networks and why structure, consistency and extensibility become critical at scale. Key Takeaways: 00:00 Introduction. 01:21 Ponder Labs helps education organizations bring data from many systems together so it becomes useful for teachers, school leaders and administrators. 03:10 Airflow serves as the backbone for orchestrating ingestion, transformation and reverse ETL across client data platforms. 05:43 Everything is triggered from Airflow to maintain dependency, visibility and a single operational picture. 09:05 Managing hundreds of DAGs requires a focus on structure, visibility and consistency across teams. 09:51 Treating DAGs like APIs helps teams scale without needing deep knowledge of upstream logic. 12:00 Custom plugins like schedule insights help predict DAG run times across layered dependencies. 15:00 AI-powered Airflow chat enables non-technical stakeholders to understand DAG ownership dependencies and cluster activity. 22:06 Migrating plugins to Airflow 3 improves developer experience through cleaner APIs and faster extensibility. Resources Mentioned: Egor Tarasenko https://www.linkedin.com/in/egorseno/ Apache Airflow https://airflow.apache.org dbt https://www.getdbt.com Astronomer Astro Platform https://www.astronomer.io Egor Tarasenko on Substack  https://egortarasenko.substack.com Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge. In this episode, we’re joined by Ethan Shalev, Data Engineer at Wix, to discuss how Wix operates Airflow at massive scale, migrates to Airflow 3 and uses AI to accelerate development. Key Takeaways: 00:00 Introduction. 02:13 Wix structures data engineering across multiple product-focused organizations. 03:40 Migrating nearly 8,000 DAGs to Airflow 3 requires careful planning. 04:31 Migration creates an opportunity to remove long-standing legacy Airflow code. 05:32 Internal playbooks and Cursor rules standardize and speed up DAG migrations. 07:39 Airflow 3 introduces backfills, DAG versioning and asset-aware scheduling. 09:16 Deferrable operators reduce scheduler congestion in large Airflow environments. 12:54 AI-generated code still requires review and strong testing practices. 14:52 Moving to managed Airflow reduces operational burden on internal platform teams. 15:57 Improving multi-tenancy and UI personalization remains a key Airflow need. Resources Mentioned: Ethan Shalev https://www.linkedin.com/in/eshalev/ Wix | LinkedIn https://www.linkedin.com/company/wix-com/ Wix | Website https://www.wix.com/ Apache Airflow https://airflow.apache.org/ Astronomer https://www.astronomer.io/ Trino https://trino.io/ Apache Iceberg https://iceberg.apache.org/ Cursor https://cursor.sh/ Airflow Summit https://airflowsummit.org/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

Strong data orchestration is as much about culture and visibility as it is about technology. As data platforms scale, teams need systems that reduce cognitive load while increasing reliability and observability. In this episode, Carlos Daniel Puerto Niño, Senior Analytics Engineer and Data Analyst at Addi, joins us to share how Addi uses Airflow to support batch orchestration, manage organizational complexity and improve monitoring across its data platform. Key Takeaways: 00:00 Introduction. 01:25 Changes in company strategy increase data platform complexity over time. 04:00 Centralized data teams help manage organizational and technical change. 06:08 Scalable architectures support growing data volumes and use cases. 09:10 Adopting orchestration tools introduces operational and maintenance challenges. 14:43 Abstraction layers lower technical barriers for onboarding new team members. 15:36 Modularity and visibility improve the reliability of data pipelines. 18:14 Integrated monitoring supports faster incident response and resolution. 22:19 Limited access to orchestration metadata constrains proactive analysis. Resources Mentioned: Carlos Daniel Puerto Niño https://www.linkedin.com/in/carlospuertoni%C3%B1o/ Addi | LinkedIn https://www.linkedin.com/company/addicol/ Addi | Website https://www.addi.com Apache Airflow https://airflow.apache.org/ Astronomer https://www.astronomer.io/ Databricks https://www.databricks.com/ dbt https://www.getdbt.com/ Grafana https://grafana.com/ Slack https://slack.com/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

Real-time data expectations are reshaping how modern data teams think about orchestration and dependencies. As event-driven architectures become more common, teams need to rethink how pipelines react to data changes, rather than schedules. In this episode, Andrea Bombino, Co-Founder and Head of Analytics Engineering at Astrafy, joins us to discuss how event-driven scheduling in Airflow is evolving and how Astrafy applies it to deliver faster, more responsive data pipelines. Key Takeaways: 00:00 Introduction. 02:02 Astrafy’s role in guiding clients across the modern data stack. 03:15 Strong DAG dependencies create challenges for time-based scheduling. 04:48 Event-driven pipelines respond to increasing real-time data demands. 05:30 Airflow 3 introduces native support for event-driven orchestration. 06:27 Sensor-based workflows reveal scalability and efficiency limitations. 11:32 Event-driven assets improve efficiency and pipeline elegance. 14:45 Governance and cross-instance coordination emerge as ongoing challenges. Resources Mentioned: Andrea Bombino https://www.linkedin.com/in/andrea-bombino/ Astrafy | LinkedIn https://www.linkedin.com/company/astrafy/ Astrafy | Website https://www.astrafy.io Apache Airflow https://airflow.apache.org/ Google Cloud https://cloud.google.com/ Google Pub/Sub https://cloud.google.com/pubsub Google BigQuery https://cloud.google.com/bigquery Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

A strong data-driven mindset underpins how fintech teams scale analytics, infrastructure and decision-making across the business. In this episode, Jaime Oliveira, Lead Data Engineer at Uphold, joins us to discuss how Uphold structures its data organization and orchestration strategy. Jaime shares how the team uses Airflow and dbt to support analytics, reporting and data activation while evolving their approach as the stack grows. Key Takeaways: 00:00 Introduction. 01:23 A data-driven mindset supports product development and business decisions. 02:55 Diverse ingestion pipelines enable scalable analytics. 04:18 A single orchestration platform simplifies analytics workflows. 05:17 Early experience with orchestration tools shapes engineering practices. 08:16 Analytics orchestration works best when aligned with transformation workflows. 09:25 Infrastructure choices involve tradeoffs in testing, visibility and overhead. 16:39 More collaborative workflow tools could improve accessibility and autonomy. Resources Mentioned: Jaime Oliveira https://www.linkedin.com/in/jaime-oliveira-b075855a/ Uphold | LinkedIn https://www.linkedin.com/company/upholdinc/ Uphold | Website https://uphold.com Apache Airflow https://airflow.apache.org dbt https://www.getdbt.com Snowflake https://www.snowflake.com Kubernetes https://kubernetes.io Astronomer Cosmos https://astronomer.github.io/astronomer-cosmos Cosmos e-book https://www.astronomer.io/ebooks/orchestrating-dbt-with-airflow-using-cosmos/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

Building reliable data pipelines at scale requires more than writing code. It depends on thoughtful design, infrastructure trade-offs and an understanding of how orchestration platforms evolve over time. In this episode, Airflow best practices shaped by real-world implementation are examined. Bhavani Ravi, Independent Software Consultant and Apache Airflow Champion, shares lessons on pipeline design, architectural decisions and the evolution of the Airflow ecosystem in modern data environments. Key Takeaways: 00:00 Introduction. 01:30 Independent consulting supports effective Airflow adoption. 02:38 Early challenges shaped modern Airflow practices. 03:21 Airflow setup has become significantly simpler. 04:30 New features expanded workflow capabilities. 06:03 Frequent releases support long-term sustainability. 07:34 Community and providers strengthen the ecosystem. 10:03 Pipeline design should come before coding. 10:55 Decoupling logic requires careful trade-offs. 13:30 Plugins extend Airflow into new use cases. Resources Mentioned: Bhavani Ravi https://www.linkedin.com/in/bhavanicodes/ Apache Airflow https://airflow.apache.org/ Kubernetes https://kubernetes.io/ Azure Fabric https://learn.microsoft.com/en-us/fabric/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

Conviva operates at a massive scale, delivering outcome-based intelligence for digital businesses through real-time and batch data processing. As new use cases emerged, the team needed a way to extend a streaming-first architecture without rebuilding core systems. In this episode, Han Zhang joins us to explain how Conviva uses Apache Airflow as the orchestration backbone for its batch workloads, how the control plane is designed and what trade-offs shaped their platform decisions. Key Takeaways: 00:00 Introduction. 01:17 Large-scale data platforms require low-latency processing capabilities. 02:08 Batch workloads can complement streaming pipelines for additional use cases. 03:45 An orchestration framework can act as the core coordination layer. 06:12 Batch processing enables workloads that streaming alone cannot support. 08:50 Ecosystem maturity and observability are key orchestration considerations. 10:15 Built-in run history and logs make failures easier to diagnose. 14:20 Platform users can monitor workflows without managing orchestration logic. 17:08 Identity, secrets and scheduling present ongoing optimization challenges. 19:59 Configuration history and change visibility improve operational reliability. Resources Mentioned: Han Zhang https://www.linkedin.com/in/zhanghan177 Conviva | Website http://www.conviva.com Apache Airflow https://airflow.apache.org/ Celery https://docs.celeryq.dev/ Temporal https://temporal.io/ Kubernetes https://kubernetes.io/ LDAP https://ldap.com/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

Data platforms are moving from batch-first pipelines to near real-time systems where orchestration, observability, scalability and governance all have to work together. In this episode, Arun Karthik, Director, Data Solutions Engineering at Condé Nast Technology Lab, joins us to share how data engineering evolves from relational databases and ETL into distributed processing, modern orchestration with Apache Airflow and managed Airflow with Astronomer. Key Takeaways: 00:00 Introduction. 02:13 Early data systems rely heavily on relational databases and batch-oriented processing models. 07:01 Scheduling requirements evolve beyond fixed time windows as dependencies increase. 10:14 Ease of use and developer experience influence adoption of orchestration frameworks. 13:22 Operating open source orchestration tools requires ongoing engineering effort. 14:45 Managed services help teams reduce infrastructure and maintenance responsibilities. 17:27 Observability improves confidence in pipeline execution and system health. 19:12 Governance considerations grow in importance as data platforms mature. 20:46 Building data systems requires balancing speed, reliability and long-term sustainability. Resources Mentioned: Arun Karthik https://www.linkedin.com/in/earunkarthik/ Condé Nast Technology Lab | LinkedIn https://www.linkedin.com/company/conde-nast-technology-lab/ Condé Nast Technology Lab | Website https://www.condenast.com/ Apache Airflow https://airflow.apache.org/ Astronomer https://www.astronomer.io/ Apache Spark https://spark.apache.org/ Apache Hadoop https://hadoop.apache.org/ Jenkins https://www.jenkins.io/ dbt Labs https://www.getdbt.com/product/what-is-dbt Amazon Web Services https://aws.amazon.com/free/?trk=54026797-7540-48d8-9f6b-0db2c3a0040c&sc_channel=ps&trk=54026797-7540-48d8-9f6b-0db2c3a0040c&sc_channel=ps&ef_id=CjwKCAiAmp3LBhAkEiwAJM2JUKIc3E2I-hDlF6fRWgZn5n2-RWX-kEDAVApJYd88wwlsiyosV71VixoCmRoQAvD_BwE:G:s&s_kwcid=AL!4422!3!785574063524!e!!g!!amazon%20web%20services!23291338728!189486861095&gad_campaignid=23291338728&gbraid=0AAAAADjHtp813XNbg7azDj5QMwJPbGNqZ&gclid=CjwKCAiAmp3LBhAkEiwAJM2JUKIc3E2I-hDlF6fRWgZn5n2-RWX-kEDAVApJYd88wwlsiyosV71VixoCmRoQAvD_BwE Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow