Conference talks, keynotes, meetups, and panels on open source data infrastructure: catalogs, pipelines, DataOps, and what AI changes about all of it. Recordings are linked where they exist. For the weekly show, see Open Lakehouse + AI.
33 talks · 11 podcast and interview appearances
2026
Getting Your Agents to Write (Good) Pipelines on Apache Spark
Upcoming · talk OSA CON 2026 · San Francisco + online · 2026-11-02
Practical patterns for AI-assisted Spark pipelines: catching structural mistakes before data moves, keeping credentials and session context away from agents via Spark Connect, and structuring skills and prompts so one-shot generations land.
A Decade of Spark + Airflow: How Two Communities Built the De Facto Standard for Data Engineering
talk Airflow Summit 2026 · Austin, TX · 2026-09-02
With Jarek Potiuk.
A community history of how Spark and Airflow co-evolved, from BashOperator wrapping spark-submit to asset-aware scheduling plus Declarative Pipelines, and why the pairing outlasted managed alternatives.
Declarative Pipelines Meet Declarative Orchestration: Spark Declarative Pipelines + Airflow 3
talk Airflow Summit 2026 · Austin, TX · 2026-08-31
With Andreas Neumann.
Builds a production Spark Declarative Pipeline orchestrated by Airflow 3, showing how dependency graphs replace imperative task chains and how testing and recovery change when the pipeline is declarative end to end.
From Repetition to Reuse: The Evolution of Apache Spark™ Declarative Pipelines
talk Data + AI Summit 2026 · San Francisco, CA · June 15-18, 2026
With Andreas Neumann.
How SDP (introduced in Spark 4.1) separates what a pipeline does from how it runs, covering dependency resolution, parallelization, checkpoint coordination, failure recovery, incremental processing and testing patterns.
Vibe Check: Can AI Actually Write Spark? Live Coding w/ Databricks & Snowflake
webinar (livestream guest) Temporal livestream · Virtual · 2026-05-13
With Holden Karau.
Live-coding session testing Claude Code on real Spark problems; Lisa showed using a skill file to get correct Declarative Pipelines code and discussed local mode, Arrow UDFs and Python UDF transpilation for AI-assisted Spark.
Production-Ready Data Engineering Made Simple with Apache Spark’s new Declarative Pipelines
meetup NYC Apache Spark Meetup (sponsored by DataFlint) · New York, NY · 2026-02-04
How experienced and new Spark developers can use Spark 4.1 Declarative Pipelines for simpler debugging and pipeline orchestration without hand-managing execution logic.
2025
Designing Data Infrastructure in the Age of Generative AI
talk O’Reilly Data Superstream: Data Engineering in the Age of AI · Virtual · Nov 2025; clip published 2026-02-03
What AI-scale data systems need: multimodal and semi-structured data, format features like variant and binary columns in Iceberg/Delta/Parquet, Delta Kernel, and a catalog-first architecture with Unity Catalog OSS.
Designing Data Infrastructure in the Age of Generative AI
talk AI By the Bay 2025 · Oakland, CA · Nov 17-19, 2025
Surveys data platforming challenges for AI agents and the rising importance of open table formats, catalogs and embedded systems, using Spark, Unity Catalog OSS and Iceberg as a reference architecture.
UC: The Everything Catalog
meetup Databricks DevConnect Washington DC · Washington, DC · 2025-11-04
Session on Unity Catalog as a single catalog for data and AI assets (DevConnect technical meetup).
Fundamentals of DataOps: A Practitioner’s Guide
talk All Things Open 2025 · Raleigh, NC · Oct 12-14, 2025
A practical roadmap for building DataOps infrastructure from scratch, integrating CI/CD, containers, cloud-native and data infrastructure, MLOps and security with open-source tools.
Why Open Source is Key to Future Data and AI Governance
keynote Community Over Code North America 2025 (ASF) · Minneapolis, MN · Sep 11-14, 2025
Keynote (AI track) on why open source is central to data and AI governance; listed under Datastrato at time of announcement.
Unity Catalog: The Everything Catalog
meetup Databricks DevConnect Vancouver · Vancouver, BC (Deloitte offices) · 2025-09-04
Session on Unity Catalog as a single catalog for data and AI assets (DevConnect technical meetup).
Using metadata to power and govern MCP and A2A in open lakehouse architecture
meetup Forge Utah !meetup · Utah (streamed) · 2025-07-16
Why data catalogs and metadata are the path to governed AI development and agent protocols (MCP, A2A), with Apache Gravitino as a vendor-neutral example.
Apache Gravitino: A Multi-regional, Geo-distributed Meta Datalake
talk Open Source Summit North America 2025 · Denver, CO · 2025-06-24
With Justin Mclean.
Gravitino as a single source of truth for multi-regional metadata with geo-distributed support, unified data and AI asset management, and centralized governance.
Lightning Talk: One Year In: The State of the DataOps Initiative
talk (lightning) Open Source Summit North America 2025 · Denver, CO · 2025-06-24
Progress report on the CDF DataOps Initiative’s first year: principles, blog posts, community meetings, a vendor-neutral DataOps landscape and workshop formats bridging CI/CD and data platforms.
A Deep Dive into Apache Gravitino’s Proxy-Enabled Iceberg Rest Catalog
meetup Apache Iceberg Meetup · Palo Alto, CA · 2025-05-16
Technical walkthrough of Gravitino’s Iceberg REST catalog service acting as a proxy in front of other catalogs (paraphrased from title; no abstract published).
Streamlining metadata management with PostgreSQL and Apache Gravitino
talk Postgres Conference 2025 · Orlando, FL · 2025-03-21, 14:00 EDT, Seminole B, Ops track
How Gravitino, backed by PostgreSQL, provides centralized, secure, geo-distributed metadata storage and simpler administration for data and AI assets.
Fundamentals of DataOps (CD Workshop)
workshop Continuous Delivery Foundation Workshop · Virtual · 2025-03-27
Beginner’s roadmap for building DataOps infrastructure from scratch with CI/CD, containers, data infrastructure, MLOps and security using open-source tools.
How to Stream Data into Apache Iceberg with Timeplus: Efficient and Elastic
webinar (panel guest) Timeplus Live webinar (Timeplus Enterprise 2.8 launch) · Virtual · Mar 2025
With Jove Zhong (host), Ken Chen.
Guest panel discussion on Apache Iceberg’s open format and catalogs, alongside a demo of Timeplus’s C++ Iceberg read/write.
How to Write a Strong Conference Proposal?
panel By the Bay (FunctionalTV) panel · virtual (online panel; venue not stated)
With Alexy Khrabrov, Baruch Sadogursky, Oli Dinov (host).
Panel on structuring talk proposals, scoping, common rejection reasons and how technical depth and takeaways improve acceptance.
History and Future of Iceberg REST Catalogs
talk Data Day Texas 2025 · Austin, TX · 2025-01-25
Why the Iceberg community defined a REST catalog spec instead of shipping a service, and what the wave of third-party REST catalog providers means for the ecosystem.
Fundamentals of DataOps
talk Data Day Texas 2025 · Austin, TX · 2025-01-25
Roadmap for implementing DataOps infrastructure with CI/CD, containerization, data infrastructure, MLOps and security using open-source tooling.
The New, the old, and the current state of Data Catalogs
panel (moderated discussion) Data Day Texas 2025, Sunday Data Discussions · Austin, TX · 2025-01-26
Lisa-led open discussion on the state of data catalogs (no description published).
2024
To Mesh, or Not to Mesh? How to Know When Fabric is Good Enough
meetup Data for AI Presents: The Open Source Afterparty w/ DAGWorks, Marquez, & Datastrato · San Francisco, CA (hosted at Airbyte) · 2024-10-29
With Willy Lulciuc, DAGWorks speaker.
When a data fabric is sufficient versus when to adopt a data mesh; Lisa also co-hosted the event.
To Mesh, or Not to Mesh? How to Know When a Fabric is Good Enough
talk BITS 2024 (Bletchley Institute Technology Summit) · New York, NY · Oct 25-26, 2024
Reasons to choose a data mesh or a data fabric, and how to tell when a fabric is good enough.
Metadata Lakes for Next-Gen AI/ML
meetup Unstructured Data Meetup New York (Zilliz) · New York, NY · 2024-10-23
With Tim Spann, Uri Goren, Naren H.
How catalogs evolve for high-velocity unstructured data, activating metadata for RAG pipelines, with Gravitino’s open-source, multi-cloud, geo-distributed cataloging.
The rise of open source data catalogs featuring: Unity Catalog, DataHub, Apache Gravitino, and Apache Polaris (Incubating)
panel Open Source Data Summit 2024 · Virtual · 2024-10-02
With Denny Lee, Shirshanka Das, Kyle Weller, Russell Spitzer.
Panel on why open-source governance matters for lakehouses and what Unity Catalog, DataHub, Gravitino and Polaris each bring.
The Role of Data in Generative AI
panel LF AI & Data Foundation online panel · Virtual · 2024-09-12, 10-11 AM PDT
With Denny Lee, Nick Schifano, Anni Lai (moderator).
Panel on data collection, preprocessing, governance, bias, privacy regulation and dataset quality for generative AI.
REST Catalogs in Iceberg
talk Chill Data Summit On Tour: Bay Area (Upsolver) · San Carlos, CA · 2024-08-15
History of the Iceberg REST catalog spec, why the community chose a spec over a service, and the third-party catalog services it spawned.
Using Metadata to Optimize Machine Learning Workflows
meetup San Francisco Open Source Data Meetup (Aiven) · San Francisco, CA · 2024-06-13
With Alice Lyu.
Practical ways to use metadata to optimize ML workflows, improve discoverability and streamline data management on any stack.
Metadata Lakes for Next-Gen AI/ML
meetup Unstructured Data Meetup South Bay Edition (Zilliz, SAP) · Palo Alto, CA · 2024-06-25
With Kate Grossman, Yi Wang.
Metadata lakes as a flexible way to serve unstructured-data and RAG needs, featuring Gravitino’s multi-cloud catalog approach.
2023
Open Source DataOps and MLOps Strategies
talk Linux Foundation Member Summit 2023 · Monterey, CA · 2023-10-25
Tackles interoperability, standardization and vendor lock-in, and strategies for using open source across DataOps and MLOps from pipelines to model serving.
2021
BiocSwirl: Interactive R Tutorials for Bioinformatics
talk (short) BioC2021 (Bioconductor conference), short talk · Virtual · Aug 2021
With Julia Philipp, Matt Moss, Almas Khan, Jasdeep Singh, Mariam Arab, Sourav Singh (co-authors).
BiocSwirl, an R package of interactive, self-paced bioinformatics courses with real-time syntax feedback for beginners.
Podcasts and interviews
What’s New in Apache Spark? (ep 78 with Lisa Cao @Databricks)
podcast guest It’s About Data (podcast, Matt Housley) · Virtual
With Matt Housley (host).
Discussion of major new features in Apache Spark 4.1 and how the platform is evolving for new workloads and a broader user base.
Can you even build Databricks entirely out of Open Source Software?
video guest Databricks YouTube (OverArchitected) · Virtual · 2026-04-01
Whiteboards a fully open-source lakehouse (object storage, Delta/Iceberg, UC/Polaris/Gravitino catalogs, Spark, SDP, orchestration) based on Lisa’s lakehouse-at-home project.
Beyond the Pipeline: Why DataOps is the New Urgency in ML and Data Engineering (Ep 136)
podcast guest Cloud Native Podcast (Saim Safder) · Virtual
With Saim Safder (host).
Why DataOps matters now, DataOps vs DevOps vs MLOps, and how teams can adopt a practical, open-source-first approach to data delivery.
How to Pivot into Product Management Without Losing Your Technical Edge
podcast guest From a Woman to a Leader (podcast, Limor Bergman Gross) · Virtual
With Limor Bergman Gross (host).
Lisa’s move from data engineering to product management, product mindset, systems thinking and building credibility across the IC-to-PM transition.
Revolutionizing Data Governance with DataStrato’s Unified Open Source Approach
podcast guest The Data Engineering Show (Firebolt) · Virtual
With The Data Engineering Show hosts.
Data catalogs and Apache Gravitino as a unified metadata lake for access management and governance across sources, plus unifying AI/ML and big data stacks.
S02E02 Lisa N Cao: Apache Gravitino, Future of Data Ops & Metadata Management for AI
podcast guest Specific Data & Strategy (podcast, Karl Ivo Sokolov) · Virtual
With Karl Ivo Sokolov (host).
Metadata, the complexity of data catalogs, Gravitino’s vision for governance, DataOps myths and the future of data engineering under AI.
Fika Interview with Lisa Cao
interview Neo4j Fika interviews, recorded at Data Day Texas 2025 · Austin, TX
Short interview on running the Open Source Afterparty AI meetup in SF and Gravitino as a ‘catalog of catalogs’ bridging AI and data stacks.
Confessions of a Data Engineer: From Code to Open Source & AI Product Leadership
podcast guest (fireside) W2D2 (Women in Data) conversation · Virtual
With Lindsey Robertson (host).
Conversation on Lisa’s path from data engineer to product manager in open source data architecture.
Build Propulsion Lab at Community Over Code 2024 with Lisa Cao on challenges in data architecture
interview Develocity Build Propulsion Lab at Community Over Code NA 2024 · Denver, CO · Oct 2024
Short on-site interview about challenges in data architecture.
Open Source, Data Engineering, GovTech and Diversity in Tech
podcast guest Tech Talks with Madona (podcast) · Virtual · 2023-11-02
With Madona Wambua (host).
Lisa’s path across startups, nonprofits, GovTech and biotech, observability and reliability in data systems, and diversity advocacy in tech.
Muse and Metrics, S5E8 - The Whole Picture with Lisa Cao
podcast guest Muse and Metrics (360 Talk Radio, Women Techmakers season) · Virtual · 2023-11-01
With Philippa Burgess (host).
Interview with Lisa as a Google Women Techmakers Ambassador and data engineer, themed ‘The Whole Picture’.