Data Superstream: Data Lakes and Warehouses

Schedule

The time frames are only estimates and may vary according to how the class is progressing.

Alistair Croll: Introduction (5 minutes) - 8:00am PT | 11:00am ET | 4:00pm UTC/GMT

• Alistair Croll welcomes you to the Data Superstream.

Tejas Manohar: Keynote—Automated Decisioning: How are Data Teams Transforming Business? (15 minutes) - 8:05am PT | 11:05am ET | 4:05pm UTC/GMT

• Data organizations were traditionally built to derive insights and serve them to business leaders to enable better strategic decision-making. With the explosion of digital data, a lot of these decisions—from personalized marketing campaigns to advertising bidding to customer service and billing processes—are becoming automated and streamlined. Businesses are moving from human-in-the-loop systems to humans creating the loop. Tejas Manohar gives you a deeper look into this transformation and discusses how to best prepare your business and data team.
• Tejas Manohar is co-CEO and cofounder of Hightouch. The company was born out of his realization that the challenges of building a best-in-class customer data platform (CDP) would be better solved on top of cloud data warehouses, an idea that took shape during his tenure as an engineer at Segment, the leading company in the CDP space. When Tejas isn’t thinking about data, he plays competitive table tennis and enjoys hiking and anything outdoors.

Christina Taylor: Simplify Data Lake Implementation—Let’s Fall Back in Love with Streaming (30 minutes) - 8:20am PT | 11:20am ET | 4:20pm UTC/GMT

• As data approaches real time, building a centralized data repository can become complex and costly. How do we simplify data lake implementation and maintenance? How do we reduce points of failures and customer bugs? Christina Taylor offers an alternative approach to data streaming: incremental processing. With long-term stewardship and developer happiness in mind, she presents different flavors of incremental ingestion, from event-based change data capture to direct polling and query federation, reviews the pitfalls and trade-offs of each, and briefly discusses implementing incremental sourcing in SAAS applications.
• Christina Taylor is a senior staff data engineer at Catalyst Software, where she makes data-informed architectural decisions to build data products that support advanced analytics, machine learning, and customer-facing applications. She’s passionate about open source, multicloud, scalable and efficient data pipelines, and has a keen interest in interdisciplinary areas such as cloud FinOps, DevOps and MLOps. Christina also loves to learn, share, and contribute to the open source community as a technical writer and presenter.

Alex Merced: Better Together—When Data Virtualization, Data Lakehouses, and Data Mesh Collide (Sponsored by Dremio) (30 minutes) - 8:50am PT | 11:50am ET | 4:50pm UTC/GMT

• We find ourselves at the intersection of three powerful data trends: data virtualization, data lakehouses, and data mesh. In a world where data is everywhere, freshness is paramount, and accountability is crucial, these trends offer a glimpse into a future where data flows seamlessly, insights are delivered in real time, and governance is crystal clear. Alex Merced explains how data virtualization unifies disparate data sources, how data lakehouses drive data at the speed of business, and how data mesh provides a framework for responsible data governance. Fasten your seatbelts for a journey into the future of data excellence.
• Alex Merced is a developer advocate for Dremio and has worked as a developer and instructor for companies like GenEd Systems, Crossfield Digital, CampusGuard, and General Assembly. He’s passionate about technology and has put out tech content on outlets such as blogs, videos, and his podcasts Datanation and Web Dev 101. Alex has contributed a variety of libraries in the JavaScript and Python worlds, including SencilloDB, CoquitoJS, dremio-simple-query, and more.
• This session will be followed by a 30-minute Q&A in a breakout room. Stop by if you have more questions for Alex.
• Break (10 minutes)

Chang She: Enhancing Lakehouse Infrastructure for Multimodal AI with Lance Columnar Format (30 minutes) - 9:30am PT | 12:30pm ET | 5:30pm UTC/GMT

• AI brings a host of new challenges to data lakehouses. Existing formats like parquet and ORC are not well suited for ML/AI workloads, including vector search, deep learning, model evaluations, and EDA for unstructured data. Lance is a new open source columnar format that offers more than an order of magnitude better performance for AI workloads and is optimized for modern storage options. Join Chang She as he dives into this new format, reviews how it's designed, and explores how it can be applied to AI workloads to offer significantly better performance with nearly zero additional effort.
• Chang She is CEO and cofounder of LanceDB, a next-generation database for AI. A serial entrepreneur, he’s been building DS/ML tooling for nearly two decades and is one of the original contributors to the pandas library. Previously, Chang was VP of engineering at Tubi TV, where he focused on personalized recommendations and ML experimentation.

Zack Klein: Leveraging Data Contracts in the Modern Data Stack (40 minutes) - 10:00am PT | 1:00pm ET | 6:00pm UTC/GMT

• The explosion of businesses relying on data day-to-day has put modern data teams in a tough spot: they must deliver accurate, reliable, high-quality data pipelines while relying on increasingly complex sources upstream. For data teams in this position, one practical approach involves the use of data contracts. Zack Klein discusses the fundamental components of data quality and trust, the evolving role of the data platform, and how data contracts are one way that data teams can keep their customers happy and sleeping soundly at night.
• Zack Klein is a software engineer at Whatnot, where he thoroughly enjoys building data products and narrowly avoiding breaking production each day. Previously, he worked on big data platforms at Blackstone and HBO. When he is not coding or writing, Zack plays his telecaster, watches the San Francisco Giants, or listens to Dawes.
• Break (5 minutes)

Shankar Narayanan, SGS: Developing and Deploying ML Models on a Cloud Data Warehouse (30 minutes) - 10:45am PT | 1:45pm ET | 6:45pm UTC/GMT

• As more organizations collect and consume an increasing amount of data, there’s a commensurate demand for performing advanced analytics on this data to enable tools like ML models. To address this need, cloud data warehouses allow users to deploy ML models directly on the warehouse without having to extract the data. Shankar Narayanan discusses how Snowflake allows you to consume data using Data Cloud and the Snowflake Marketplace, build models using SnowparkML, and deploy them with Streamlit and the Snowflake Native Apps Framework.
• Shankar Narayanan is a principal architect at Microsoft responsible for supporting Microsoft integrations with Snowflake and helping Snowflake users achieve success on their data journey. He has decades of experience implementing BI and analytics solutions for organizations across various industries, as well as for some of the preeminent Fortune 500 companies. Shankar is also a Snowflake Data Superhero.

Carlos Costa: Standardizing Lakehouse Data Modeling and Pipelining (40 minutes) - 11:15am PT | 2:15pm ET | 7:15pm UTC/GMT

• The use of data modeling and common data pipelining standards may sometimes feel like a lost art in the data engineering space. The data lakehouse brought a more agile version of the data warehouse, but given the need for fast action and the ever-growing requirements of organizational use cases, data product teams often prioritize quickly created MVP data models and pipelines in the lakehouse over the use of shared organizational standards. There might be a better approach. Carlos Costas explores why the most optimal solution is somewhere in the middle, between a very structured, rigid lakehouse and a lakehouse without any rules. He dives into a set of data modeling and pipelining instruments that can help data product teams achieve simple, flexible, open, and efficient data models and data pipelines in their lakehouse.
• Carlos Costa is a director of data engineering at Adidas and an Invited Professor at the University of Minho in Portugal. A data enthusiast who’s always hungry for new challenges, he looks to push the barriers of knowledge and technological innovation. He has several years of experience in data and analytics, including hands-on practice, leadership, project and product management, research, and teaching. Carlos contributes to the scientific and technical data communities through open source projects, scientific publications, and speaking at public events.

Alistair Croll: Closing Remarks (5 minutes) - 11:55am PT | 2:55pm ET | 7:55pm UTC/GMT

• Alistair Croll closes out today’s event.

Upcoming Data Superstream events:

• Data Products and Data Teams - October 16, 2024