Airflow Sucks for MLOps // Stephen Bailey // MLOps Podcast #141

MLOps Coffee Sessions #141 with Stephen Bailey, Airflow Sucks for MLOps co-hosted by Joe Reis. // Abstract Stephen discusses his experience working with data platforms, particularly the challenges of training and sharing knowledge among different stakeholders. This talk highlights the importance of having clear priorities and a sense of practicality and mentions the use of modular job design and data classification to make it easier for end users to understand which data to use.    Stephen also mentions the importance of being able to move quickly and not getting bogged down in the quest for perfection. We recommend Stephen's blog post "Airflow's Problem" for further reading. // Bio Stephen has worked as a data scientist, analyst, manager, and engineer, and loves all the domains equally. He currently works at Whatnot, a collectibles marketplace that focuses on live shopping, and has previously worked in privacy tech at Immuta. He has his Ph.D. from Vanderbilt University in educational cognitive neuroscience, but it has yet to help him understand why his three children are so crazy. // MLOps Jobs board   https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Airflow's Problem blog post: https://stkbailey.substack.com/p/airflows-problem Airflow's Problem and the reception it got on Hacker News: https://news.ycombinator.com/item?id=32317558 --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Joe on LinkedIn: https://www.linkedin.com/in/josephreis/ Connect with Stephen on LinkedIn: https://www.linkedin.com/in/stkbailey/ Timestamps: [00:00] Stephen's preferred coffee [00:19] Introduction to co-host Joe Reis [01:40] Takeaways [06:29] Subscribe to our newsletters! [06:55] Shout out to our sponsor, Wallaroo! [08:05] Whatnot [10:47] Stephen's side hustle [14:35] Stephen's work breakdown at Whatnot [18:03] Fundamental tensions in the data world [21:27] Initial questions to answer that you were on the right path [24:06] Recommender systems [28:15] Coordinating with ML teams [29:43] Daxter [31:38] Too advanced, more challenging [34:37] Orchestration layer [36:14] Decision criteria [39:23] Human design aspect of Daxter [40:53] Orchestration layer centralization and sharing knowledge with stakeholders [46:18] Airflow's Problem and the reception it got on Hacker News [51:00] Wrap up