Experiment Tracking in the Age of LLMs // Piotr Niedźwiedź // MLOps Podcast #168

MLOps Coffee Sessions #168 with Piotr Niedźwiedź, Experiment Tracking in the Age of LLMs, co-hosted by Vishnu Rachakonda. // Abstract Piotr shares his journey as an entrepreneur and the importance of focusing on core values to achieve success. He highlights the mission of Neptune to support ML teams by providing them with control and confidence in their models. The conversation delves into the role of experiment tracking in understanding and debugging models, comparing experiments, and versioning models. Piotr introduces the concept of prompt engineering as a different approach to building models, emphasizing the need for prompt validation and testing methods. // Bio Piotr is the CEO of neptune.ai. Day to day, apart from running the company, he focuses on the product side of things. Strategy, planning, ideation, getting deep into user needs and use cases. He really likes it. Piotr's path to ML started with software engineering. Always liked math and started programming when he was 7. In high school, Piotr got into algorithmics and programming competitions and loved competing with the best. That got him into the best CS and Maths program in Poland which funny enough today specializes in machine learning. Piotr did his internships at Facebook and Google and was offered to stay in the Valley. But something about being a FAANG engineer didn’t feel right. He had this spark to do more, build something himself. So with a few of his friends from the algo days, they started Codilime, a software consultancy, and later a sister company Deepsense.ai machine learning consultancy, where he was a CTO. When he came to the ML space from software engineering, he was surprised by the messy experimentation practices, lack of control over model building, and a missing ecosystem of tools to help people deliver models confidently. It was a stark contrast to the software development ecosystem, where you have mature tools for DevOps, observability, or orchestration to execute efficiently in production. And then, one day, some ML engineers from Deepsense.ai came to him and showed him this tool for tracking experiments they built during a Kaggle competition (which they won btw), and he knew this could be big. He asked around, and everyone was struggling with managing experiments. He decided to spin it off as a VC-funded product company, and the rest is history. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links https://neptune.ai/blog/author/piotr-niedzwiedz https://www.youtube.com/playlist?list=PLKePQLVx9tOfKFbg9GY2Anl41Be4T1-m5 https://thesequence.substack.com/p/-piotr-niedzwiedz-neptunes-ceo-on https://open.spotify.com/episode/2KEqTMAHODbPKdUEtlrhm7?si=ed862b2ac7534e39https://www.linkedin.com/in/piotrniedzwiedz/ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Piotr on LinkedIn: https://www.linkedin.com/in/piotrniedzwiedz/ Timestamps: [00:00] Introduction to Piotr Niedźwiedź [01:35] Please like, share, and subscribe to our MLOps channels! [01:58] Wojciech Zaremba [05:20] The Olympiad [06:31] Building own company [12:16] Talking outside Poland with the same passion [13:45] Adapting with Neptune [19:35] Core values focus [22:02] Product Vision changes with advances [29:36] Control and confidence [30:05] Experiment tracking existing use cases [37:25] Control pane [38:59] Piotr's prediction [43:20] WiFi issues around the world [44:09] Wrap up