All the Hard Stuff with LLMs in Product Development // Phillip Carter // MLOps Podcast #170

MLOps Coffee Sessions #170 with Phillip Carter, All the Hard Stuff with LLMs in Product Development. We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O // Abstract Delve into challenges in implementing LLMs, such as security concerns and collaborative measures against attacks. Emphasize the role of ML engineers and product managers in successful implementation. Explore identifying leading indicators and measuring ROI for impactful AI initiatives. // Bio Phillip is on the product team at Honeycomb where he works on a bunch of different developer tooling things. He's an OpenTelemetry maintainer -- chances are if you've read the docs to learn how to use OTel, you've read his words. He's also Honeycomb's (accidental) prompt engineering expert by virtue of building and shipping products that use LLMs. In a past life, he worked on developer tools at Microsoft, helping bring the first cross-platform version of .NET into the world and grow to 5 million active developers. When not doing computer stuff, you'll find Phillip in the mountains riding a snowboard or backpacking in the Cascades. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links ⁠Website: https://phillipcarter.dev/ https://www.honeycomb.io/blog/improving-llms-production-observability https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm https://phillipcarter.dev/posts/how-to-make-an-fsharp-code-fixer/ The "hard stuff" post: https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm Our follow-up on iterating on LLMs in prod: https://www.honeycomb.io/blog/improving-llms-production-observability --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Phillip on LinkedIn: https://www.linkedin.com/in/phillip-carter-4714a135/ Timestamps: [00:00] Phillip's preferred coffee [00:33] Takeaways [01:53] Please like, share, and subscribe to our MLOps channels! [02:45] Phillip's background [07:15] Querying Natural Language [11:25] Function calls [14:29] Pasting errors or traces [16:30] Error patterns [20:22] Honeycomb's Improvement cycle [23:20] Prompt boxes rationale [28:06] Prompt injection cycles [32:11] Injection Attempt [33:30] UI undervalued, charging the AI feature [35:11] ROI cost [44:26] Bridging ML and Product Perspective [52:53] AI Model Trade-offs [56:33] Query assistant [59:07] Honeycomb is hiring! [1:00:08] Wrap up