#118 - Data Engineering (with Gal Shpantzer)

Our systems generate fantastic amounts of information, but do we have a complete understanding of how we collect, analyze, manage, store, and retrieve possibly petabytes a day? Gal Shpantzer has been doing InfoSec for over 20 years and has managed some huge data engineering projects, and offers a lot of actionable insights in this CISO Tradecraft episode. Gal's LinkedIn Page - https://www.linkedin.com/in/riskmanagement/ Gal's Twitter Page - https://twitter.com/Shpantzer Full Transcript - https://docs.google.com/document/d/14RXnsVttvKlRi6VL94BTrItCjOAjgGem/ Chapters 00:00 Introduction 02:00 How do you Architect Big Data Data Infrastructure 03:33 Are you taking a look at Ransomware? 06:11 Web Scale Technologies are used mostly in Marketing & Fraud Detection 08:11 Data Engineering - The Mindset Shift 10:51 The Iron Triangle of Data Engineering 13:55 Can I Outsource My Logging Pipeline to a Vendor 15:37 Kafka & Flink - Data Engineering in the Pipeline 18:12 Streaming Analytics & Kafka 22:08 How to Enable Data Science Analytics with Streaming Analytics 26:33 Streaming Analytics 30:25 Data Engineering - Is there a Security Log 32:30 Streaming Analytics is a Weird Thing 35:50 How to Get a Handle on a Big Data Pipeline 39:11 Data Engineering Hacks for Big Data Analytics