66: Tracing Production with Kai Wern Choong

We talk with Kai Wern Choong about his experience with tracing performance problems in production. We talk about what “tracing” means in a BEAM system, available tools like recon, great resources for learning how the tools work and general approaches for troubleshooting live production Elixir systems. Kai also shares his Livebook notebook setup for demonstrating tracing techniques in an interactive way. Show Notes online - http://podcast.thinkingelixir.com/66 Elixir Community News https://twitter.com/hexpm/status/1439268469296570368 – New version of Hex released that fixes issues with version downgrades and slow version resolutions. Run mix local.hex to upgrade. https://hexdocs.pm/credo/1.6.0-rc.0/changelog.html#1-6-0 – Credo 1.6.0 was released with new "First Run Mode" https://twitter.com/elixirphoenix/status/1438222537301843970 – Removing Phoenix compiler config change can speed up compilation times when on up-to-date versions https://github.com/Qqwy/elixir-type_check – New library called TypeCheck https://blog.oestrich.org/2021/09/introducing-aino/ – Eric Osterich is experimenting with an alternative HTTP framework for Elixir https://github.com/oestrich/aino – The Aino project https://techcrunch.com/2021/09/14/logistics-startup-stord-raises-90m-in-kleiner-perkins-led-round-becomes-a-unicorn-and-acquires-another-company/ – Stord, an Elixir company, raised $90M and became a "unicorn" https://spawnfest.org/ – SpawnFest 2021 happened https://github.com/spawnfest/eflambe – SpawnFest submission - rapid generation of flamegraphs https://github.com/spawnfest/Sketch – SpawnFest submission - library for creating generative art with Elixir https://github.com/spawnfest/lotus – SpawnFest submission - Lotus is a Surface UI wrapper for UIKit https://github.com/spawnfest/Discovery – SpawnFest submission - Platform for hosting realtime, stateful servers with zero downtime deployment and horizontal scaling on Kubernetes https://github.com/spawnfest/crash – SpawnFest submission - Crash is a simple (distributed) docker-on-docker Continuous Integration system written in Elixir Do you have some Elixir news to share? Tell us at @ThinkingElixir or email at [email protected] Discussion Resources https://kaiwern.com/posts/2021/06/27/debugging-with-tracing-in-elixir-with-recon_trace/ https://www.naluri.life/ https://ferd.github.io/recon/recon.html https://hex.pm/packages/recon https://twitter.com/kw7oe/status/1409126300636254219 https://github.com/kw7oe/livebook-notebooks/blob/main/debugging-with-tracing-in-elixir-with-recon_trace.livemd – Livebook notebook showing tracing examples https://www.youtube.com/watch?v=sR9h3DZAA74 – Debugging Live Systems on the BEAM talk by Jeffery Utter https://www.youtube.com/watch?v=OR2Gc6_Le2U – Operable Erlang and Elixir talk by Fred Hebert (skip to 25:51, where he started talking about tracing) https://kaiwern.com/posts/2020/11/02/debugging-with-tracing-in-elixir/ – Tracing with dbg modules https://en.wikipedia.org/wiki/Tracing_(software) https://erlang.org/doc/man/dbg.html – dbg module in Erlang :dbg.fun2ms https://kubernetes.io/blog/2017/12/using-ebpf-in-kubernetes/ https://github.com/cilium/cilium https://opentelemetry.io/docs/erlang/ https://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html https://www.erlang-solutions.com/capabilities/wombatoam/ https://hex.pm/packages/recon_ex https://erlang-in-anger.com/ – Erlang in Anger (Chapter 9) https://www.thegreatcodeadventure.com/testing-genservers-with-erlang-trace/ – Testing GenServers with Erlang Trace http://blog.plataformatec.com.br/2016/04/how-to-trace-elixir-nodes-with-erlyberly/ – How to trace Elixir nodes with Erlyberly - Plataformatec Blog http://erlang.org/doc/man/seq_trace.html#sequential-tracing – Seq_trace http://erlang.org/doc/man/erlang.html#trace-3 – Erlang:trace https://github.com/andytill/erlyberly – Erlang tracing for the masses https://github.com/tatsuya6502/recon_ex – Elixir wrapper for Recon, tools to diagnose Erlang VM safely in production Guest Information https://twitter.com/kw7oe – on Twitter https://github.com/kw7oe/ – on Github https://kaiwern.com/ – Blog Find us online Message the show - @ThinkingElixir Email the show - [email protected] Mark Ericksen - @brainlid David Bernheisel - @bernheisel Cade Ward - @cadebward