EMx 069: Distributed Databases with Wiebe-Marten ("Marten") Wijnja

Episode Summary   In this week’s episode of Elixir Mix the panel follows up with Wiebe-Marten ("Marten") Wijnja about his talk at Elixir Conf EU, where he spoke about the distributed system his team was working on.    They start by discussing the eight fallacies of distributed computing that Marten talked about in talk. He lays out a couple of the fallacies and invites listeners to watch his talk for all eight. Marten explains that these fallacies most commonly happen to developers who are used to working with a single system. The panel discusses how to be mindful of these mistakes and how developers take for granted of how easy one system is to use.    Marten gives some tool recommendations to help with these fallacies. TLA+ is a small programming language that lets the developer describe their system and it will point out when something is wrong but it works purely on concepts. Erlang quick check implementation is also a tool that will help combat these problems. The last suggestion which was given by the panel is a library called comcast on github that will simulate poor network connections so the developer can see how the system runs on a poor connection.    Marten describes the byzantine problem. Two nodes or generals are trying to agree on something but communication keeps failing. The various outcomes are considered and Mark Ericksen gives an additional example of he and Josh Adams trying to connect to record a podcast, and how the miscommunication could change the outcome. This is a big problem that complicates using a distributed system.    The panel discussed CRDT’s and how they are a better way for nodes to sync up. Marten gives a very simple example of a CRDT as a counter. The panel discusses when to use CRDT’s and when not too. Marten explains what questions to ask before using CRDT’s.    Another way of solving the byzantine problem is by connecting the nodes. The panel discusses the tools they use to connect their nodes. Partisan is one tool, instead of connecting all nodes, each node connects to a specific number of nodes. That way if one node goes down the whole system doesn’t stop, while at the same time not, overwhelming the nodes. Libcluster, another tool, uses Kubernetes and has multiple strategies for connecting nodes so developers can choose the right one for their system.     The panel asks Marten about multicall and abcast. Marten explains that these tools help one node talk to all the other nodes in a cluster, and multicall will gather the results. Multicall also tells the developer which nodes failed to respond to the request. Mark shares an example of using these tools to effectively communicate between gen servers.    In Marten’s talk, he described four distributed databases. The panel asks Marten to talk about each one of them. The first one is mnesia. Marten talks about his first experience with Mnesia and how he thought it was amazing. He soon realized while it is still a great tool it also has its quirks.    He explains that each of these databases has its own quirks. Mnesia doesn't do conflict resolution, that along with a few other things the developer will need to build themselves. This can be a good and bad thing because developers can customize the database to their needs but it’s not ready out of the box. Mark explains the use cases mnesia is good for and even references the mnesia documentation.     Cassandra is the next database Marten describes. Cassandra is the database discord uses. Cassandra does not let developers control their own conflict resolution. It always uses the latest time-stamp and with nodes that can be confusing.    Couchdb is another database they discuss. Again, couchdb is also not made to deal with conflicts. It will either solve them randomly or the developer can opt into resolving it themselves. The panel discusses times when this is useful, such as when connectivity is intermittent.    Riak is the final database and the one Marten’s...