Richard Bucker

Hot code replacement - Erlang, Elixir and so on

Posted at — Feb 8, 2015

I have to make this quick. Hot code replacement is interest, fun, and very complicated. It’s so complicated it’s not worth the effort and in the few use-cases where it is a requirement might and should be in the far minority.

The first problem with hot code replacement is scope and idempotence. Hot plugging a small module that is only part of the transaction cannot be reproduced. Creating monolithic modules might as well be implemented as blue/green.

Second, when hot replacing code there is challenge to migrate state from the current module to the new one. Since a module can have more than one process maintaining state moving the state from the current to the new and preventing forked transactions… it’s all too difficult. (it is possible to store state in a cache like redis but that breaks a few idioms like the benefits of immutability)

Next, while hot code replacement is considered a rockstar level achievement it’s complicated and while it can be made to be reliable and reproducible there is a challenge that the full system might not startup from scratch should it be required. At some point you have to be confident that a restart is possible.

One of the last blockers on the list is parity with the VCS and CI. The continuous integration system is going to build the monolithic application while the hot plug is just a fraction and most CI systems do not address the module as the most atomic component. While partitioning the application along module lines is possible it does seem to be an anti pattern as google seems to like monolithic VCS.

Elixir might be a reasonable gateway from ruby to erlang and while hot code replacements are fun and interesting they create a serious number of operational challenges for the audit sensitive and long transaction applications.

UPDATE: I just started reading this paper. While it starts to talk about DSU, dynamic software updates, I find myself considering compiler versions. The question being; is there parity between the compilers or does the DSU use the compiler of the current version.

Which brings me to another question… what happens when the erlang VM needs to be updated? At that point you’re back in the blue/green deploy pipeline. Blue/green is the best way to migrate an application but the transition has to be managed.