Richard Bucker

Eventually Consistent Storage Will Save Mankind

Posted at — Oct 3, 2011

I recently read a tweet from @justinsheehy, the very public face of Riak @ He wrote:Paraphrasing @GeorgeReese: to be protected from failure, put as much of your data in an eventually-consistent system as possible.In response, and without thinking too deeply, I asked the question:@justinsheehy @georgereese good point so why not a flat file and import later? Why all the extra cycles/rotations to write to any type DB?And then @georgereese and I started to converse at 140-byte intervals until he sent me a link to this article: Eventual consistency - Wikipedia, the free encyclopedia.  At that moment I realized that my original question was really more of a statement; in it’s absolute simplest form; the wiki definition for eventual consistency can be applied to a flatfile on a DOS-based computer so long as you take backups and restore them on another computer… at some point in time.That said, I think; and I could be wrong, Sheehy and Reese were probably talking about Riak which has a lot more moving parts in it than -say… a zipped-flatfile and rsync… and there is plenty of computer science reference material that discusses BLOC (bugs per line of code).I’m currently designing and implementing a credit card payment gateway. It’s not overly complicated, however, the most interesting piece of this implementation is the use of Redis as the storage engine. While Redis stores everything in memory, I have enabled the feature/function that saves the data to disk; so while I have not enabled replication… this “system” can be described as eventually consistent.Eventually Consistent is so much more interesting when applied generally and globally across systems instead of narrowly defined applications.In the interest of full disclosure; I recently interviewed with @justinsheehy for a position on the Riak project. While I recognized that I had not performed well after only having a few hours sleep thanks to my pair of newborns I have not yet received any formal feedback. This conversation and post are meant to be informative and with the sincere hope that one day basho might offer me a position.