Richard Bucker

A new approach : HamsterDB

Posted at — Jun 23, 2011

Revisiting my favorite subject again, credit card processing, the hamsterDB’s description on the NoSQL website triggered an alarm.hamsterDB - (embedded solution) ACID Compliance, Lock Free Architecture (transactions fail on conflict rather than block), Transaction logging & fail recovery (redo logs), In Memory support – can be used as a non-persisted cache, B+ Trees – supportedThe key words being “lock free”. In any typical CC issuing system you can expect to see transaction times from 50 to 500ms depending on the amount of work the authorization system has to perform, DB latency and locking.Typical transaction workflow looks like some code that just tries to get some data from the DB, do some work, get some more data from the DB and do some more work. And while performing I/O with the DB you always have to be ready for a failure. Typical failures are deadlocks, consistency because another process updated a record and so on. And when you think about the breadcrumbs and trying to recover from these failures it simply makes the code more complex.update account set balance=balance-10.00, version=version+1 where ccnumber=? and version=11;With the hamster approach you can and should get all the data that you need from the DB upfront at the beginning of the transaction. Keep in mind that in some use-cases this data could be prohibitively large so it’s best to completely understand the scope. Then do the workflow as you would normally… leaving you with a set of DB updates/inserts that need to be executed. So execute them.Now if anything goes wrong you have choices. 1) retry the DB changes from the last step; 2) retry the workflow from step two; 3) retry the entire transaction. It simply depends on the nature of the write failure and what you determined was the best recovery.And here’s why this is better. Given the performance profile for an authorization (50-500ms) and the timeout that is permissible by the association (10-45seconds depending in the trantype) you can retry this transaction internally almost any number of times in order to get a positive response… providing the error was internal and not hardware, network ete related.One other thing that did not escape my eye. Kevin Smith (formerly from Riak) is the erlang client maintainer. Optional encryption(great for PCI)On the downside there is no replication, sharding, python, perl, or traditional C. However, the approach would be interesting for other platforms… almost there.