Richard Bucker

MongoDB, Partitioning, Sharding, Benchmarks, and application design

Posted at — Feb 7, 2015

10 years ago I ran some benchmarks for an OLTP application I was building. One of the driving design requirements was response time.  The aggregate response time had to be shorter than 250ms. The initial implementation relied on 100% of the work being performed in the application which meant that the network was chatty as the application retrieved data from over 100 tables. The multiple network roundtrips, latency, and transaction lock escalation, not only increased transaction time but also limited the number of concurrent transactions.The second implementation cached data in the app server. That design also failed. The third implementation, which was elected to production, divided the work between the application server and the database. The application managed the protocol impedance and transaction validation between the payment network and the database. The business logic was implemented in the database  as series of runtime configurable stored procedures. While using T-SQL for every transaction and in such a dynamic structure was not optimum it was more performant than the alternatives.Once the DBAs exhausted all other options from primary keys, covered indexes, better extents, replication, sharding, partitioning, high availability/load balancing, re-indexing, replication … we started to evaluate other DB technologies and hardware in order to scale in hardware instead of software. This option never received any traction because all the all of the business intelligence was written in T-SQL and moving to Oracle or other DB meant nearly a complete rewrite…. and version 2 … even with an army of contractors will never be completed. (mythical man month)If we had implemented the fourth option (still on the drawing board at the time) we would have had a window of opportunity to get on the new hardware.The application was written in Java and so implementing the business logic in Java in a separate micro service that would run on the DB would have been the best of both worlds. In today’s container world this type of service is referred to as a sidekick or ambassador. This application was going to act as a shim to the DB and perform all the heavy lifting/computation of the application.In today’s application design we are faced with a number of challenges. It’s no longer as simple as installing an application on the same core as the database. Now there are CI/CD considerations as well as production discipline when actually pressing the GO-LIVE button. Version management over a cluster of master-master database servers that follow the CAP theorem while trying to update the on-server application … there are just so many moving parts that continuous operation is a dance between the OPS and the DEV teams.