Richard Bucker

Benchmarks - Hello World

Posted at — Sep 16, 2011

If you’ve ever worked for a credit card company around the winter holiday season then you appreciate, as I do, benchmarking your application; early and often. Usually, however, this sort of benchmarking is meant to identify slow parts of the system as well as potential bugs that might have mad it through development. In this case testing usually centers around a very small set of transactions. Transaction types with the highest volume.The “hello world” benchmark, on the other hand, is the complete opposite. It tells you some interesting things about your application but only the smallest bit.Consider the hello_world that the different web frameworks are now benchmarking against. Even the slowest webservers are functioning under 100ms and most of that time is socket negotiation. For example, in tornadoweb, only executes about 35 lines of code in order to get the desired response. Since TW is single threaded all of the speed and performance is pushed to the python implementation and the underlying libs. But the “work” is non-existant and is not being tested at all. At that point we’d be talking about the implementation language and it’s performance within the context of the transaction. And that has nothing to do with benchmarking a web framework.So I’m of the opinion that benchmarking individual webservers is junk. The results are meaningless. How do we get better numbers?Well, you can use the hello world benchmark to start, however, you need to include similar benchmarks of other webservers that use the same strategy. For example; a hello world benchmark for Mojolicious should be used in conjunction with a benchmark of Tornadoweb. To my knowledge they share enough similarities in their infrastructure/implementation that you might be able to extract the actual performance of the webserver away from the dependencies. (both webservers are single threaded and use [prefork] multi-process in order to scale.)Of course I could be totally wrong about tornadoweb and mojolicious, however, the ideal design for this type of framework is a very lightweight webserver that forwards requests to an MQ that a worker processes. (I’m not sure if there is a pattern name here.)