Richard Bucker

Beanstalkd client/worker sample code

Posted at — Oct 19, 2011

The following code is not tested and there is no defensive coding at all. Those are activities for the reader for now. In the next few days I’ll implement the same code in perl and C. ┬áThe advantage of this strategy is that some A/B testing and monitoring will let you know which modules need to be rewritten for performance etc… The other side effect is that you can be language agnostic. (I might just try Lua too)One of the other great side effects of this design is that since there are multiple small applications that are distributed they are easier to debug and extend. (think of all of the advantages of microkernels.)The Client:# load the required librariesimport beanstalkc# make a connection to the beanstalk brokerbeanstalk = beanstalkc.Connection(host=‘localhost’, port=14711)# select the tube that is going to forward the message to your “worker”# you can have multiple workers listening on the same tube or different tubes# or a combination.beanstalk.use(‘msg_for_worker’)# this is my message’s transaction id, it is also the key used to locate the# data in the cache and it is the name of the response tube.msg_id = str(uuid.uuid1())# HERE store the full transaction in the redis DB and use the msg_id as the key# start watching the response tubebeanstalk.watch(msg_id)# send the message to the workerbeanstalk.put(msg_id)# wait for a response (timeout is in seconds)job = beanstalk.reserve(timeout=15)print job.body# stop watching the response tubebeanstalk.ignore(msg_id)# DONEThe Worker:# load the required librariesimport beanstalkc# make a connection to the beanstalk brokerbeanstalk = beanstalkc.Connection(host=‘localhost’, port=14711)# setup a watch for incoming messages over the well known tubebeanstalk.watch(‘msg_for_worker’)while True: # wait for a request (timeout is in seconds) job = beanstalk.reserve(timeout=15) if job: msg_id = job.body # HERE we retrieve the message/transaction body from the redis DB. and do our work. # open a tube to the client response tube beanstalk.use(msg_id) # send the response message to the client beanstalk.put(msg_id)