Richard Bucker

Mojolicious plus Redis equals RestMQ-AEpl

Posted at — Nov 2, 2011

[Update 2012-05-09] Bruno, thank you for pointing out my link was broken.When I initially abandoned MojoX::Redis in order to use AnyEvent::Redis, it was because the MojoX version was not working properly or as expected. I dug deep into the code and doc and I finally managed to get things working. (documented here).It still bothered me that the AnyEvent::Redis was not working and I even more upset that the doc was so poor as to not include examples that demonstrated that it worked. (same general complaint that I have for the MojoX version.)This is what I have to say about that: I have the code working. It’s pretty simple. I have a project online at bitbucket (here) where I provide all the code. (sorry it’s not cleaner; for now)So here are the bits that are important. Include these libs. Many are probably not necessary, however, I installed them and it works.  Someone else can trim them.use EV;use JSON;use AnyEvent;use AnyEvent::Redis;use Mojolicious::Lite;Make a connection to your redis server. Unlike the MojoX implementation one does not need to overload anything (ioloop). Just get a connection instance. I performed this function in the main body of the Mojolicious::Lite module. Therefore when the module is loaded it will be executed immediately and not when the first transaction is executed.my $redis = AnyEvent::Redis->new( host => ‘127.0.0.1’, port => 6379, encoding => ‘utf8’, on_error => sub { warn @_ }, );And then within my GET function I do some work, however, there is one small block of code that needs to be asynchronously synchronized. (The update needs to be performed on the DB and the results need to be returned for future updates).post ‘/q/:queue’ => sub { my $self = shift; my $result = undef; my $queue = $self->param(‘queue’); my $value = $self->param(‘value’); if (! defined $queue) { $self->app->log->debug(‘queue was not in the URL (’.$queue.')'); $self->render_not_found; } else { my $uuid = undef; my $lkey = undef; my $q1 = $queue . $QUEUE_SUFFIX; my $q2 = $queue . $UUID_SUFFIX; $redis->all_cv(undef); $redis->all_cv->begin; $redis->incr($q2, sub{$uuid=shift;$redis->all_cv->end;}); $redis->all_cv->recv; $self->app->log->debug(‘the uuid is (’.($uuid||‘undefined’).')'); $lkey = $queue . ‘:’ . $uuid; $redis->sadd($QUEUESET, $q1); $redis->set($lkey, $value); $redis->lpush($q1, $lkey); $self->app->log->debug(‘the uuid q is (’.($q2||‘undefined’).')'); $self->render(text => ‘{ok, ' . $lkey . ‘}'); }};And now for the explanation. Very many of the examples look like this: $redis->all_cv->begin; $redis->incr($q2, sub{$uuid=shift;$redis->all_cv->end;}); $redis->all_cv->recv;And this is what the code is supposed to do: ->begin(); increment the counter do some redis work and do it in the background-ish In some cases the ->recv() will execute first and in others the ->end() will. If the ->recv() executes first then it will block (nicely) until the count = 0 or ->send() is called if -> end() executes first then the counter is decremented (in this case back to zero). And and artificial ->send() is executed. This ->send() has memory for any upcoming -recv() and will let the code pass right through onto the next statement.The trouble with this flow is not whether or not it works. It does.  But what happens on the second pass. So I called the URL once and it worked great. I blocked on the ->recv() and all was well.On the second pass through the code blew passed the ->recv() and it never waited for the results from the call to the DB. The reason for this is important. Since AnyEvent is using a “condvar” (condition variable)… and the variable is now set to True so that the ->recv() will pass… subsequent calls to ->recv() see the current value of the condvar. And since I only created the connection to the DB once, I was reusing the same instance every time and so the subsequent calls to the condvar worked as they should but not as expected.Luckily there is a work around:$redis->all_cv(undef);By adding this code I forced AnyEvent::Redis to create a new condvar instance for the upcoming call. This meant that the flag was set to it’s new-instance default value.All this junk because AE-condvar does not have a reset or clear method.