Speeding up Koha with memcached

Anyone who has ever watched the MySQL query log while koha serves requests knows that koha makes A LOT of repetitive SQL queries for every request.  Most of these requests are for individual values in the systempreferences table.  There is also another large batch of queries for data from the language_descriptions table.  The queries are not that heavy by themselves, but there are way too many of them happening per request, and it’s for the same data every time.

In an attempt to minimize the number of repetitive queries, I tried to make koha use memcached.

First, let’s look at the problem we’re faced with:

All those queries are happening on every request.  The goal is to eliminate as many of those as we can using memcached to store the data once we retrieve it from MySQL.  That way, we don’t have to ask MySQL the same thing over and over again.

The patches that implement this can be found at:  http://arwen.metavore.com/~jsb/patches/memcached/ .

After those patches have been applied to the koha source, the number of queries made to MySQL are drastically reduced.

Unfortunately, preliminary benchmarks showed that this only improved the request rate by about 10%.  It was hoped that the speed increase would be more dramatic, but we’ll take what we can get.

Note that this set of patches has not (yet) been included into the main branch, but it may find itself there in the future.

The original announcement can be found at: http://lists.koha.org/pipermail/koha-patches/2009-February/002507.html.

Tags: , ,

3 Responses to “Speeding up Koha with memcached”

  1. Chris Says:

    I did some work on this last year, and a bit more this year.

    http://blog.bigballofwax.co.nz/2008/06/22/testing-koha-with-memcached/

    http://blog.bigballofwax.co.nz/2008/06/28/too-much-weights-not-enough-speed-work/

    http://blog.bigballofwax.co.nz/2008/07/18/a-geeky-way-to-spend-your-lunch-break/

    My caching branch in git is here
    http://git.workbuffer.org/cgi-bin/gitweb.cgi?p=koha.git;a=shortlog;h=refs/heads/cache

    http://git.workbuffer.org/cgi-bin/gitweb.cgi?p=koha.git;a=blob;f=lib/Koha/Cache.pm;hb=refs/heads/cache
    Might be of interest to you in the work you are doing.

  2. Clay Fouts Says:

    I think the real caching wins will come from storing MARC binary data blobs for searches. Anything that requires a lot of serialized processor time, like parsing XML data. The systempreference and language queries are annoying and numerous, but as you point out they’re also pretty lightweight.

    Threading and parallelizing these sorts of queries will also enable Koha to take better advantage of multi-core systems. That would be much more difficult to pull off, I imagine.

  3. chris Says:

    The language query might be small, but the code wrapped around it is heavy. By caching the resulting data structure as a serialised object. We can cut the execution time of every page on the opac by a factor of 10.
    As you point out, caching the results of a database query, when that query is small and fast isn’t much of a win. In some cases its a loss. But if you then do processing of the resulting data, say parsing xml, or building a complicated hash by doing lots of other small queries, then caching the resulting data structure is a big when.

Leave a Reply