Scripting Redis using Lua

Redis is an awesome data structures server. It provides access to mutable data structures via a set of commands.

Data structures store tied to simple a TCP server, how cool is that!!

While Redis is fairly easy to use via “a set of commands”, issues start arising when we try to implement complex operations that comprise multiple Redis commands.

For example, let's consider an operation that performs the following steps:

1. Pushes a list of integers to a Redis list mylist.
2. Update the value of key sum that stores the sum of all integers in list mylist.

Let’s try to implement this; I’m using redis-py, a Python Redis client.

import redis
r = redis.Redis(host='localhost', port=6379, db=0)

new_list = [2,1,2]

# Pushing the new_list to the redis list
r.lpush('mylist', *new_list)

# Calculating the sum of the new_list
sum_of_new_list = sum(new_list)

# ... here we can have multiple application-level operations 
# ... before calling the next Redis command

# Incrementing the valuof of the redis key "sum" by the sum_of_new_list
r.incrby('sum', sum_of_new_list)

While at first glance the above implementation looks fine, but it is vulnerable to the following problems:

1. The atomicity. Ah, a good old problem. Redis commands r.lpush('mylist', *new_list) and r.incrby('sum', sum_of_new_list) should be atomic, i.e either both of them should execute or none of them should execute. Atomicity is not guaranteed in the above implementation.
2. The application (python script in this case) is handling data-level logic.

Using Lua script

This is where Lua script comes handy. Redis includes server-side scripting with the Lua programming language. This lets you perform a variety of operations inside Redis, which can both simplify your code and increase performance.

EVAL command is used to evaluate scripts using the Lua interpreter built into Redis.

The first argument of EVAL is a Lua 5.1 script. The script does not need to define a Lua function (and should not). It is just a Lua program that will run in the context of the Redis server.

The second argument of EVAL is the number of arguments that follows the script (starting from the third argument) that represent Redis key names. The arguments can be accessed by Lua using the KEYS global variable in the form of a one-based array (so KEYS[1], KEYS[2], ...).

All the additional arguments should not represent key names and can be accessed by Lua using the ARGV global variable, very similar to what happens with keys (so ARGV[1], ARGV[2], ...).

We can call Redis commands from a Lua script using the function redis.call().

Let’s try to write a command using the above information:

Redis return values are converted into Lua data types when Lua calls a Redis command using call(). Similarly, Lua data types are converted into the Redis protocol when calling a Redis command and when a Lua script returns a value so that scripts can control what EVAL will return to the client.

Redis uses the same Lua interpreter to run all the commands. Also, Redis guarantees that a script is executed in an atomic way: no other script or Redis command will be executed while a script is being executed.

Loading Lua scripts into Redis

Instead of providing Lua script every time to Redis, we can load the Lua script once using SCRIPT LOAD command and then subsequently run it using the EVALSHA command.

The first and the only argument of SCRIPT LOAD is a Lua 5.1 script. The script does not need to define a Lua function (and should not). It is just a Lua program that will run in the context of the Redis server.

After executing SCRIPT LOAD command, a SHA1 digest of the script will be returned. We can call the script by providing SHA1 digest to the EVALSHA command along with needed KEYS and ARGV

Let’s try to write a command using the above information.

Powerful, isn’t it? The script is guaranteed to stay in the script cache forever (unless SCRIPT FLUSH is called).

Now that we have understood how to use Lua scripts with Redis, let’s try to implement the same operation using Lua Scripts. I’m using redis-py, a python redis client.

First off let's create the script.lua file that will hold the Lua Script.

-- script.lua
local sum_of_new_list = 0

-- Pushing the new_list to the redis list 
-- Also calculating the sum of the new_list
for key,value in ipairs(ARGV) do
    sum_of_new_list = sum_of_new_list + tonumber(value)
    redis.call('LPUSH', KEYS[1], value)
end

-- Incrementing the value of the redis key "sum" by the sum_of_new_list
local total_sum = redis.call('INCRBY', KEYS[2], sum_of_new_list)

-- Finally returning the total_sum
return total_sum

Now let’s implement main.py where we will import script.lua and run it on the Lua interpreter of Redis.

# main.py
import redis
r = redis.Redis(host='localhost', port=6379, db=0)

# Reading script.lua file into a string
f = open("script.lua", "r")
lua = f.read()

new_list = [2,1,2]

# Registering the script on the redis server
operation = r.register_script(lua)

# Executing the script
op_return = operation(keys=['mylist', 'sum'], args=new_list)

print(op_return)

Awesome!, we have successfully implemented the needed operation using Lua script. The application-level logic (main.py) and data-level logic (script.lua) are separated. Our Lua script is Atomic and ensures that keys mylist and sum are always in sync.

Well, that’s it for now! Peace 🍻