Scripting Redis using Lua
Redis is an awesome data structures server. It provides access to mutable data structures via a set of commands.
Data structures store tied to simple a TCP server, how cool is that!!
While Redis is fairly easy to use via “a set of commands”, issues start arising when we try to implement complex operations that comprise multiple Redis commands.
For example, let's consider an operation that performs the following steps:
- Pushes a list of integers to a Redis list
- Update the value of key
sumthat stores the sum of all integers in list
Let’s try to implement this; I’m using redis-py, a Python Redis client.
import redis r = redis.Redis(host='localhost', port=6379, db=0) new_list = [2,1,2] # Pushing the new_list to the redis list r.lpush('mylist', *new_list) # Calculating the sum of the new_list sum_of_new_list = sum(new_list) # ... here we can have multiple application-level operations # ... before calling the next Redis command # Incrementing the valuof of the redis key "sum" by the sum_of_new_list r.incrby('sum', sum_of_new_list)
While at first glance the above implementation looks fine, but it is vulnerable to the following problems:
- The atomicity. Ah, a good old problem. Redis commands
r.incrby('sum', sum_of_new_list)should be atomic, i.e either both of them should execute or none of them should execute. Atomicity is not guaranteed in the above implementation.
- The application (python script in this case) is handling data-level logic.
Using Lua script
This is where Lua script comes handy. Redis includes server-side scripting with the Lua programming language. This lets you perform a variety of operations inside Redis, which can both simplify your code and increase performance.
EVAL command is used to evaluate scripts using the Lua interpreter built into Redis.
The first argument of EVAL is a Lua 5.1 script. The script does not need to define a Lua function (and should not). It is just a Lua program that will run in the context of the Redis server.
The second argument of EVAL is the number of arguments that follows the script (starting from the third argument) that represent Redis key names. The arguments can be accessed by Lua using the
KEYS global variable in the form of a one-based array (so
All the additional arguments should not represent key names and can be accessed by Lua using the
ARGV global variable, very similar to what happens with keys (so
We can call Redis commands from a Lua script using the function
Let’s try to write a command using the above information:
Redis return values are converted into Lua data types when Lua calls a Redis command using
call(). Similarly, Lua data types are converted into the Redis protocol when calling a Redis command and when a Lua script returns a value so that scripts can control what EVAL will return to the client.
Redis uses the same Lua interpreter to run all the commands. Also, Redis guarantees that a script is executed in an atomic way: no other script or Redis command will be executed while a script is being executed.
Loading Lua scripts into Redis
Instead of providing Lua script every time to Redis, we can load the Lua script once using
SCRIPT LOAD command and then subsequently run it using the
The first and the only argument of
SCRIPT LOAD is a Lua 5.1 script. The script does not need to define a Lua function (and should not). It is just a Lua program that will run in the context of the Redis server.
SCRIPT LOAD command, a SHA1 digest of the script will be returned. We can call the script by providing SHA1 digest to the EVALSHA command along with needed
Let’s try to write a command using the above information.
Powerful, isn’t it? The script is guaranteed to stay in the script cache forever (unless
SCRIPT FLUSH is called).
Now that we have understood how to use Lua scripts with Redis, let’s try to implement the same operation using Lua Scripts. I’m using redis-py, a python redis client.
First off let's create the
script.lua file that will hold the Lua Script.
-- script.lua local sum_of_new_list = 0 -- Pushing the new_list to the redis list -- Also calculating the sum of the new_list for key,value in ipairs(ARGV) do sum_of_new_list = sum_of_new_list + tonumber(value) redis.call('LPUSH', KEYS, value) end -- Incrementing the value of the redis key "sum" by the sum_of_new_list local total_sum = redis.call('INCRBY', KEYS, sum_of_new_list) -- Finally returning the total_sum return total_sum
Now let’s implement
main.py where we will import
script.lua and run it on the Lua interpreter of Redis.
# main.py import redis r = redis.Redis(host='localhost', port=6379, db=0) # Reading script.lua file into a string f = open("script.lua", "r") lua = f.read() new_list = [2,1,2] # Registering the script on the redis server operation = r.register_script(lua) # Executing the script op_return = operation(keys=['mylist', 'sum'], args=new_list) print(op_return)
Awesome!, we have successfully implemented the needed operation using Lua script. The application-level logic (
main.py) and data-level logic (
script.lua) are separated. Our Lua script is Atomic and ensures that keys
sum are always in sync.
Well, that’s it for now! Peace 🍻