Protecting from a real redis vulnerability using read-only, distroless and security profiles with containers
Mon, May 9, 2022
TL;DR;
An example of how to protect from a real vulnerability using a combination of read-only, distroless and a security profile in docker. The security profile alone would have worked but following defense in depth, it is good practice to apply all three.
Longer version
The driver behind this blog post is two fold. The first is that currently I am spending a lot of time (and enjoying it) exploring the different methods available to lock down and secure linux instances and containers. The second is after reading a blog post from SysDig on “Compromising Read-Only Containers with Fileless Malware” By Nicholas Lang (you can find the article here https://sysdig.com/blog/containers-read-only-fileless-malware/). I really enjoyed the article, it introduced me to an interesting attack vector which was available to replicate easily following his instructions and the link to the vulnerability itself. It also introduced me to https://hub.docker.com/u/vulhub/ to get copies of container with a vulnerabilty so they can be examined.
The author demonstrated a small bit of the power of the SysDig tool Falco. I think this is really valuable being able to get this sort of telemetry out of a running system with the minimum of effort. What I want to show in this post is an alternative way to:
- See the issue
- Protect from the issue
- Block similar issues from even occurring
To do this I am using a combination of readonly containers, distroless containers and using a security profile tool apparmor. I could achieve this simply using the security profile tool alone, but defense in depth is a good practice.
1 - We first need a distroless redis container
There are various different examples of distroless redis out there but I wanted to do it myself to get more familiar with the process, following on from my first usage here https://andrewrea.co.uk/posts/distroless-nginx/. The other thing about my approach is that I avoid using wildcards completely so it is explicity what is being pulled into the distroless container image.
The above link shows how I go about finding the different dependencies for this post I am just going to post the resulting Dockerfile.
FROM vulhub/redis:5.0.7 as build
FROM gcr.io/distroless/base-debian11
COPY --from=build /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2
COPY --from=build /lib/x86_64-linux-gnu/libselinux.so.1 /lib/x86_64-linux-gnu/libselinux.so.1
COPY --from=build /lib/x86_64-linux-gnu/libpcre2-8.so.0.9.0 /lib/x86_64-linux-gnu/libpcre2-8.so.0
COPY --from=build /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libdl.so.2
COPY --from=build /lib/x86_64-linux-gnu/libssl.so.1.1 /lib/x86_64-linux-gnu/libssl.so.1.1
COPY --from=build /lib/x86_64-linux-gnu/libatomic.so.1 /lib/x86_64-linux-gnu/libatomic.so.1
COPY --from=build /lib/x86_64-linux-gnu/liblua5.1-cjson.so.0 /lib/x86_64-linux-gnu/liblua5.1-cjson.so.0
COPY --from=build /lib/x86_64-linux-gnu/liblua5.1-bitop.so.0 /lib/x86_64-linux-gnu/liblua5.1-bitop.so.0
COPY --from=build /lib/x86_64-linux-gnu/liblua5.1.so.0 /lib/x86_64-linux-gnu/liblua5.1.so.0
COPY --from=build /lib/x86_64-linux-gnu/libjemalloc.so.2 /lib/x86_64-linux-gnu/libjemalloc.so.2
COPY --from=build /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libm.so.6
COPY --from=build /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/librt.so.1
COPY --from=build /lib/x86_64-linux-gnu/libhiredis.so.0.14 /lib/x86_64-linux-gnu/libhiredis.so.0.14
COPY --from=build /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libpthread.so.0
COPY --from=build /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libpthread-2.28.so
COPY --from=build /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc.so.6
COPY --from=build /lib/x86_64-linux-gnu/libstdc++.so.6 /lib/x86_64-linux-gnu/libstdc++.so.6
COPY --from=build /lib/x86_64-linux-gnu/libgcc_s.so.1 /lib/x86_64-linux-gnu/libgcc_s.so.1
COPY --from=build /usr/lib/x86_64-linux-gnu/libdl-2.31.so /usr/lib/x86_64-linux-gnu/libdl-2.31.so
COPY --from=build /usr/bin/redis-server /usr/bin/redis-server
COPY --from=build /etc/redis /etc/redis
COPY --from=build /var/log/redis /var/log/redis
COPY --from=build /etc/passwd /etc/passwd
COPY --from=build /etc/group /etc/group
USER redis
ENTRYPOINT ["/usr/bin/redis-server", "--protected-mode", "no"]
Running docker with readonly and mapping a couple of volumes is inside a small run.sh file I made:
docker run -it \
--rm \
--read-only \
-p 6379:6379 \
--name redis-vuln \
--mount type=tmpfs,destination=/var/lib/redis,tmpfs-size=2m \
--mount type=tmpfs,destination=/run/redis,tmpfs-size=2m \
reaandrew/vuln-redis-distroless
2 - Check if the vulnerability from the blog post is an issue
Here I want to check and see if the vulnerability is still an issue using this distroless version of the vulnerable redis-server executable.
My first attempt failed with the following:
> redis-cli
127.0.0.1:6379> eval 'local io_l = package.loadlib("/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.popen("id","r"); local res = f:read("*a"); f:close(); return res' 0
(error) ERR Error running script (call to f_9cabd633a6136fa5cee1fc0747179be1c83cd700): @user_script:1: user_script:1: attempt to index local 'f' (a nil value)
I found I needed to also copy the sh executable too so the io.popen would work correctly. I added the following line to the Dockerfile:
COPY --from=build /usr/bin/sh /bin/sh
Which resulted in the following from the redis-cli output:
127.0.0.1:6379> eval 'local io_l = package.loadlib("/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.popen("id","r"); local res = f:read("*a"); f:close(); return res' 0
""
And the following in the redis-server output:
sh: 1: id: not found
At this point I can see one mitigation for this specific attack and those which rely on shell scripting is that the attack is pretty much neutralised when the required tools are not present on the target instance. In this case the example of the vulnerability used id, touch, cat and ls, mktemp. If we run all the examples from the SysDig blogpost we get similar output.
redis-cli output:
127.0.0.1:6379> eval 'local io_l = package.loadlib("/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.popen("id","r"); local res = f:read("*a"); f:close(); return res' 0
""
127.0.0.1:6379> eval 'local io_l = package.loadlib("/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.popen("ls /tmp", "r"); local res = f:read("*a"); f:close(); return res' 0
""
127.0.0.1:6379> eval 'local io_l = package.loadlib("/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.popen("TMPFILE=$(mktemp -p /dev/shm); echo derp > $TMPFILE; cat $TMPFILE", "r"); local res = f:read("*a"); f:close(); return res' 0
""
127.0.0.1:6379> eval 'local io_l = package.loadlib("/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.popen("touch /tmp/abc123", "r"); local res = f:read("*a"); f:close(); return res' 0
""
And the corresponding redis-server output:
sh: 1: id: not found
sh: 1: ls: not found
sh: 1: mktemp: not found
sh: 1: cannot create : Directory nonexistent
sh: 1: cat: not found
sh: 1: touch: not found
3 - There is still an issue!
The above code showed how shell scripting is rendered useless with the vulnerable image but it doesn’t rule out using another method to write to the shared memory (/dev/shm) which in this example I am going to use lua itself.
Running the following line in the redis-cli shows we can still create files!
127.0.0.1:6379> eval 'local io_l = package.loadlib("/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.open("/dev/shm/test.txt","w"); f:write("Hello, World!"); f:close(); f = io.open("/dev/shm/test.txt","r"); local res = f:read("*a"); f:close(); print(res); return res' 0
"Hello, World!"
4 - Add a security profile to mix to prevent writing to /dev/shm
For this I used App Armor. This is the first time I have delved into this tool and it wasn’t the easiest thing to grok but eventually I landed on a profile which demonstrated what I wanted.
One of the tools which is a massive help in understanding what you need for the profile is:
sudo aa-genprof /usr/bin/redis-server
It is an interactive shell which prompts you with options following realtime usage of the target executable.
On top of this I also following this page which an NGINX example profile https://docs.docker.com/engine/security/apparmor/.
Both of these helped me to understand more but I ended up writing the following profile to demonstrate the solution:
#include <tunables/global>
profile redis-vuln flags=(attach_disconnected,mediate_deleted) {
#include <abstractions/base>
network inet tcp,
network inet udp,
network inet icmp,
network inet6 stream,
deny network raw,
deny network packet,
file,
umount,
deny /bin/** wl,
deny /boot/** wl,
deny /dev/** wl,
deny /etc/** wl,
deny /home/** wl,
deny /lib/** wl,
deny /lib64/** wl,
deny /media/** wl,
deny /mnt/** wl,
deny /opt/** wl,
deny /proc/** wl,
deny /root/** wl,
deny /sbin/** wl,
deny /srv/** wl,
deny /tmp/** wl,
deny /sys/** wl,
deny /usr/** wl,
}
After adding this profile to the kernel I had to modify the run command too to include the security-opt of app armor and my new profile:
docker run -it \
--rm \
--read-only \
--security-opt "apparmor=redis-vuln" \
-p 6379:6379 \
--name redis-vuln \
--mount type=tmpfs,destination=/var/lib/redis,tmpfs-size=2m \
--mount type=tmpfs,destination=/run/redis,tmpfs-size=2m \
reaandrew/vuln-redis-distroless
Now when I run the container and try to execute the vulnerability by using lua itself to write the file I get the following output:
127.0.0.1:6379> eval 'local io_l = package.loadlib("/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.open("/dev/shm/test.txt","w"); f:write("Hello, World!"); f:close(); f = io.open("/dev/shm/test.txt","r"); local res = f:read("*a"); f:close(); print(res); return res' 0
(error) ERR Error running script (call to f_3d5e6f22f4a8c348100df77017c90ab5d287dcdd): @user_script:1: user_script:1: attempt to index local 'f' (a nil value)
The error is quite ambiguous but this is the result of applying that security profile to the container. To do one more final test I commented out the line which denies access to the /dev/* paths and tried again which produced the following output:
127.0.0.1:6379> eval 'local io_l = package.loadlib("/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.open("/dev/shm/test.txt","w"); f:write("Hello, World!"); f:close(); f = io.open("/dev/shm/test.txt","r"); local res = f:read("*a"); f:close(); print(res); return res' 0
"Hello, World!"
And the above test used the following profile which shows the line commented out for dev/**:
#include <tunables/global>
profile redis-vuln flags=(attach_disconnected,mediate_deleted) {
#include <abstractions/base>
network inet tcp,
network inet udp,
network inet icmp,
network inet6 stream,
deny network raw,
deny network packet,
file,
umount,
deny /bin/** wl,
deny /boot/** wl,
#deny /dev/** wl,
deny /etc/** wl,
deny /home/** wl,
deny /lib/** wl,
deny /lib64/** wl,
deny /media/** wl,
deny /mnt/** wl,
deny /opt/** wl,
deny /proc/** wl,
deny /root/** wl,
deny /sbin/** wl,
deny /srv/** wl,
deny /tmp/** wl,
deny /sys/** wl,
deny /usr/** wl,
}
Conclusion
Nothing is a pancea, using readonly is not protection for everything, neither is distroless nor is just using security profiles. This post shows that using them together, in this context, did solve the issue, it protected against similar vulnerabilities which rely on shell scripting and also those which use Lua to write files. It is not known whether it protects against anything else that isn’t mentioned here.
