Increasing the memory for Redis on Domino deployments

Issue: If we notice logs are not appearing in time/missing or loading logs is rather slow, one of the suggestions is to check if the Redis service has enough memory to operate. 

Troubleshooting:  There are 2 methods to identify redis memory usage. First method is to login to the redis container and directly connect to redis. so we can request the stats for the service. 

To do this you will first need to get the redis password. This is stored in the redis-ha secret in kubernetes and is base64 encoded.  Use the following command to get the authentication: 

kubectl get secret redis-ha -n domino-platform -o json|jq -r .data | awk '{print $2}' | sed -n 2p | cut -f 2 -d "\"" | base64 -d

 Once we have the password we can then execute to redis container and run our info command:

$ kubectl exec -it -n domino-platform redis-ha-server-0 -c redis -- sh

This should present us with a shell. Next we will use netcat to connect to the redis service and authenticate: 

/data $ nc localhost 6379

auth <REDACTED>

+OK

Once authenticate we will now request the stats with command "info" . We are interested in the used_memory (used_memory_human), maxmemory (maxmemory_human) and  evicted_keys .

info memory

...TRUNCATED
# Memory
used_memory:1072964592
used_memory_human:1023.26M
...TRUNCATED
maxmemory:1073741824
maxmemory_human:1.00G
...TRUNCATED
evicted_keys:9060

As this output suggest we are fully using the maxmemory which then causes old keys to be evicted before any new data is inserted and thus causing latency. 

The second method to identify the usage would be to use the grafana dashboard. 

For how to access grafana please see this article here .  

On the Prometheus graph you will find redis statistics. Here we can see if we have used memory hitting maxmemory constantly and also if we have a high number of evictions. 

Solution: Once we can identify we need more memory for redis we can increase this quite easily. Naturally the pod for redis will have certain amount of memory request and limit set, however the service itself will try to use only whatever is set in the redis configmap .

To edit the configmap use: 

kubectl edit cm redis-ha-configmap -n domino-platform

Under the redis.conf you should find maxmemory setting. Change this to maximum of 45% of the limit for the redis pod:

...TRUNCATED
 redis.conf: |
dir "/data"
maxmemory 1gb
maxmemory-policy allkeys-lru
min-slaves-max-lag 5
...TRUNCATED