How to resize the datadir for mongoDB or other kubernetes volumes

Version:

Not Domino version specific

Issue:

You may find through a Domino admin-toolkit run or other troubleshooting that one of the kubernetes micro services has nearly filled it's disk.  The example in this article is for the mongodb datadir volume, but it is applicable to all domino platform services volumes in kubernetes.  

Example output of admin-toolkit test_platform_pvcs test showing nearly full datadir-mongodb-replicaset volumes.

domino_admin_toolkit/checks/test_platform_pvcs.py:52: in test_platform_PVCs
assert all(
E AssertionError: Some domino-platform namespace PVCs are above 80% disk usage.
E assert False
E + where False = all(<generator object test_platform_PVCs.<locals>.<genexpr> at 0x7f90f31cd700>)
-------------------------------Captured log call-------------------------------- 
INFO root:test_platform_pvcs.py:18 Checking PVC usage of pods in domino-platform namespace...
INFO root:test_platform_pvcs.py:50 
| PVC NAME                            | % DISK USAGE | % INODE USAGE   |
|-------------------------------------+--------------+-----------------|
| clone-dir-repoman-0                 | 0.14         | 0.00            |
| data-mlflow-postgresql-0            | 1.47         | 0.03            |
| data-mlflow-postgresql-slave-0      | 1.27         | 0.03            |
| data-postgresql-0                   | 4.69         | 0.06            |
| data-postgresql-slave-0             | 4.67         | 0.06            |
| data-rabbitmq-ha-311-0              | 0.41         | 0.00            |
| data-zookeeper-0                    | 0.80         | 0.01            |
| datadir-mongodb-replicaset-0        | 96.27        | 0.05            |
| elasticsearch-elasticsearch-tls-0   | 0.42         | 0.01            |
| grafana-grafana-0                   | 0.45         | 0.00            |
| clone-dir-repoman-1                 | 0.15         | 0.00            |
| data-dmm-redis-ha-server-0          | 17.22        | 0.00            |
| data-rabbitmq-ha-311-1              | 0.42         | 0.00            |
| data-redis-ha-server-0              | 5.90         | 0.00            |
| data-zookeeper-1                    | 0.50         | 0.00            |
| datadir-mongodb-primary-0           | 1.67         | 0.00            |
| datadir-mongodb-replicaset-1        | 96.17        | 0.05            |
| datadir-mongodb-secondary-0         | 1.68         | 0.00            |
| elasticsearch-elasticsearch-tls-1   | 0.45         | 0.01            |
| storage-volume-prometheus-server-0  | 70.91        | 0.00            |
| data-rabbitmq-ha-311-2              | 0.43         | 0.00            |
| data-zookeeper-2                    | 0.50         | 0.00            |
| datadir-mongodb-replicaset-2        | 96.11        | 0.05            |
| elasticsearch-elasticsearch-tls-2   | 0.39         | 0.01            |
| git-storage-git-0                   | 2.65         | 1.39            |

Resolution:

Resizing a PV involves both editing the size of the volume’s PVC and restarting the pods which are using it.  This can be done live and will not impact the data on the existing volume.  Note that although the change is non destructive, a restart of the pod is required.

This is an example of resizing the mongo pod in AWS.

1. Find the relevant PVC to be resized with kubectl get pvc --all-namespaces | grep mongodb-replicaset

$ kubectl get pvc --all-namespaces | grep mongodb-replicaset
domino-platform datadir-mongodb-replicaset-0 Bound pvc-996e6422-2b1a-4fe4-bea2-a2c9b9bd286f 10Gi RWO dominodisk 13d

2. Describe the pvc for reference with kubectl describe pvc -n <platform namespace> <pvc name>

$ kubectl describe pvc -n domino-platform datadir-mongodb-replicaset-0
Name: datadir-mongodb-replicaset-0
Namespace: domino-platform
StorageClass: dominodisk
Status: Bound
Volume: pvc-996e6422-2b1a-4fe4-bea2-a2c9b9bd286f
Labels: app.kubernetes.io/instance=mongodb-replicaset
app.kubernetes.io/name=mongodb-replicaset
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
volume.kubernetes.io/selected-node: ip-10-0-40-173.us-west-2.compute.internal
volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: mongodb-replicaset-0
Events: <none>
(base) ➜ ~ kubectl describe pvc -n domino-platform datadir-mongodb-replicaset-0
Name: datadir-mongodb-replicaset-0
Namespace: domino-platform
StorageClass: dominodisk
Status: Bound
Volume: pvc-996e6422-2b1a-4fe4-bea2-a2c9b9bd286f
Labels: app.kubernetes.io/instance=mongodb-replicaset
app.kubernetes.io/name=mongodb-replicaset
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
volume.kubernetes.io/selected-node: ip-10-0-40-173.us-west-2.compute.internal
volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: mongodb-replicaset-0
Events: <none>

Note the Storage class, in this case dominodisk.

3. Check that the storage class from the PVC description can be resized with kubectl get sc

$ kubectl get sc
NAME             PROVISIONER           RECLAIMPOLICY VOLUMEBINDINGMODE    ALLOWVOLUMEEXPANSION AGE
dominodisk       ebs.csi.aws.com       Delete        WaitForFirstConsumer false                13d
dominoshared     efs.csi.aws.com       Delete        Immediate            false                13d
gp2 (default)    kubernetes.io/aws-ebs Delete        WaitForFirstConsumer false                13d

In this case it can’t, so edit the storage class to allow volume expansion withkubectl edit sc dominodisk and add allowVolumeExpansion: true

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
creationTimestamp: "2023-07-05T13:55:53Z"
name: dominodisk
resourceVersion: "2571"
uid: d8589489-d300-4634-8669-cece9d62544a
parameters:
encrypted: "true"
kmsKeyId: <redacted>
type: gp3
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

$ kubectl get sc
NAME             PROVISIONER           RECLAIMPOLICY VOLUMEBINDINGMODE    ALLOWVOLUMEEXPANSION AGE
dominodisk       ebs.csi.aws.com       Delete        WaitForFirstConsumer true                 13d
dominoshared     efs.csi.aws.com       Delete        Immediate            false                13d
gp2 (default)    kubernetes.io/aws-ebs Delete        WaitForFirstConsumer false                13d

4. Edit the pvc with kubectl edit pv -n <platform namespace> <pvc name>

$ kubectl edit pvc -n domino-platform datadir-mongodb-replicaset-0

Change the resource request block in the spec

resources:
  requests:
    storage: 10Gi

resources:
  requests:
    storage: 50Gi

5. Recheck the size of the pv/pvc. You may need to restart pods to pick up the change. You’ll see a message explaining that in the pvc description.

References/Internal Records:

https://dominodatalab.atlassian.net/wiki/spaces/ENG/pages/1315865073/Resizing+a+persistent+volume