Out of space error while writing to /var/opt/git in Nucleus FrontEnd logs

Version:

Domino 4.x & 5.x+

Issue:

If you are seeing a widespread outage on your deployment with multiple users unable to create projects, sync workspaces, view the files dashboard in a project or any other operations failures that are indicative of inability to access the Domino File System that backs projects, then check the Nucleus Frontend logs for 'out of space errors'.  

Note that individual users with Out of Space errors in their executions may just need to review the following article: 

https://tickets.dominodatalab.com/hc/en-us/articles/360060906852-Why-did-my-execution-fail-with-the-error-No-space-left-on-device-

This article addresses a more systemwide failure with the following or similar types of errors in the Nucleus Frontend logs:

2022-11-28T18:39:42.113384993Z Caused by: domino.common.util.http.HttpException: Code: 500, Error: Script error, headers: Map(Transfer-Encoding -> List(chunked), Server -> List(Apache/2.4.38 (Debian)), Connection -> List(close), Content-Type -> List(text/html), Date -> List(Mon, 28 Nov 2022 18:39:42 GMT)), Body: Mon Nov 28 18:39:42 UTC 2022: Creating repository /var/opt/git/projectrepos/6384/6384ffee9eb31350d1764551.git 2022-11-28T18:39:42.113391897Z mkdir: cannot create directory '/var/opt/git/projectrepos/6384/6384ffee9eb31350d1764551.git': No space left on device

Root Cause:

This can be caused by the Git-0 pod actually filling the /var/opt/git filesystem or running out of inodes.  To verify the issue, exec in to the Git-0 pod and verify the status of the /var/opt/git mount point:

df
& 
df -i /var/opt/git

If either indicates the filesystem is full or out of inodes then the PVC for the git-0 pod will need to be resized.

Resolution:

To resize the PVC, follow the this process:

1. Find the relevant PVC to be resized withkubectl get pvc --all-namespaces

$ kubectl get pvc --all-namespaces | grep git
domino-platform git-storage-git-0 Bound pvc-9c338304-d622-4bcc-ae2e-9fad62996894 20Gi RWO dominodisk 14

2. Describe the PVC for reference with kubectl describe pvc -n <platform namespace> <pvc name>

$ kubectl describe pvc -n domino-platform git-storage-git-0
Name: git-storage-git-0
Namespace: domino-platform
StorageClass: dominodisk
Status: Bound
Volume: pvc-9c338304-d622-4bcc-ae2e-9fad62996894
Labels: app.kubernetes.io/instance=git
app.kubernetes.io/name=git
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
volume.kubernetes.io/selected-node: ip-10-0-88-171.us-west-2.compute.internal
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 20Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: git-0
Events: <none>

3. Check that the storage size can be resized, ALLOWVOLUMEEXPANSION is true:

$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
dominodisk ebs.csi.aws.com Delete WaitForFirstConsumer false 14d
dominoshared efs.csi.aws.com Delete Immediate true 14d

If AllowVolumeExpansion is false:

kubectl edit sc dominodisk

and add:

allowVolumeExpansion: true

  $ kubectl get sc

  NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
  dominodisk ebs.csi.aws.com Delete WaitForFirstConsumer true 14d
  dominoshared efs.csi.aws.com Delete Immediate true 14d

4. Edit the pvc with kubectl edit pvc -n <platform namespace> <pvc name>

Change the resource request block in the spec:

resources: 
  requests: 
    storage: 20Gi

resources: 
  requests: 
    storage: 50Gi