Central host high disk usage (/domino/logs) in Domino 3.x and below

To troubleshoot this problem, ssh to the central server and run the following commands to check the disk usage:

sudo -i
df -h

You will get output similar to this:

Filesystem              Size  Used Avail Use% Mounted on
...
/dev/nvme0n1p1           30G  6.3G   23G  22% /
...
/dev/mapper/vg0-domino   71G   25G   43G  37% /domino
/dev/mapper/vg0-logs    7.8G  7.3G   49M  99% /domino/logs
...

High disk usage is considered to be above 75%.

If you see high disk usage on the filesystem mounted on:

• /domino/logs - follow the rest of this article
• /domino - refer to troubleshooting /domino high disk usage
• anything else - refer to troubleshooting / high disk usage

Now lets diagnose the cause of high disk usage:

du /domino/logs/* --max-depth=1 -kh 2>/dev/null | sort -hr | head

You will get output similar to this:

5.9G /domino/logs/dispatcher/central
6.0G /domino/logs/dispatcher
16K  /domino/logs/lost+found
4.0K /domino/logs/upgrades

If most of disk space is consumed by dispatcher/central as in the example above, please follow the rest of this article. If it is consumed by something else, get in touch with Domino support.

Check the contents of dispatcher/central directory:

ls -lah /domino/logs/dispatcher/central

Sample output:

total 5.9G
drwxrwxr-x 2 domino domino 4.0K Apr  6 00:00 .
drwxr-x--- 3 domino domino 4.0K Apr  6 12:00 ..
-rw-r--r-- 1 domino domino 564M Apr  5 18:06 1616522817971-13
-rw-r--r-- 1 domino domino 5.4G Apr  6 12:39 1616522817971-14

To clean the disk we just remove the files from the dispatcher/central directory and then check its contents again:

rm /domino/logs/dispatcher/central/*
ls -lah /domino/logs/dispatcher/central

Sample output:

total 8.0K
drwxrwxr-x 2 domino domino 4.0K Apr  6 00:00 .
drwxr-x--- 3 domino domino 4.0K Apr  6 12:00 ..

Check the disk usage again:

df -h /domino/logs

Sample output:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg0-logs  7.8G  1.1G  5.4G  19% /domino/logs

To make sure this doesn't happen in the future, change configuration settings that control logging. On the Admin page, go to Advanced > Central Config. Set values as below:

common: com.cerebro.domino.centralLogging.mode = ''
role=dispatcher: com.cerebro.domino.centralLogging.mode = ''
common: com.cerebro.domino.centralLogging.submitter.targetHost = ''

In order for changes to take effect, you’ll need to restart both the Domino frontend and dispatcher servers. Frontend can be restarted from the Admin UI and there is no impact to the users as the instances are restarted one by one.

Dispatcher on the other hand has one instance only, so there will be impact of less than 5 minutes when it is restarted. The impact is that new runs / jobs / workspaces may not start and active runs / jobs / workspaces may fail to sync. To restart the dispatcher you need to run the following command from the central host.

docker restart domino-dispatcher