Symptom: When trying to access the prediction data logs, the fluent bit logs are saying
"Permission denied" on the scratch path, ie: /domino/shared/scratch/model-621cfa8b4d88bf0d96ddf421-5c7c56857b-kpft9.0b72e5bf-9866-429c-81a5-7e11eaf6be4d.621cfa8b4d88bf0d96ddf421.20220228T180700UTC.log
This happens sometimes with upgraded Domino instances, where the
scratch directory in the shared store was created previously. It could get created with write permissions only for the root user. The fluentd containers in the model-api pods do not run as root, and hence cannot write to it.
The fix is to chmod the directory to allow writes by *all*. The easiest way to fix this is to shell/command-line into a pod running as root and which mounts the shared-store volume, and run chmod on the scratch directory to allow permissions for all. The
replicator pod running in the compute namespace is suitable for this.
With kubectl access to the K8S cluster, the following should help resolve.
# Get the name of the replicator pod
kubectl -n <compute namespace> get pod -l app.kubernetes.io/name=replicator
# Shell into the replicator container within the replicator pod
kubectl -n <compute namespace> exec -t -i <replicator pod name> -c replicator -- /bin/bash
# The above will launch a shell in the container. Now chmod the scratch directory
chmod 0777 /domino/shared/scratch
# Exit the shell 11exit