All smarter-device-manager pods in CrashLoopBackOff condition
domino-platform smarter-device-manager-28z8f 0/1 CrashLoopBackOff 3681 (35s ago) 13d 10.195.103.77 aaa100182 <none> <none> domino-platform smarter-device-manager-2g7lh 0/1 CrashLoopBackOff 3195 (2m16s ago) 23d 10.195.103.136 aaa100488 <none> <none> domino-platform smarter-device-manager-2kwd7 0/1 CrashLoopBackOff 3675 (5m21s ago) 13d 10.195.103.29 aaa1617 <none> <none> domino-platform smarter-device-manager-2v948 0/1 CrashLoopBackOff 3681 (2m4s ago) 13d 10.195.146.47 aaa2016 <none> <none>
Pod describe output shows following failure,
Message: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/var/lib/kubelet/pods/696fd7f9-d9f8-4b84-b4d2-af467966649a/containers/smarter-device-manager/0b4905f9" to rootfs at "/dev/termination-log" caused: open /apps/domino/scratch/docker/overlay2/024cd1c64ac13fe459f8a656f720f9edc180fd91175e69feb72e37ba5fcd7cef/merged/dev/termination-log: read-only file system: unknown
/devfrom the host into its container, read-only
Kubernetes tries to create the file into the host’s
/dev, but can’t because it’s mounted read-only
The fix has been submitted and will be available in Domino version 5.6. In the meantime you can manually edit the smart-device-manager daemonset, below are the steps:
- Check the smarter-device-manager daemonset,
kubectl get ds -n domino-platform | grep smarter
- Make backup of smarter-device-manger daemonset,
kubectl get ds -n domino-platform smarter-device-manager -o yaml > smart-device-manager.yaml
- Manually edit the smarter-device-manager daemonset, change
- Above edit should trigger restart. Please check see if pods are getting replaced by new pods. Sometime when pods in a bad state the updates fails to happen and if that's the case please proceed with step 5.
- Restart smarter-device-manager daemonset,
kubectl rollout restart ds/smarter-device-manager -n domino-platform
- Check smarter-device-manager pods to ensure its now in running status.
If you have any further questions, please contact Domino Support at email@example.com.