Articles in this section

Nucleus Services Will Not Start Due to Failing Replication in MongoDB

Curtiss Porter
  • Updated
Follow

If you encounter a scenario where the nucleus frontend and dispatcher pods are stuck in a CrashLoopBackOff state, and you find entries in their logs such as:

Commands:

kubectl logs -n <domino-platform-namespace> nucleus-dispatcher-<id> --tail 1000
kubectl logs -n <domino-platform-namespace> nucleus-frontend-<id> -c nucleus-frontend --tail 1000

Logs:

1) Error in custom provider, org.quartz.SchedulerConfigException: Error while initializing the indexes
[See nested exception: com.mongodb.MongoWriteConcernException: waiting for replication timed out]

This is pointing to an issue with replication in at least one of the mongodb-replicaset pods. A tell-tale sign will be a mongodb-replicaset pod with a high restart count, and entries such as this in its logs:

Commands:

kubectl logs -n <domino-platform-namespace> mongodb-replicaset-<id> -c mongodb-replicaset --tail 1000
Logs:
2022-03-01T18:20:13.599+0000 I REPL [rsBackgroundSync] transition to SECONDARY from ROLLBACK
2022-03-01T18:20:13.599+0000 I REPL [rsBackgroundSync] Resetting sync source to empty, which was mongodb-replicaset-0.mongodb-replicaset.domino-platform.svc.cluster.local:27017
2022-03-01T18:20:13.599+0000 F REPL [rsBackgroundSync] Rollback failed with unrecoverable error: UnrecoverableRollbackError: Recover to stable timestamp failed.
2022-03-01T18:20:13.599+0000 F - [rsBackgroundSync] Fatal assertion 50666 UnrecoverableRollbackError: Recover to stable timestamp failed. at src/mongo/db/repl/bgsync.cpp 720

Nucleus services are dependent on mongoDB and therefore will not start if replication is failing in the mongodb statefulset.

 

To recover from this situation and get the nucleus services started, one must terminate the failing mongodb-replicaset pod by scaling the mongoDB stateful set down to the number of working pods, usually 2, and then back up to the original number of pods, usually 3 in most deployments.

 

Following are the commands that will accomplish this:
Identify the statefulset for the mongoDB replicaset:
kubectl get statefulsets -n <domino-platform-namespace> | grep mongo
Scale the statefulset down to working number of pods:
kubectl scale statefulsets -n <domino-platform-namespace> mongodb-replicaset --replicas=2
Check the mongodb-replicasetpods to validate its scaled down:
kubectl get pods -n <domino-platform-namespace> | grep mongodb-replicaset
Scale the stateful set back up to its original configuration:
kubectl scale statefulsets -n <domino-platform-namespace> mongodb-replicaset --replicas=3
Check the mongodb-replicasetpods to validate its scaled back up:
kubectl get pods -n <domino-platform-namespace> | grep mongodb-replicaset
The nucleus services may start automatically once mongo is restored, and you can check their status with:
kubectl get pods -n <domino-platform-namespace> | grep nucleus
If needed you can force a restart of the nucleus pods by deleting them one by one:
kubectl delete pod -n <domino-platform-namespace> <nucleus-service-name>
If the guidance in this article does not work in your particular situation, please open a support ticket with us for further assistance.

Related articles

Comments

0 comments

Please sign in to leave a comment.