lookup failures 'mongodb-replicaset on 10.85.0.10:53: no such host'

Version:

Any platform that has updated the CoreDNS version 1.83 +

Issue:

After upgrading AKS from 1.19 to 1.21 mongodb-replicaset-0 pod can’t start properly, on inspection of logs we see:

[root@solo]# kubectl logs mongodb-replicaset-0 -n domino-platform -c mongodb-replicaset
2022/05/20 09:01:12 Determined Domain to be domino-platform.svc.cluster.local
2022/05/20 09:01:12 lookup mongodb-replicaset on 10.85.0.10:53: no such host
2022/05/20 09:01:13 lookup mongodb-replicaset on 10.85.0.10:53: no such host
2022/05/20 09:01:14 lookup mongodb-replicaset on 10.85.0.10:53: no such host

Root Cause:

With CoreDNS v 1..8.4, DNS resolution rules changed from publishing pod addresses irrespective of their ‘readiness’ and the servicepublishNotReadyAddressessetting to ensuring pods are ready or the option is enabled.

For the mongodb-replicaset deployed with Domino, a previous, alpha, version of the publishNotReadyAddresses setting had been used to ensure that multiple replicas could talk to each other without being ‘ready.’ This option is no longer respected in CoreDNS and, prior to v1.8.4

Resolution:

Resolution is a 3 step process, we firstly must validate the CoreDNS Version using the command below:

$ kubectl get deploy -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'; echo

Apply the following patch to enable mongodb-replicaset service lookups on a more recent CoreDNS version:

$ kubectl patch -n <domino-platform> service mongodb-replicaset --type=json -p '[{"op": "replace", "path": "/spec/publishNotReadyAddresses", "value": true}]'

MongoDB will pick up the service entry change but for good measure a rolling restart of the pods may be required:

Notes:

This has been seen on a small number of self installed and self managed AKS Domino environments.

In Domino 5.x, the correct publishNotReadyAddresses setting has been applied to the MongoDB service.