microk8s: Service Endpoints not resolving

Hi Canonical Team,

We started facing issues with Microk8s for the past two days where our pods are not able to communicate with each other with the service endpoint url eg:minio-service.default.svc.cluster.local, When we tried to spin up a dummy dns pod and tried to nslookup from there, we could see it’s not resolving. also even cluster.local is not being resolved but kubernetes.default is getting resolved.

microk8s kubectl exec -i -t dnsutils -- nslookup cluster.local
Server:		10.152.183.10
Address:	10.152.183.10#53

** server can't find cluster.local.ec2.internal: SERVFAIL

command terminated with exit code 1

root@ip-10-0-1-94:/home/ubuntu# microk8s kubectl exec -i -t dnsutils -- nslookup kubernetes.default
Server:		10.152.183.10
Address:	10.152.183.10#53

Name:	kubernetes.default.svc.cluster.local
Address: 10.152.183.1

Weird thing is when i just nslookup svc.ns it’s resolving without any issues.

root@ip-10-0-1-94:/home/ubuntu# microk8s kubectl exec -i -t dnsutils -- nslookup minio-service.default
Server:		10.152.183.10
Address:	10.152.183.10#53

Name:	minio-service.default.svc.cluster.local
Address: 10.152.183.246

root@ip-10-0-1-94:/home/ubuntu# microk8s kubectl exec -i -t dnsutils -- nslookup minio-service.default.svc.cluster.local
Server:		10.152.183.10
Address:	10.152.183.10#53

** server can't find minio-service.default.svc.cluster.local.ec2.internal: SERVFAIL

command terminated with exit code 1

Attaching tarball for reference. inspection-report-20211021_073518.tar.gz core-dns.log

Also the dns entry for ec2.internal is from /run/systemd/resolve/resolv.conf and not from /etc/resolv.conf as mentioned in below link. known issues Was there any breaking change or are we missing something since we started facing this issues only for the past three days.