Application Fails Due to DNS Resolution Issue in Onprem Environment

Problem

An internal application was crashing due to DNS resolution failures from within a pod in an airgapped Kubernetes environment.

Environment

Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher
Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher

Cause

The environment is air-gapped and cannot reach external nameservers, the internal nameservers defined on the node are not configured with the application's FQDN address mapping for domain name resolution. To work around this, an entry for <external-domain> was added to the /etc/hosts file on each node.

While this allows resolution from the node shell, it does not apply inside pods, as pod DNS resolution is handled by CoreDNS.

Diagnostics

Testing from the pod:

$ kubectl run test-dns --rm -it --image=busybox --restart=Never -- nslookup <FQDN> Server: [NAMESERVER_IP] Address: [NAMESERVER_IP]#53 ** server can't find [FQDN]: NXDOMAIN

Testing from Node:

$ kubectl run test-dns --rm -it --image=busybox --restart=Never -- nslookup <FQDN> Server: [NAMESERVER_IP] Address: [NAMESERVER_IP]#53 Non-authoritative answer: Name: [FQDN] Address: [EXTERNAL_IP_ADDRESS_OF_SERVICE_MAPPED_TO_THE FQDN]

This confirms resolution works on the node via /etc/hosts or another resolver, but not within pods.

Resolution

Edit the CoreDNS ConfigMap:

$ kubectl edit configmap coredns -n kube-system

Modify the Corefile section by adding or modifying the hosts block with the correct nameserver IP and fully qualified domain name (FQDN). For example:

apiVersion: v1 data: Corefile: | .:53 { errors health { lameduck 5s } ready ... ... hosts { <EXTERNAL_IP_OF_SERVICE_MAPPED_TO_FQDN> <FQDN> fallthrough }

Replace <EXTERNAL_IP_OF_SERVICE_MAPPED_TO_FQDN> with the actual IP address and <FQDN> with the correct FQDN of the application.

Save and exit the editor. This will update the ConfigMap.
Restart the CoreDNS pods to apply the new configuration:

$ kubectl rollout restart deployment coredns -n kube-system

Validate the coredns pods are up and running

$ kubectl get pods -n kube-system -o wide | grep -i coredns kube-system coredns-fc6bdfd64-7jvqk 1/1 Running 0 8s [POD_IP] [NODE_IP] <none> <none>

Validation

After restarting CoreDNS, validate that the DNS resolution works:

Use a test pod to perform an nslookup:

$ kubectl run test-dns --rm -it --image=busybox --restart=Never -- nslookup <FQDN> Server: [Nameserver-IP] Address: [Nameserver-IP]#53 Non-authoritative answer: Name: [FQDN] Address: [EXTERNAL_IP_OF_SERVICE_MAPPED_TO_FQDN]

Ensure the application no longer crashes due to DNS resolution issues.
Confirm logs of the application or CoreDNS show successful resolution.

Additional Information

The hosts plugin in CoreDNS is similar to the /etc/hosts file and allows static DNS entries.
fallthrough ensures that if a name is not matched in the hosts section, CoreDNS will continue to other plugins like kubernetes or forward.
Use kubectl logs -n kube-system -l k8s-app=kube-dns to check CoreDNS logs if resolution still fails.

Last updated on

Was this page helpful?