Openstack: Failed to list *v1.Endpoints, Services

[apurvann]

I am trying to deploy the cluster on Openstack and have been running into this error

kubectl exec -ti busybox -- nslookup kubernetes.default
Server:    10.233.0.3
Address 1: 10.233.0.3

nslookup: can't resolve 'kubernetes.default'
command terminated with exit code 1

Next ->
While checking the logs of the kube-dns pods via kubectl logs --namespace=kube-system kube-dns-cf9d8c47-vhwpn kubedns

The stacktrace looks like :

I1124 14:08:23.850502       1 dns.go:48] version: 1.14.6-3-gc36cb11
I1124 14:08:23.851899       1 server.go:69] Using configuration read from directory: /kube-dns-config with period 10s
I1124 14:08:23.851941       1 server.go:112] FLAG: --alsologtostderr="false"
I1124 14:08:23.851947       1 server.go:112] FLAG: --config-dir="/kube-dns-config"
I1124 14:08:23.851952       1 server.go:112] FLAG: --config-map=""
I1124 14:08:23.851955       1 server.go:112] FLAG: --config-map-namespace="kube-system"
I1124 14:08:23.851958       1 server.go:112] FLAG: --config-period="10s"
I1124 14:08:23.851962       1 server.go:112] FLAG: --dns-bind-address="0.0.0.0"
I1124 14:08:23.851964       1 server.go:112] FLAG: --dns-port="10053"
I1124 14:08:23.851969       1 server.go:112] FLAG: --domain="cluster.local."
I1124 14:08:23.851973       1 server.go:112] FLAG: --federations=""
I1124 14:08:23.851977       1 server.go:112] FLAG: --healthz-port="8081"
I1124 14:08:23.851980       1 server.go:112] FLAG: --initial-sync-timeout="1m0s"
I1124 14:08:23.851983       1 server.go:112] FLAG: --kube-master-url=""
I1124 14:08:23.851987       1 server.go:112] FLAG: --kubecfg-file=""
I1124 14:08:23.851990       1 server.go:112] FLAG: --log-backtrace-at=":0"
I1124 14:08:23.851995       1 server.go:112] FLAG: --log-dir=""
I1124 14:08:23.851998       1 server.go:112] FLAG: --log-flush-frequency="5s"
I1124 14:08:23.852001       1 server.go:112] FLAG: --logtostderr="true"
I1124 14:08:23.852004       1 server.go:112] FLAG: --nameservers=""
I1124 14:08:23.852007       1 server.go:112] FLAG: --stderrthreshold="2"
I1124 14:08:23.852010       1 server.go:112] FLAG: --v="2"
I1124 14:08:23.852013       1 server.go:112] FLAG: --version="false"
I1124 14:08:23.852025       1 server.go:112] FLAG: --vmodule=""
I1124 14:08:23.852096       1 server.go:194] Starting SkyDNS server (0.0.0.0:10053)
I1124 14:08:23.852326       1 server.go:213] Skydns metrics enabled (/metrics:10055)
I1124 14:08:23.852335       1 dns.go:146] Starting endpointsController
I1124 14:08:23.852339       1 dns.go:149] Starting serviceController
I1124 14:08:23.852517       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I1124 14:08:23.852524       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I1124 14:08:24.353007       1 dns.go:170] Initialized services and endpoints from apiserver
I1124 14:08:24.353034       1 server.go:128] Setting up Healthz Handler (/readiness)
I1124 14:08:24.353045       1 server.go:133] Setting up cache handler (/cache)
I1124 14:08:24.353051       1 server.go:119] Status HTTP port 8081
I1124 14:25:11.893813       1 streamwatcher.go:103] Unexpected EOF during watch stream event decoding: unexpected EOF
I1124 14:25:11.894599       1 streamwatcher.go:103] Unexpected EOF during watch stream event decoding: unexpected EOF
E1124 14:25:11.895133       1 reflector.go:304] k8s.io/dns/pkg/dns/dns.go:150: Failed to watch *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=3708&timeoutSeconds=400&watch=true: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:11.895492       1 reflector.go:304] k8s.io/dns/pkg/dns/dns.go:147: Failed to watch *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=4193&timeoutSeconds=442&watch=true: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:12.896534       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:12.897497       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:13.897852       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:13.898735       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:14.899188       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:14.900152       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:15.900597       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:15.901432       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:16.902006       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:16.903053       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:17.903781       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:17.904249       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:18.905085       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:18.905952       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:19.906532       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:19.907473       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:20.907869       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:20.908612       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:21.909228       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:21.910065       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:22.910652       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:22.911478       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:25.247446       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: unexpected EOF
E1124 14:25:25.247651       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: unexpected EOF
E1124 14:25:26.248739       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:26.253183       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:27.250260       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:27.254321       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:28.251459       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:28.255263       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:29.252734       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:29.256313       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:30.254248       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:30.257243       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:31.255622       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:31.258301       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:32.257013       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:32.259198       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:33.258629       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:33.260348       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:34.260129       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:34.261346       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:35.261520       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:35.262499       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:36.262732       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:36.263538       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:37.264140       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:37.265030       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:38.265983       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:38.266392       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:39.267295       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:39.268196       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:40.268475       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:40.269307       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:41.269781       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:41.270742       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:42.271665       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:42.272027       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:43.273459       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:43.273859       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:44.274773       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:44.275809       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:45.275969       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:45.276866       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:46.277765       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:46.278255       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:47.279369       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:47.279735       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:48.280747       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:48.281499       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:49.281798       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:49.282728       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:50.283600       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:50.284145       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:51.284847       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:51.285732       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:52.286525       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:52.286975       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:53.287787       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:53.288686       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:54.289730       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused
E1124 14:25:54.289934       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: connection refused

I have seen that kubespray uses kube_service_addresses (default 10.233.0.0/18) and kube_pods_subnet (default 10.233.64.0/18.) , but the network that I am using has the IP range 192.168.1.0/24 , hence the masters and nodes are assigned an IP from the same pool.

Any ideas what could be causing this?

[apurvann]

Also, I tried to curl to the kube-dns proxy:

curl http://localhost:8080/api/v1/namespaces/kube-system/services/kube-dns/proxy
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "no endpoints available for service \"kube-dns\"",
  "reason": "ServiceUnavailable",
  "code": 503
}

[apurvann]

I found the solution, instead of using kubernetes.default, I used <instance_name>.cluster.local as it was mentioned in my /etc/hosts

$ kubectl exec -ti busybox -- nslookup <instance_name>.cluster.local
Server:    10.233.0.3
Address 1: 10.233.0.3 kube-dns.kube-system.svc.cluster.local

Name:      <instance_name>.cluster.local
Address 1: 192.168.1.7 <instance_name>