# Pastebin cJXd0NFH [rwellum@k8s ~]$ kubectl describe po kube-dns-3913472980-h457k -n kube-system Name: kube-dns-3913472980-h457k Namespace: kube-system Node: k8s/10.240.43.175 Start Time: Wed, 17 May 2017 09:16:10 -0400 Labels: k8s-app=kube-dns pod-template-hash=3913472980 Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kube-dns-3913472980","uid":"b4e46ff0-3b02-11e7-9672-52540052... scheduler.alpha.kubernetes.io/critical-pod= Status: Running IP: Controllers: ReplicaSet/kube-dns-3913472980 Containers: kubedns: Container ID: docker://307e39d9213d946e45689c25b6f44c7f8c62001c9bc14b38342773f02f0ca84d Image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1 Image ID: docker-pullable://gcr.io/google_containers/k8s-dns-kube-dns-amd64@sha256:33914315e600dfb756e550828307dfa2b21fb6db24fe3fe495e33d1022f9245d Ports: 10053/UDP, 10053/TCP, 10055/TCP Args: --domain=cluster.local. --dns-port=10053 --config-dir=/kube-dns-config --v=2 State: Waiting Reason: rpc error: code = 2 desc = failed to start container "307e39d9213d946e45689c25b6f44c7f8c62001c9bc14b38342773f02f0ca84d": Error response from daemon: {"message":"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827"} Last State: Terminated Reason: ContainerCannotRun Message: cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827 Exit Code: 128 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Wed, 17 May 2017 09:19:19 -0400 Ready: False Restart Count: 0 Limits: memory: 170Mi Requests: cpu: 100m memory: 70Mi Liveness: http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s #success=1 #failure=5 Readiness: http-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3 Environment: PROMETHEUS_PORT: 10055 Mounts: /kube-dns-config from kube-dns-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-scw9p (ro) dnsmasq: Container ID: docker://2f7f262c90a351452b3015c8a9901804843798bd98ef0173a8a5ba7623076045 Image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1 Image ID: docker-pullable://gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64@sha256:89c9a1d3cfbf370a9c1a949f39f92c1dc2dbe8c3e6cc1802b7f2b48e4dfe9a9e Ports: 53/UDP, 53/TCP Args: -v=2 -logtostderr -configDir=/etc/k8s/dns/dnsmasq-nanny -restartDnsmasq=true -- -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053 State: Waiting Reason: rpc error: code = 2 desc = failed to start container "2f7f262c90a351452b3015c8a9901804843798bd98ef0173a8a5ba7623076045": Error response from daemon: {"message":"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827"} Last State: Terminated Reason: ContainerCannotRun Message: cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827 Exit Code: 128 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Wed, 17 May 2017 09:19:20 -0400 Ready: False Restart Count: 0 Requests: cpu: 150m memory: 20Mi Liveness: http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5 Environment: Mounts: /etc/k8s/dns/dnsmasq-nanny from kube-dns-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-scw9p (ro) sidecar: Container ID: docker://b49abb18c677015610a9b3b4dc70ae1ec18ccd9af7d1ac1357294ba6642bf823 Image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1 Image ID: docker-pullable://gcr.io/google_containers/k8s-dns-sidecar-amd64@sha256:d33a91a5d65c223f410891001cd379ac734d036429e033865d700a4176e944b0 Port: 10054/TCP Args: --v=2 --logtostderr --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A State: Waiting Reason: rpc error: code = 2 desc = failed to start container "b49abb18c677015610a9b3b4dc70ae1ec18ccd9af7d1ac1357294ba6642bf823": Error response from daemon: {"message":"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827"} Last State: Terminated Reason: ContainerCannotRun Message: cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827 Exit Code: 128 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Wed, 17 May 2017 09:19:17 -0400 Ready: False Restart Count: 0 Requests: cpu: 10m memory: 20Mi Liveness: http-get http://:10054/metrics delay=60s timeout=5s period=10s #success=1 #failure=5 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-scw9p (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: kube-dns-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: kube-dns Optional: true kube-dns-token-scw9p: Type: Secret (a volume populated by a Secret) SecretName: kube-dns-token-scw9p Optional: false QoS Class: Burstable Node-Selectors: Tolerations: CriticalAddonsOnly=:Exists node-role.kubernetes.io/master=:NoSchedule node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 5m 4m 8 default-scheduler Warning FailedScheduling no nodes available to schedule pods 3m 3m 1 default-scheduler Normal Scheduled Successfully assigned kube-dns-3913472980-h457k to k8s 1m 1m 1 kubelet, k8s Warning FailedSync Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: 483938996c0fdc0e7ce894f7e1214f6b0a45960640fbac7b6a9db5c65be5c77b 51s 51s 1 kubelet, k8s Warning FailedSync Error syncing pod, skipping: failed to "KillPodSandbox" for "b4eaec1d-3b02-11e7-9672-52540052d3ab" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"kube-dns-3913472980-h457k_kube-system\" network: CNI failed to retrieve network namespace path: Error response from daemon: {\"message\":\"devmapper: Unknown device bca34cfee54b429563f75529cc2eadc985e3e5d5f25ca10f15eb5bf19e210c6f\"}" 49s 49s 1 kubelet, k8s Warning FailedSync Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: 8af6d91aaec099300a04d0fb1f9e042bcaa484d2596952696e1df5116c28565f 3m 21s 31 kubelet, k8s Warning FailedSync Error syncing pod, skipping: failed to "CreatePodSandbox" for "kube-dns-3913472980-h457k_kube-system(b4eaec1d-3b02-11e7-9672-52540052d3ab)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-dns-3913472980-h457k_kube-system(b4eaec1d-3b02-11e7-9672-52540052d3ab)\" failed: rpc error: code = 2 desc = NetworkPlugin cni failed to set up pod \"kube-dns-3913472980-h457k_kube-system\" network: no IP addresses available in network: k8s-pod-network" 20s 20s 1 kubelet, k8s spec.containers{sidecar} Normal Pulled Container image "gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1" already present on machine 19s 19s 1 kubelet, k8s spec.containers{sidecar} Normal Created Created container with id b49abb18c677015610a9b3b4dc70ae1ec18ccd9af7d1ac1357294ba6642bf823 18s 18s 1 kubelet, k8s spec.containers{sidecar} Warning Failed Failed to start container with id b49abb18c677015610a9b3b4dc70ae1ec18ccd9af7d1ac1357294ba6642bf823 with error: rpc error: code = 2 desc = failed to start container "b49abb18c677015610a9b3b4dc70ae1ec18ccd9af7d1ac1357294ba6642bf823": Error response from daemon: {"message":"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827"} 18s 18s 1 kubelet, k8s spec.containers{kubedns} Normal Pulled Container image "gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1" already present on machine 17s 17s 1 kubelet, k8s spec.containers{kubedns} Normal Created Created container with id 307e39d9213d946e45689c25b6f44c7f8c62001c9bc14b38342773f02f0ca84d 17s 17s 1 kubelet, k8s spec.containers{kubedns} Warning Failed Failed to start container with id 307e39d9213d946e45689c25b6f44c7f8c62001c9bc14b38342773f02f0ca84d with error: rpc error: code = 2 desc = failed to start container "307e39d9213d946e45689c25b6f44c7f8c62001c9bc14b38342773f02f0ca84d": Error response from daemon: {"message":"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827"} 17s 17s 1 kubelet, k8s spec.containers{dnsmasq} Normal Pulled Container image "gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1" already present on machine 15s 15s 1 kubelet, k8s spec.containers{dnsmasq} Normal Created Created container with id 2f7f262c90a351452b3015c8a9901804843798bd98ef0173a8a5ba7623076045 15s 15s 1 kubelet, k8s spec.containers{dnsmasq} Warning Failed Failed to start container with id 2f7f262c90a351452b3015c8a9901804843798bd98ef0173a8a5ba7623076045 with error: rpc error: code = 2 desc = failed to start container "2f7f262c90a351452b3015c8a9901804843798bd98ef0173a8a5ba7623076045": Error response from daemon: {"message":"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827"} 15s 15s 1 kubelet, k8s Warning FailedSync Error syncing pod, skipping: [failed to "StartContainer" for "sidecar" with rpc error: code = 2 desc = failed to start container "b49abb18c677015610a9b3b4dc70ae1ec18ccd9af7d1ac1357294ba6642bf823": Error response from daemon: {"message":"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827"}: "Start Container Failed" , failed to "StartContainer" for "kubedns" with rpc error: code = 2 desc = failed to start container "307e39d9213d946e45689c25b6f44c7f8c62001c9bc14b38342773f02f0ca84d": Error response from daemon: {"message":"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827"}: "Start Container Failed" , failed to "StartContainer" for "dnsmasq" with rpc error: code = 2 desc = failed to start container "2f7f262c90a351452b3015c8a9901804843798bd98ef0173a8a5ba7623076045": Error response from daemon: {"message":"cannot join network of a non running container: 2ac7ec5813ed2024c1434afedde7355dd745641f70d69645a52288508a679827"}: "Start Container Failed" ] 3m 2s 32 kubelet, k8s Normal SandboxChanged Pod sandbox changed, it will be killed and re-created. [rwellum@k8s ~]$