Solution

2 minutes read
  1. Test

Try to create a pod:

kubectl run ghost --image=ghost:4

Verify it is running:

$ kubectl get po
NAME    READY   STATUS    RESTARTS   AGE
ghost   0/1     Pending   0          10s

There is definitely something wrong as the pod remains in pending status.

$ kubectl describe po ghost
...
Events:
  Type     Reason            Age               From               Message
  ----     ------            ----              ----               -------
  Warning  FailedScheduling  4s (x2 over 72s)  default-scheduler  0/3 nodes are available: 3 node(s) had taint {app: secure}, that the pod didn't tolerate.

It seems the 3 nodes have a taint that prevents the pod from being scheduled. We can see there are 2 taints on the master:

  • app=secure:NoSchedule
  • node-role.kubernetes.io/master:NoSchedule (default one when using kubeadm)
$ kubectl get no master -o jsonpath={.spec.taints} | jq
[
  {
    "effect": "NoSchedule",
    "key": "app",
    "value": "secure"
  },
  {
    "effect": "NoSchedule",
    "key": "node-role.kubernetes.io/master"
  }
]

Only the app=secure:NoSchedule taint exists on both worker nodes:

# worker1
$ kubectl get no worker1 -o jsonpath={.spec.taints} | jq
[
  {
    "effect": "NoSchedule",
    "key": "app",
    "value": "secure"
  }
]

$ kubectl get no worker2 -o jsonpath={.spec.taints} | jq
# worker2
[
  {
    "effect": "NoSchedule",
    "key": "app",
    "value": "secure"
  }
]
  1. Fix it

Let’s delete the ghost pod and create a new one with a toleration for the app=secure:NoSchedule taint

kubectl delete po/ghost

Creation of a pod specification

kubectl run ghost --image=ghost:4 --dry-run=client -o yaml > pod.yaml

Adding the toleration:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: ghost
  name: ghost
spec:
  tolerations:
  - key: app
    value: secure
    effect: NoSchedule
  containers:
  - image: ghost:4
    name: ghost

Creation of the new pod:

kubectl apply -f pod.yaml

The pod is now running fine:

$ kubectl get po
NAME    READY   STATUS    RESTARTS   AGE
ghost   1/1     Running   0          2s