crictl

4 minutes read

In this exercise, you’ll install crictl, a command-line interface for CRI-compatible container runtimes. crictl is an handy tool for low-level troubleshooting when kubectl is not sufficient.

  1. Install crictl on your worker Nodes

Run the following command on each worker. Make sure to select the right architecture.

VERSION="v1.33.0"
ARCH=amd64 # Use arm64 if you run your Multipass cluster on MacOS with ARM processor
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-$ARCH.tar.gz
sudo tar zxvf crictl-$VERSION-linux-$ARCH.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-$ARCH.tar.gz

# Verify the installation 
crictl --version

  1. Configure crictl to connect to the container runtime

  2. List and inspect containers using crictl

  3. Debug a failing pod using crictl commands

  4. Compare crictl output with kubectl output

  5. Clean up test resources

https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/

https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md

  1. Install crictl on a worker node

Instructions provided

  1. Configure crictl to connect to the container runtime

First, check the container runtime socket (containerd in your environment):

sudo crictl config --list

Then, verify the configuration:

sudo crictl info
  1. List and inspect containers using crictl
  • Listing all Pods, this is similar to kubectl get pods but at container runtime level
sudo crictl pods
  • Listing all containers
sudo crictl ps
  • Listing all containers including stopped ones
sudo crictl ps -a
  • Getting detailed information about containers
sudo crictl ps -v
  • listing images present on the current Node
sudo crictl images
  • Checking container resources usage
sudo crictl stats --all
  1. Debug a failing Pod using crictl commands

First, create the following Pod from your control plane Node:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: debug-test
  namespace: default
spec:
  containers:
  - name: failing-container
    image: busybox:1.37
    command: ["/bin/sh"]
    args: ["-c", "echo 'Starting...' && sleep 10 && exit 1"]
  - name: working-container
    image: busybox:1.37
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo 'Working...'; sleep 30; done"]
  restartPolicy: Always
EOF

Next, check where the Pod is running:

kubectl get pod debug-test -o wide

Next, SSH onto the corresponding worker Node.

Then, use crictl to get the Pod’s details and to troubleshoot the error you’ll find.

  • verify the Pod is on the Node
sudo crictl pods | grep debug-test
  • Get the Pod’s identifier
POD_ID=$(sudo crictl pods --name debug-test -q)
  • Get detailed pod information
sudo crictl inspectp $POD_ID
  • List containers in the pod
sudo crictl ps --pod $POD_ID
  • Get logs from the failing container
FAILING_CONTAINER_ID=$(sudo crictl ps --pod $POD_ID --name failing-container -aq)
sudo crictl logs $FAILING_CONTAINER_ID

You should only see:

Starting...
  • Get logs with timestamps
sudo crictl logs -t $FAILING_CONTAINER_ID
  • Follow logs in real-time
sudo crictl logs -f $FAILING_CONTAINER_ID
  • Inspect the failing container
sudo crictl inspect $FAILING_CONTAINER_ID
  • Execute commands in the working container
WORKING_CONTAINER_ID=$(sudo crictl ps --pod $POD_ID --name working-container -q)
sudo crictl exec -it $WORKING_CONTAINER_ID /bin/sh
  • Check container statistics
sudo crictl stats $WORKING_CONTAINER_ID
  1. Compare crictl output with kubectl output
  • From control plane (kubectl):
kubectl get pods debug-test -o wide
kubectl describe pod debug-test
kubectl logs debug-test -c failing-container
kubectl exec -it debug-test -c working-container -- /bin/sh
  • From worker node (crictl):
sudo crictl pods --name debug-test
sudo crictl inspectp $POD_ID
sudo crictl logs $FAILING_CONTAINER_ID
sudo crictl exec -it $WORKING_CONTAINER_ID /bin/sh

The key differences between these tools:

  • crictl works directly with the container runtime
  • crictl shows container runtime IDs vs Kubernetes Pod names
  • crictl can access containers even if the kubelet is not responsive
  • crictl provides runtime-specific information (cgroup, namespaces, etc.)
  1. Clean up test resources

Run the Pod deletion command from the control plane Node:

kubectl delete pod debug-test

From the worker Node, verify the containers and the Pod are gone:

sudo crictl ps --name debug-test
sudo crictl pods --name debug-test

You’ll find below examples of crictl commands for various scenario:

When kubectl is not working, but containers are running:

sudo crictl ps
sudo crictl logs <container-id>

When you need to inspect container mounts and volumes:

sudo crictl inspect <container-id> | jq '.status.mounts'

When debugging network issues at container level:

sudo crictl inspectp <pod-id> | jq '.status.network'

When checking container restart history:

sudo crictl ps -a | grep <container-name>

When you need to stop/start containers manually:

sudo crictl stop <container-id>
sudo crictl start <container-id>

When debugging image pull issues:

sudo crictl images
sudo crictl pull <image-name>
sudo crictl rmi <image-id>