How to Investigate Kubernetes Container Issues With “Kubectl Debug”

Kubernetes logo

It can be tricky to diagnose problems with running Kubernetes workloads. You might be lucky and find the cause in your application’s logs, via the kubectl logs command. In some cases there’s no avoiding a live debugging session though, where you interactively engage with your Pods to uncover issues.

The kubectl debug command simplifies these debugging tasks by providing a new ephemeral container inside your Pod. This can be used to inspect the Pod’s environment so you can start troubleshooting problems that are surfacing in your existing containers.

Preparing to Use Kubectl Debug

kubectl debug was launched with v1.18 of Kubernetes and Kubectl. It relies on ephemeral containers being available in your cluster. Ephemeral containers became a beta feature in Kubernetes v1.23 and are now enabled by default. You’ll need to manually enable the feature gate if your cluster runs an older Kubernetes version.

Ephemeral containers are designed for transitory tasks where you need to temporarily connect an extra container to an existing Pod. This is ideal for debugging operations where you want to accurately inspect a Pod without affecting live container instances.

Most container images lack debugging tools; installing them within a running container would mutate its environment and potentially cause side effects. Attaching an ephemeral container to your Pod is a safer way to debug that gives you a clean working environment. You can use a heavier image that includes all the tools you need.

Although ephemeral containers become part of their host Pod, there are still some differences to be aware of. Ephemeral containers don’t support port binds, probes, or resource reservations as they’re only temporary in nature. They’ll never be automatically restarted and can’t be changed once they’ve been created. A complete list of supported capabilities is available in the documentation.

Using Kubectl Debug

Before continuing, create a basic deployment to use for testing purposes:

$ kubectl create deployment nginx --image=nginx:latest
deployment.apps/nginx created

Next use the get pods command to find the name of your deployment’s Pod:

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-55649fd747-qsnr2   1/1     Running   0          5s

Our deployment’s Pod is called nginx-55649fd747-qsnr2.

Now you can use the kubectl debug command to start a debugging session inside your Pod:

$ kubectl debug -it --image=ubuntu:20.04 nginx-55649fd747-qsnr2

The command’s syntax is similar to a hybrid of kubectl create and kubectl debug. The unnamed argument supplied to the command identifies an existing Pod to attach to. The --image argument specifies the image to use for the new container. We’re using ubuntu:20.04 here to obtain access to the familiar commands included in the Ubuntu Linux distribution.

The -it flag is equivalent to --stdin --tty. Including these arguments will allocate a TTY to the container, attach to it, and connect your terminal’s stdin stream. This gives you an interactive shell inside your new container.

Now you can carry out your debugging tasks from within your ephemeral container.

Copying Pods

Another way to use kubectl debug is with a --copy-to argument. This creates a copy of the target Pod and adds the ephemeral container to the copy. The original Pod is left intact.

$ kubectl debug -it --image=ubuntu:20.04 --copy-to nginx-debug nginx-555649fd747-qsnr2

This feature gives you even greater assurance that changes made during debugging won’t directly impact your production application.

Copying the Pod also lets you activate process namespace sharing. This makes the existing processes in your Pod visible to your ephemeral container. It can’t be used with existing containers as their spec.shareProcessNamespace field will usually be set to false. Running kubectl debug with the --copy-to and --share-processes flag will enable process sharing on the copied Pod, making this procedure much more intuitive:

$ kubectl debug -it --image=ubuntu:20.04 --copy-to nginx-debug --share-processes nginx-555649fd747-qsnr2

The process list visible to your ephemeral Ubuntu container will now include an NGINX process:

$ ps ax
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    9 root      0:00 nginx: master process nginx -g daemon off;

This process is still running in the separate NGINX container within your Pod. Namespace sharing also provides access to the target container’s filesystem via /proc:

$ ls /proc/9/root/etc/nginx
conf.d fastcgi_params mime.types modules nginx.conf ...

Copying the Pod in this way is therefore a powerful debugging tool. You can readily inspect the Pod’s files and processes using a separate container that’s prepared with familiar tools.

Optional Arguments

The --copy-to flag always leaves the original Pod intact by default. You can make the operation act as a replacement instead using --replace. This will stop the first Pod.

$ kubectl debug -it --image=ubuntu:20.04 --copy-to nginx-debug --replace nginx-555649fd747-qsnr2

Kubernetes will schedule the copied Pod to any available Node. This can be problematic if you want to ensure a consistent test environment. Adding --same-node will schedule the copy to the existing Pod’s Node, eliminating any differences that may exist between machines in your cluster.

$ kubectl debug -it --image=ubuntu:20.04 --copy-to nginx-debug --same-node nginx-555649fd747-qsnr2

Another useful option is --env to set extra environment variables inside your ephemeral container. You may need to use this to configure debugging tools or override values inherited from your target Pod.

$ kubectl debug -it --image=ubuntu:20.04 --copy-to nginx-debug --env EDITOR=/usr/bin/nano nginx-555649fd747-qsnr2

Finally, remember that containers created by kubectl debug don’t have to be interactive. You can easily run one-off commands against your Pods using kubectl exec-like syntax. The --attach argument is supported to control whether your shell’s connected to the container when you’re not running with -i (--stdin).

$ kubectl debug --image=ubuntu:20.04 --copy-to nginx-debug --share-processes --attach true nginx-555649fd747-qsnr2 -- ls /proc/9/root/etc/nginx
conf.d fastcgi_params mime.types modules nginx.conf ...

Conclusion

Ephemeral containers and the kubectl debug command provide a simplified debugging experience for Kubernetes workloads. You can run commands inside a Pod using a different image to your regular containers. This lets you access debugging tools that aren’t included in your application’s image.

kubectl debug can also create copies of Pods and share their processes with the original. This mechanism lets you inspect the processes in the target Pod’s containers, from a separate ephemeral container that you’ve got full control over. It provides more advanced debugging options when you need to interrogate running processes.