ACING THE CKAD

Debugging and Fixing Kubernetes Apps during the CKAD Exam

During the CKAD exam, there’s a chance that you’ll make a mistake while writing or modifying a YAML configuration file, or you may be tasked with debugging an existing issue as one of the questions. Regardless of what measure this “issue” comes in, your goal would be to figure out what is wrong (debug) and fix the issue.

At the risk of sounding cliché, there’s an old Japanese proverb that says, “Everyone makes mistakes. That’s why there’s an eraser on every pencil.” Humans are prone to making mistakes in all areas of life — while coding, writing, deploying applications, etc., but there’s always room for recognizing and fixing these mistakes.

This article highlights my approach when diagnosing and debugging applications running in Kubernetes. It also includes 5 Kubernetes debugging tips.

Debugging an error caused by your mistake

Let’s assume you were asked to solve the questions below during the CKAD exam.

Question 1

Create a Persistent Volume called pod-volume. It should use a storage class name of manual, use RWX as the accessModes, and have a size of 5Gi. This volume should use the hostPath /opt/volume/nginx.

Answer

To solve this task, you’ll typically have to create a new YAML file using this vim command vim pv.yaml and add the required content.

// The YAML configuration file for the question above should look like this:
apiVersion: v1
kind: PersistentVolume
metadata:
 name: pod-volume
spec:
 capacity:
   storage: 5Gi
 accessModes:
   - ReadWriteMany
 storageClassName: manual
 hostath:
   path: /opt/volume/nginx

Then, you’ll have to run the kubectl create -f pv.yaml command to create the Persistent Volume. If the YAML file above didn’t have any issues, the Persistent Volume would have been created successfully. But that’s not the case here. If you tried to create the Persistent Volume using the YAML configuration file above, you would get this error message below:

This is because I made a mistake somewhere. Even though I purposely made this mistake to show you how I would ideally debug the error, there’s a high chance a mistake like this can happen unintentionally during the CKAD exam, so let’s talk about how to approach debugging in this case.

Debugging Process

To debug an error like this, I’d typically read the error message carefully and pick out the primary area of concern.

If you look closely at the error message, you’ll notice the area of concern is stated as:

unknown field “hostath” in io.k8s.api.core.v1.PersistentVolumeSpec

This means that there’s no field named “hostath” in the Persistent Volume specification and that you may have either misspelled it, added it under the wrong spec, or something else.

The first thing I’ll do after discovering this would be to read the question again and confirm if I added the fields I was supposed to add properly. On doing this, I’ll realize I was expected to set the “volume hostPath” not the “volume hostath”. I’ll open up the YAML file on the vim editor, and correct the spelling error.

// Updated version of the YAML file
apiVersion: v1
kind: PersistentVolume
metadata:
 name: pod-volume
spec:
 capacity:
   storage: 5Gi
 accessModes:
   - ReadWriteMany
 storageClassName: manual
 hostPath:
   path: /opt/volume/nginx

I’ll then run the kubectl create -f pv.yaml command again to confirm if the error has now been fixed. If the Persistent Volume is created successfully, then I can move to the next question. If not, I’ll continue debugging.

An essential tip to prevent errors like this during the exam is to always copy and paste the fields (e.g namespace, hostPath, object name, etc.) into the YAML file instead of typing it yourself. The former is faster and reduces the chances of you making a spelling error.

Question 2

Sometimes, you won’t figure out the issue by rereading the question or the error message. You may have to read the documentation or review your YAML configuration file carefully. Let’s cover that scenario using the question below:

Create a cronjob called dice that runs every one minute using the kodekloud/throw-dice image. This cronjob should be non-parallel and should complete the task at least once. It should also have a backoffLimit of 25, and if the task isn’t completed within 20 seconds, the job should fail, and pods should be terminated.

Answer

In the world of Kubernetes, a cronjob can be created using the imperative or declarative approach. For the purpose of this article, I will be using the declarative approach.

To learn more about the difference between the two approaches, check out “CKAD Tips: Kubernetes Object Management Techniques.”

// The YAML configuration file for the question above should look like this:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
 name: dice
spec:
  completions: 1
  backoffLimit: 25
  activeDeadlineSeconds: 20
 schedule: "*/1 * * * *"
 jobTemplate:
   spec:
     template:
       spec:
         containers:
         - name: dice
           image: kodekloud/throw-dice
         restartPolicy: Never

On running the kubectl create -f cronjob.yaml command, you’ll get the error below:

Debugging Process

To debug an issue like this, I’ll read the error message carefully and pick out the primary area of concern.

If you look closely at this error message, you’ll notice the area of concern is stated as “error validating unknown fields activeDeadlineSeconds, backoffLimit, and completions”. This could mean you spelled something incorrectly, you added the wrong fields, or added the right fields into the wrong place. We are going to explore all of these options!

• First, I’ll recheck the question to ensure all fields were added properly. Everything looks good, so I’ll keep debugging!
• I’ll go over to the Kubernetes documentation and quickly read through what creating a cronjob entails and look at the sample YAML file. On doing this, I’ll realize that the activeDeadlineSeconds, backoffLimit, and completions fields should have been added into the jobTemplate spec and not the spec of the cronjob.

// Updated version of the YAML file
apiVersion: batch/v1
kind: CronJob
metadata:
 name: dice
spec:
 schedule: "*/1 * * * *"
 jobTemplate:
   spec:
     completions: 1
     backoffLimit: 25
     activeDeadlineSeconds: 20
     template:
       spec:
         containers:
         - name: dice
           image: kodekloud/throw-dice
         restartPolicy: Never

• After updating the YAML file, I’ll run the kubectl create -f cronjob.yaml command again to see if the cronjob was created successfully. And if it is, I’ll take another step further to see if it is running via the kubectl get jobs command.

Debugging an existing issue

Let’s assume you were asked to debug the issue below during the CKAD exam.

Issue

We have deployed a new pod called secure-pod and a service called secure-service. Incoming or outgoing connections to this pod are not working. Troubleshoot why this is happening and ensure that incoming connections from the pod webapp-color are successful.

Debugging Process

To debug an issue like this, I’ll need to confirm if the specified Kubernetes objects exist. Once I do that, I’ll need to verify if they work as intended. Chances are one of them will not be working as it should be, then I’ll dig deeper into that Object and figure out the issue.

• First, I’ll run the kubectl get pods, svc command to confirm if the pods and services are up and running.

• Everything looks good. So, I will exec into the webapp-color pod using the kubectl exec -it webapp-color —- sh command.

• Then, I’ll connect to the service using this command nc -z -v -w 1 secure-service 80. Based on the response, it is now evident that something is blocking the connection.

• In Kubernetes, you can use a Network Policy to block or allow incoming and outgoing connections across pods. So, the next thing I’ll do is verify if there’s an existing network policy blocking the incoming connection to the pod using the kubectl get netpol command.

• On doing this, I can see a network policy exists. Now, I’ll delete the existing network policy and create another one that ensures incoming connections from the pod webapp-color are successful.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allow-traffic
 namespace: default
spec:
 podSelector:
   matchLabels:
     run: secure-pod
 policyTypes:
 - Ingress
 ingress:
 - from:
   - podSelector:
       matchLabels:
         name: webapp-color
   ports:
   - protocol: TCP
     port: 80

• Once again, I’ll exec into the pod and connect it to the service. This time around, the connection works, and I can smile knowing that my job is done and move to the next question. 😅

Debugging Tips for the CKAD Exam

1. Most of the time, the reason why something isn’t working is highlighted in the error message. So, always read the error messages carefully.
2. Always remember to use the documentation when you are stuck. While preparing for the CKAD exam, I made mistakes along the way. For instance, I would mistakenly add fields under the wrong spec or misspell something because I typed it out myself. But once I looked at the documentation, I could always figure out what I did wrong.
3. There is no one-size-fits-all approach to debugging. I believe it can be done in different ways by different people and as you keep gaining experience in your Kubernetes, you’ll discover better ways of debugging. So keep practicing!
4. I recently joined the Klustered show hosted by David Flanagan. It was interesting to see that even though everyone had a different approach to debugging Kubernetes applications or clusters. It always revolved around “thinking of the components that would’ve made the application or cluster work well”, “testing these different components to identify what isn’t working well” and once you’ve identified it, “fix the problem”. You should watch the Klustered show, it is very insightful.
5. There’s a great Twitter thread by Daniel Bryant on how to debug Kubernetes applications & clusters. You should check it out.

Conclusion

I hope you found this article useful and have learned a thing or two about debugging in Kubernetes, especially in reference to the CKAD exam.

Most of the questions used in this article are inspired by the KodeKloud CKAD course. I recommend checking it out.

Ace the CKAD with me

This article is part of the CKAD Tips weekly series, where I write articles that will enable anyone studying for the CKAD to excel during the examinations.

Stay up to date on the latest additions to this series by visiting our Certified Kubernetes Application Developer: Everything You Need to Know page and kickstart your Kubernetes learning via our Kubernetes Developer Learning Center.

The CKAD series, so far:

• Frequently Asked Questions About the CKAD Exam, Answered
• 6 Reasons You Should Take the CKAD
• How to create & connect to a Google Cloud VM using SSH on Mac/Linux
• Kubernetes Architecture Explained
• Kubernetes Object Management Techniques — why you should know them before taking the CKAD exam
• CKAD/CKA Exam Tips from 10 People Who Passed the Exam
• How to Use the Vim Editor
• Debugging and Fixing Kubernetes Apps during the CKAD Exam