An easy way to a highly available service with Kubernetes probes, Kotlin and Helm

A simple explanation with examples of how Kubernetes probes work and help keeping your services always up.

Ready to run. From unsplash by Braden Collum

Prerequisites (with the versions used in this tutorial):

- Docker (Engine 19.03.13)

- Minikube (1.15.1)

- Helm (3.4.0)

- Kubectl (matching kubernetes version)

Code is hosted in Github

About Probes

Probes (sometimes heartbeat or health checks) are a fundamental feature of Microservices. They perform constant checks to know the state of a service and enable recovery whenever things go sour.

You can set up your probes as a really simple endpoint that returns a valid HTTP status code whenever the service is running or they can be more complex and look for things like the state of the database connection or whether the service can communicate with an external dependency, send messages to a queue, etc…

Kubernetes provides out-of-the-box options to set up those checks. If configured together with a multi-replica, multi-node environment, probes allow for a proactive error handling with important features to recover from failures.

They are divided in three categories:

Liveness Probe

Liveness Probe is used to know when a pod/container should be restarted because it is running but not functional. It helps increasing availability by bringing the service back to its original state.

Whenever the initialDelaySeconds value is passed, Kubernetes starts to perform the specified command and if the response is something other than 200–399 it fails. Unless failureThreshold value is changed, the pod will be restarted after 3 failures in a row.

Readiness Probe

Readiness Probe is similar to the liveness probe in terms of options and implementation but has the function to inform Kubernetes when a pod is ready to receive traffic. Sometimes an application has to perform tasks on startup. Maybe load some data in memory, warm up a cache, process information before receiving new requests, you name it. In these scenarios, even if the service is actually running it isn’t ready to process any request.

Kubernetes changes the pod condition to ready whenever the ready endpoint returns an HTTP code between 200–399. successThreshold value can be used to determine how many times the return has to be valid before the pod receives traffic.

Startup Probe

Startup Probe provides a delay for both other probes to start. It’s specially helpful when there’s a slow starting container which guaranteed takes longer to finalize and make the liveness and readiness probe endpoints available. We will skip it in this tutorial.

Implementation starting point

In the last article of this series we created a Kotlin app with Spring Boot and deployed on Kubernetes using Helm. Now we will extend it to include liveness and readiness probes.

Writing probe endpoints

In the Application.kt file, let’s create two new endpoints:

Note that this will always return success, provided that the service is running. Later we will simulate failures.

Extending Helm deployment template

Kubernetes probes are part of the deployment specification. So in your Helm’s deployment.yaml template, add the following inside the containers spec:

In this case we are telling Kubernetes to start performing the liveness checks after 10 seconds and repeating it every 3 seconds, at the /health endpoint on the service port and readiness checks after 15 seconds, every 5 seconds on endpoint /ready.

Build and run app

To test the configuration, just build and run the app on Kubernetes (instructions on how to do this with Minikube are available here under “Starting Minikube” or you can use the run.sh script on the root of the repository).

After running kubectl get pods shows that the pod is running and the logs are printing our messages:

~/code/demos/demo-spring-boot master > kubectl get podsNAME                                READY   STATUS    RESTARTS   AGEdemo-spring-boot-7f7769cfbd-k747j   1/1     Running   0          19s~/code/demos/demo-spring-boot master > kubectl logs -f demo-spring-boot-7f7769cfbd-k747j...Liveness ProbeLiveness ProbeReadiness ProbeLiveness ProbeReadiness ProbeLiveness ProbeLiveness ProbeReadiness Probe

Probe failures

Probe failures

Now in order to show how Kubernetes probes react in case of failures, let’s simulate failures on our endpoints. I extended the Kotlin code to have a Counter object which increments a value by 1 whenever a health check happens. When the value reaches 10, our service will not be considered ready anymore because it will return a 503 Service Unavailable. With 15, health check will fail and the pod will be restarted:

I also added a couple new properties to the deployment.yaml template just to make sure that the thresholds are set to 1 and it ended up looking like this:

Now let’s build and start our service again and see the probes happening. The terminal window on top shows the pod going from READY 0/1 to READY 1/1 after a successful readiness probe:

Pod is ready after log ‘Service is ready. Count 2’

After a couple seconds, pod goes back to READY 0/1 because counter already passed 10 so it will not receive traffic anymore:

Pod is not ready after count is higher than 10

And finally, pod will be restarted with the failure of a liveness probe:

Pod restarts when counter passes 15

All these checks caused our pod lifecycle to go from starting up when it’s not ready yet to running and ready to receive traffic. Then not ready, to a complete failure causing it to restart so that it could finally be ready again. All within the timespan of a minute:

~/code/demos/demo-spring-boot master > kubectl get podsNAME                               READY   STATUS    RESTARTS   AGEdemo-spring-boot-f8bd8cdff-k8csl   0/1     Running   0          9s~/code/demos/demo-spring-boot master > kubectl get podsNAME                               READY   STATUS    RESTARTS   AGEdemo-spring-boot-f8bd8cdff-k8csl   1/1     Running   0          19s~/code/demos/demo-spring-boot master > kubectl get podsNAME                               READY   STATUS    RESTARTS   AGEdemo-spring-boot-f8bd8cdff-k8csl   0/1     Running   0          48s~/code/demos/demo-spring-boot master > kubectl get podsNAME                               READY   STATUS    RESTARTS   AGEdemo-spring-boot-f8bd8cdff-k8csl   0/1     Running   1          60s~/code/demos/demo-spring-boot master > kubectl get podsNAME                               READY   STATUS    RESTARTS   AGE

Congratulations! You learned how Kubernetes probes work in practice and might have just won a “9” in your application up time.

Congratulations! You learned how Kubernetes probes work in practice and might have just won a “9” in your application up time.

Software Engineer freelancer / farmerconnect.com. Previously IBM, Accenture, HPE. linktr.ee/gateixeira