Running a Tomcat-based Spring Boot application in Docker

In one of the projects I am working on, I was tasked from taking a Tomcat application that ran on an EC2 server, and get it running with the Spring Boot framework in a Docker container. The twist is that the application needed read-write access to the resources folder within the project. In order to do that, the application needed to be run as an "exploded" WAR file (It had been run that way on the old server as well).

I accomplished this with this Docker file:

The WAR file is compiled with the "spring-boot-maven-plugin", and includes all the dependency JARs, so the application can be run stand-alone.

That entry in the POM is:

And the "org.springframework.boot.loader.WarLauncher" Class is what Spring Boot uses to bootstrap the applicatiton on the embedded Tomcat.

Kubernetes Readiness and Liveness with Apache Kafka REST Proxy

When setting up Readiness and Liveness checks in Kubernetes for Kafka connectors, the use of the httpGet described in my previous blog post (Kubernetes Readiness and Liveness with Spring Boot Actuator) is not an option because there is no endpoint to reference. These can be deployed with the Apache Kafka REST Proxy, which gets us on the right path, but doesn't quite work how we want in this respect.

The Kafka REST Proxy provides endpoints that allow one to get some basic status info about connectors. However, the standard Kubernetes httpGet calls use status code >= 200 and < 400 to determine the status, and since the Kafka REST status endpoint always provides a 200 status code, it is not possible to use this methodology to determine if a connector is down.

What we would like to do is check the content of the status call, and do a string comparison. For example, when the service is up, the status endpoint indicates that the state is "RUNNING":

# curl http://10.30.128.1:8083/connectors/mysql-kafka-connector/status

{"name":"mysql-kafka-connector","connector":{"state":"RUNNING","worker_id":"10.30.128.1:8083"},"tasks":[{"state":"RUNNING","id":0,"worker_id":"10.30.128.1:8083"}]}

We can pause the connector using this endpoint:

# curl -i -X PUT http://10.30.128.1:8083/connectors/mysql-kafka-connector/pause

HTTP/1.1 202 Accepted

And then the state is changed to PAUSED:

# curl http://10.30.128.1:8083/connectors/mysql-kafka-connector/status

{"name":"mysql-kafka-connector","connector":{"state":"PAUSED","worker_id":"10.30.128.1:8083"},"tasks":[{"state":"PAUSED","id":0,"worker_id":"10.30.128.1:8083"}]}


To accomplish this check, we can leverage the exec command probe:

readinessProbe: 

  exec: 

    command:

      - /bin/sh 

      - -c 

      - curl -s http://127.0.0.1:8083/connectors/mysql-kafka-connector/status | grep "RUNNING"

  initialDelaySeconds: 240

  periodSeconds: 5

  timeoutSeconds: 5 

  successThreshold: 1

  failureThreshold: 10 

livenessProbe:

  exec: 

    command: 

      - /bin/sh

      - -c 

      - curl -s http://127.0.0.1:8083/connectors/mysql-kafka-connector/status | grep "RUNNING"

  initialDelaySeconds: 300 

  periodSeconds: 60

  timeoutSeconds: 10 

  successThreshold: 1

  failureThreshold: 3

The the exec command allows us to execute a shell command. In this case:

• running the shell (/bin/sh)
• telling it to run a single command (-c)
• with the command being a cURL call to the specific connector status endpoint, and grepping for the string "RUNNING"

When the grep is successful, Kubernetes interprets this as a success. If the grep comes back empty (i.e., "RUNNING" is not found), then it gets viewed as a failure. You can then test this on the server by pausing the service in question as described above.

To get the service running again and start passing readiness and liveness again, then you will want to use the RESUME endpoint.

# curl -i -X PUT http://10.30.128.1:8083/connectors/mysql-kafka-connector/pause

HTTP/1.1 202 Accepted

Kubernetes Readiness and Liveness with Spring Boot Actuator

In Kubernetes, "readiness" is the indicator that the service is ready to accept traffic, and is only performed at the beginning of a pod's life cycle. "Liveness" is a periodic health check that should indicate that the service is still functional within the pod.
More details can be found in the documentation at
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/

Spring Boot is a stand-alone Spring environment well suited for micro services.
https://projects.spring.io/spring-boot/

Spring Boot has a bundle available called "Actuator" that exposes several helpful endpoints, one of which is a "health" endpoint. The simple version of this endpoint returns a simple JSON with a status UP or DOWN.
https://github.com/spring-projects/spring-boot/tree/master/spring-boot-actuator

One particular feature of the health endpoint that is useful is that besides the text indicator in the JSON response, it also signals up and down through the status code. UP = 200, and DOWN = 503 (Service Unavailable).

Putting this all together, the readiness and liveness configuration in the Kubernetes deployment YAML can look something like this:

readinessProbe:
httpGet:
  scheme: HTTP
  path: /health
  port: 8080
initialDelaySeconds: 240
periodSeconds: 5
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 10
livenessProbe:
httpGet:
  scheme: HTTP
  path: /health
  port: 8080
initialDelaySeconds: 300
periodSeconds: 60
timeoutSeconds: 10
successThreshold: 1
failureThreshold: 3

In this example, we instruct Kubernetes to wait 240 seconds to allow the application to start before performing the check. It will retry up to 10 times with a five second pause between each try. It will wait for a maximum of 5 seconds for a result to be returned. After 10 failures, the pod will be restarted.

For the liveness check, we instruct Kubernetes to wait 300 seconds, and to check every 60 seconds. If three failures occur in a row, the pod will be restarted.