Spring Boot + K8S : Event-Driven Microservices

 

The purpose of this blog post is to explain, how we can achieve event-driven microservices with spring boot and kubernetes.

The original source of this blog post is below youtube channel.

 

• Please find the source code for this post in here:

https://github.com/dhanuka84/kbe-sb-microservices/tree/fixes

What is the difference

 https://github.com/dhanuka84/kbe-sb-microservices/pull/1/commits

1. Fixed unit test failure due to SQL script not compatible with H2 DB

2. Images can be build locally and use same docker images with minikube.

3. MySQL datasource username configuration fix.

4. Simple and Short explanation.

How to build and test

1. Build  java project & run unit tests

mvn clean package

2.  Start minikube

start minikube

# this is important otherwise k8s cluster can't pull the docker image

eval $(minikube docker-env)

3. Build docker images

mvn spring-boot:build-image

Deploy containers in kubernetes

minikube tunnel

kubectl apply -f ./kbe-sb-microservices/k8s-scripts

Explaining the flow and components

1. Gateway : this will route API path to the correct URI

2. Order-Service  Micro-Service: placing an order

• Let's take order service micro-service and it's placeOrder() method/API.

• Controller will call the BeerOrder service and then it's business logic

• Then BeerOrderManager will call state machine to start the spring boot event bus flow.

• According to state machine configuration, next  event is BeerOrderEventEnum.VALIDATE_ORDER and next action is validateBeerOrder .

• Now this will trigger the validateBeerOrder action and it will send ValidateBeerOrderRequest to VALIDATE_ORDER_QUEUE JMS queue.

• Now our second micro-service called Beer-Service will listen to this VALIDATE_ORDER_QUEUE, and validate the beer order and send the result to another JMS queue called VALIDATE_ORDER_RESULT_QUEUE

Then again, Order-Service micro-service will listen to validation result queue and process the order and update the status of the order.

Monitoring

• Filebeat will collect all the docker logs and ingest them into Elastisearch.
• So then we can monitor application logs using Kibana.

Key takeaways

• Microservices communicated with each other in an event-driven manner using JMS queue.
• Spring Boot state machine (event bus) has been used to simplifying the business flow of the application. 
• Using above two techniques, events and states has been managed both in standalone and distributed way.
• Centralized logging can be used to trouble shoot and monitor distributed applications.

Spring Boot Exception Handling & Validation

 

The purpose of the blog post is to explain how to do exception handling and data transferring.

Please find the source code for above application. You have to select branch called improved .

This is the improved version of the code explained in previous blog post.

Rest Controller

• Only Rest API related coding included.
• DTO send to service layer

Service Implementation

• Validation happen inside the business logic .
• Exception propagate to controller level.
• DTO used to communicate with web layer.

Controller Advice

• This will capture each exception and send relevant response to client through controller.

Testing

I have used postman to test Rest API.

• You can see that HTTP response code is 204 : No Content
• We are sending timestamp, which is older than 60 seconds time period
• Inside the service layer, we have validated this and throw TimeExceeded  Exception.
• According to our controller advice, we should send HTTP response code 204 : No Content

Layered Architecture of Spring Boot Application

 

The purpose of this blogpost is to explain main components and layers of a simple spring boot application and how to develop them.

Please find the source code for above application.

Rest Controller

https://github.com/dhanuka84/sb-transaction-service/blob/main/src/main/java/com/sb/transaction/service/web/controllers/TransactionController.java

• This is sample post method.
• Request body can be validated with @Validate annotation.
• HTTP status code for success scenario can be mentioned with @ResponseStatus annotation.
• URI is /sb/transactions.
• We can use ResponseStatusException for other HTTP status code based on each scenario.

Service Implementation

https://github.com/dhanuka84/sb-transaction-service/blob/main/src/main/java/com/sb/transaction/service/services/TransactionServiceImpl.java

• Logging can be initialized by lombok @Slf4j annotation .
• Constructor with argument can be created with lombok @RequiredArgsConstructor annotation .
• We can map DTO to domain object and wise versa using transactionMapper .
• Using transactionRepository we can persist the domain object in database through JPA/JDBC .

JPA Repository/Backend Layer

https://github.com/dhanuka84/sb-transaction-service/blob/main/src/main/java/com/sb/transaction/service/repositories/TransactionRepository.java

• Most of the commonly used CRUD operations based on domain entity and primary key can be found from JpaRepository .
• For the native query, we have used a custom method. 

https://github.com/dhanuka84/sb-transaction-service/blob/main/src/main/java/com/sb/transaction/service/domain/Transaction.java

• Native query can be found in the same domain entity class Transaction .
• The name of the native query should be : (Entity class name) . ( Domain Repository method name) - @NamedNativeQuery(name = "Transaction.findStatistics",
• @SqlResultSetMapping(name = "Mapping.StatisticsDto" : This will map sql result set to DTO.
• Please note that constructor of DTO have to create manually since order of argument data type should match with @SqlResultSetMapping columns.

public StatisticsDto(BigDecimal sum, BigDecimal avg, BigDecimal max, BigDecimal min,Long count) 

Spring Boot Application : Where Every things Begins

https://github.com/dhanuka84/sb-transaction-service/blob/main/src/main/java/com/sb/TransactionApplication.java

Data Mapper

https://github.com/dhanuka84/sb-transaction-service/blob/main/src/main/java/com/sb/transaction/service/web/mappers/TransactionMapper.java

• We can use multiple mapper classes like DateMapper.class .
•  componentModel = "spring" should be mentioned, then only mapstruct can be used within spring.
• DateMapper useful when we deserialize JSON to java object and wise versa since parser can identify Java 8 Date Time package.

Mockito with Spring Boot

https://github.com/dhanuka84/sb-transaction-service/blob/main/src/test/java/com/sb/transaction/service/web/controllers/TestTransactionControllerTest.java

 

• Don't forget to import correct @Test annotation.

                import org.junit.jupiter.api.Test;

Spring Configuration

https://github.com/dhanuka84/sb-transaction-service/blob/main/src/main/resources/application.properties

• Server port and name can be configured here

Application and Docker image building

Kubernetes : Vault Vs Secrets

hashicorp.com: Vault & Kubernetes: Better Together

1. Using Static secrets 

(can apply TTL)

2. Using Dynamic secrets 

(Secrets are also easy to rotate and revoke; if an employee leaves your organization, you can easily and securely revoke their access.)

3. Using transit encryption in Kubernetes

(Vault provides “encryption as a service,” encrypting data in transit (with TLS) and at rest (using AES 256-bit CBC encryption).)

 

 

Key k8s components

 

1. MutatingAdmissionWebhook

An admission controller is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object, but after the request is authenticated and authorized.

Using Admission Controllers

 

2. Service Account

A service account provides an identity for processes that run in a Pod.

Configure Service Accounts for Pods

 

How it works

 

The Service Account assigns an identity to a pod, which is used to grant access to secrets in Vault whereas the webhook is used to inject an init container into a Pod that mounts the Secret from Vault to a temporary volume. 

 

 

Example:

1. Application Pod

Init Containers:

  vault-agent-init:

    Container ID:  docker://eb9a90c1b4105102e6180bd622e954e90fc34259da37b167c9e1ea5718152db1

    Image:         vault:1.7.0

    Image ID:      docker-pullable://vault@sha256:635cf1c3f9b10fe03aad375f94cc61f63d74a189662165285a8bf1c189ea04b8

    Port:          <none>

    Host Port:     <none>

Containers:

  orgchart:

    Container ID:   docker://425f5a5b26921ee6873233b714156201754489e1edf0b59f62bc9b928599c887

    Image:          jweissig/app:0.0.1

    Image ID:       docker-pullable://jweissig/app@sha256:54e7159831602dd8ffd8b81e1d4534c664a73e88f3f340df9c637fc16a5cf0b7

   vault-agent:

    Container ID:  docker://4ddb15c3cdaa9c215031197dfdd62e43a842b92559e1d9a0596b184a21f144c7

    Image:         vault:1.7.0

    Image ID:      docker-pullable://vault@sha256:635cf1c3f9b10fe03aad375f94cc61f63d74a189662165285a8bf1c189ea04b8

 

 

2. Vault

Containers:

  vault:

    Container ID:  docker://7194b9fd5e0c071ad4dfe12b4fd622c64bef8c324b70ed503224fc3807cb2ab4

    Image:         vault:1.7.0

    Image ID:      docker-pullable://vault@sha256:635cf1c3f9b10fe03aad375f94cc61f63d74a189662165285a8bf1c189ea04b8

    Ports:         8200/TCP, 8201/TCP, 8202/TCP

    Host Ports:    0/TCP, 0/TCP, 0/TCP

3. Vault-agent-injector

Containers:

  sidecar-injector:

    Container ID:  docker://a53b4895a6479c16c5b3e9248c15fa85388f1ea3167a36dff81d4b75d954ff05

    Image:         hashicorp/vault-k8s:0.9.0

    Image ID:      docker-pullable://hashicorp/vault-k8s@sha256:65731b0513c95f683ee52528e6ccf24f6de0092700e869cdc5ff5d8354b5d86e

   Environment:

      AGENT_INJECT_LISTEN:              :8080

      AGENT_INJECT_LOG_LEVEL:           info

      AGENT_INJECT_VAULT_ADDR:          http://vault.default.svc:8200

      AGENT_INJECT_VAULT_AUTH_PATH:     auth/kubernetes

      AGENT_INJECT_VAULT_IMAGE:         vault:1.7.0

      AGENT_INJECT_TLS_AUTO:            vault-agent-injector-cfg

      AGENT_INJECT_TLS_AUTO_HOSTS:      vault-agent-injector-svc,vault-agent-injector-svc.default,vault-agent-injector-svc.default.svc

      AGENT_INJECT_LOG_FORMAT:          standard

      AGENT_INJECT_REVOKE_ON_SHUTDOWN:  false

 

 

• Agent Inject: Agent Inject is a mutation webhook controller that injects Vault Agent containers into pods meeting specific annotation criteria. (Requires Vault 1.3.1+)

 

Demo Steps

Clone github repo

$ git clone https://github.com/hashicorp/vault-guides.git $ cd  vault-guides/operations/provision-vault/kubernetes/minikube/vault-agent-sidecar

Install Vault & Vault-Agent-Injector

helm repo add hashicorp https://helm.releases.hashicorp.com helm install vault hashicorp/vault --set "server.dev.enabled=true"

NAME                                    READY   STATUS    RESTARTS   AGE vault-0                                 1/1     Running   0          8h vault-agent-injector-67887bbd49-skk4s   1/1     Running   0          8h

Enable Key-Value Secret Engine

kubectl exec -it vault-0 -- /bin/sh vault secrets enable -path=internal kv-v2

 

Create a secret at path internal/database/config with a username and password

 

vault kv put internal/database/config username="db-readonly-username" password="db-secret-password" exit

 

 

Enable the Kubernetes authentication method

 

kubectl exec -it vault-0 -- /bin/sh vault auth enable kubernetes

 

Enables clients to authenticate with a Kubernetes Service Account Token.

Configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate.

 

vault write auth/kubernetes/config \     token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \     kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt Success! Data written to: auth/kubernetes/config

 

Grant read capability to read from path: internal/data/database/config

 

vault policy write internal-app - <<EOF path "internal/data/database/config" {   capabilities = ["read"] } EOF Success! Uploaded policy: internal-app

 

Create a Kubernetes authentication role named internal-app

vault write auth/kubernetes/role/internal-app \     bound_service_account_names=internal-app \     bound_service_account_namespaces=default \     policies=internal-app \     ttl=24h Success! Data written to: auth/kubernetes/role/internal-app exit

 

Define a Kubernetes service account

kubectl get serviceaccounts NAME                   SECRETS   AGE default                1         11h vault                  1         11h vault-agent-injector   1         11h

Create a service account named internal-app .

$ cat service-account-internal-app.yml apiVersion: v1 kind: ServiceAccount metadata:   name: internal-app

 

kubectl apply --filename service-account-internal-app.yml

 

$ kubectl get serviceaccounts NAME                   SECRETS   AGE default                            1         11h internal-app                      1         9h vault                                 1         11h vault-agent-injector         1         11h

 

Launch an application

$ kubectl apply --filename deployment-orgchart.yml deployment.apps/orgchart created

 

 

kubectl get pods NAME                                                            READY   STATUS    RESTARTS   AGE orgchart-69697d9598-l878s             1/1             Running               0          6h vault-0                                                              1/1             Running               0          8h vault-agent-injector-67887bbd49-skk4s      1/1             Running               0          8h

 

Inject secrets into the pod

$ cat patch-inject-secrets.yml spec:   template:     metadata:       annotations:         vault.hashicorp.com/agent-inject: "true"         vault.hashicorp.com/role: "internal-app"         vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"

 

$ kubectl patch deployment orgchart --patch "$(cat patch-inject-secrets.yml)" deployment.apps/orgchart patched

$ kubectl get pods NAME                                                            READY       STATUS     RESTARTS   AGE orgchart-55c76c489d-6r7vc                               0/2             Init:0/1               0              23s orgchart-69697d9598-l878s                               1/1             Running             0              20m vault-0                                                                1/1             Running             0              78m vault-agent-injector-5945fb98b5-tpglz              1/1             Running             0              78m

 

$ kubectl get pods NAME                                                                    READY   STATUS    RESTARTS   AGE orgchart-55c76c489d-6r7vc                                       2/2         Running                   0          6h payroll                                                                        2/2         Running                   0          5h45m vault-0                                                                        1/1         Running                   0          8h vault-agent-injector-67887bbd49-skk4s                    1/1         Running                   0          8h

 

 

Display the secret written to the orgchart container.

$ kubectl exec \     $(kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}") \     --container orgchart -- cat /vault/secrets/database-config.txt data: map[password:db-secret-password username:db-readonly-user] metadata: map[created_time:2019-12-20T18:17:50.930264759Z deletion_time: destroyed:false version:2]

 

Pod with annotations

$ cat pod-payroll.yml apiVersion: v1 kind: Pod metadata:   name: payroll   labels:     app: payroll   annotations:     vault.hashicorp.com/agent-inject: "true"     vault.hashicorp.com/role: "internal-app"     vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"     vault.hashicorp.com/agent-inject-template-database-config.txt: |       {{- with secret "internal/data/database/config" -}}       postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard       {{- end -}} spec:   serviceAccountName: internal-app   containers:     - name: payroll       image: jweissig/app:0.0.1

 

kubectl apply --filename pod-payroll.yml

kubectl get pods NAME                                                                            READY       STATUS            RESTARTS   AGE orgchart-55c76c489d-6r7vc                                              2/2                 Running                   0          6h payroll                                                                               2/2                 Running                   0          5h45m vault-0                                                                               1/1                 Running                   0          8h vault-agent-injector-67887bbd49-skk4s                           1/1                 Running                   0          8h

$ kubectl exec \     payroll \     --container payroll -- cat /vault/secrets/database-config.txt postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard

References:

Injecting Vault Secrets Into Kubernetes Pods via a Sidecar

Injecting Secrets into Kubernetes Pods via Vault Agent Containers | Vault

Static Secrets: Key/Value Secrets Engine | Vault

Configure Service Accounts for Pods

Transform Secrets Engine | Vault

The Kubernetes API call is coming from inside the cluster!

High Availability

HashiCorp EKS Vault on AWS - Quick Start

End-to-End Automation for Vault on Kubernetes Using the Operator Pattern