9/11/2022
Kubernetes CPU Allocation for Pods
When it comes to Kubernetes resource allocation, CPU allocation is considered as a critical factor for application/pod performance.
According to Kubernetes spec, there is a way to do this using Limits and Requests.
Resource Management for Pods and Containers | Kubernetes
Assign CPU Resources to Containers and Pods | Kubernetes
But how do we make sure or derive a value for both CPU requests and limits.
I found below article:
For the love of god, stop using CPU limits on Kubernetes (updated)
Summary
Here is the FUN part :) .
Do pods always get the CPU requested by their CPU request ? Is it guaranteed?
Kubernetes CPU requests explained – Mechpen
https://en.wikipedia.org/wiki/Completely_Fair_Scheduler
How is kernel CFS working?
In CFS, every running entity, a process or a task group, has a virtual runtime (vruntime) which accounts for the entity's CPU usage.
The scheduling goal of CFS is to keep the vruntime of all running entities to be the same.
Thus the allocated CPU time of an entity is proportional to its weight
Each entity gets a portion of cpu.shares proportional to the task group's running load on the CPU
Summary:
CPU requests can't be guaranteed since CPU time depends on process load.
Kubernetes QoS classes (Quality of Service)
Configure Quality of Service for Pods | Kubernetes
What should be the best way to configure CPU limits and requests
Will adding K8S CPU limit reduce service performance? - SoByte
Set CPU requests and limits to the same value
Add an HPA (Horizontal Pods Auto scaling) that allows Pods to automatically scale up and down based on load.
Use circuit breaker pattern.
8/14/2022
Kubernetes Ingress
- Once installed increased the number of replicas at ngress-nginx-controller deployment to 2
apiVersion: apps/v1
kind: Deployment
metadata:
name: hostname-app
spec:
replicas: 2
selector:
matchLabels:
app: hostname-app
template:
metadata:
labels:
app: hostname-app
spec:
containers:
- name: hostname-app
image: k8s.gcr.io/serve_hostname:1.1
---
apiVersion: v1
kind: Service
metadata:
name: hostname-svc
spec:
ports:
- port: 80
targetPort: 9376
protocol: TCP
selector:
app: hostname-app
kubectl apply -f service-deployment.yaml
.jpg)
- You can see we have to set http Host header as Ingress-Rules host configuration.
- Also you can see the pod host name in the response is not same every time.
.jpg)
- As you can see, both controllers received requests, which means Nginx-Controller-Service has load balanced the requests between two controller pods.
.jpg)
- As you can see , now response has same pod name, for every request.
- Also requests load balanced between each controllers.
- Kubernetes Ingress is abstract service, real implementation is based on ingressClassName: nginx .
- What ever the changes made to Ingress-Rules will be reflected in Ingress-Controller Pods.
- If you don't use correct http Host header, you will get HTTP 404 response, which is the default response from Ingress-Inginx-Controller.
- Ingress-Controller will generate session cookie header for the first response based on pod name/ip (hash value), and it will included in subsequent request.
- So second request can be routed to previous same pod that initial request made.
3/26/2022
AWS SNS and Lambda Lesson Learn
The purpose of this blog post is to share my recent experience with AWS budget alerting, using above AWS stack.
We can create an alert to trigger alert message when certain threshold increase at AWS budget.
Then what happen is, that alert message will publish to SNS topic and same message will be consumed by Lambda function.
We can do transform the message and sent to Microsoft Team channel through a Webhook. Please refer this document for more details about this approach.
https://aws.amazon.com/premiumsupport/knowledge-center/sns-lambda-webhooks-chime-slack-teams/
Steps
1. Create AWS Budget Alert
a. Create AWS Budget
https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-create.html
b. Create AWS Alert
2. Create Lambda function
https://docs.aws.amazon.com/lambda/latest/dg/getting-started-create-function.html
Now in our case we are using Python Lambda function. so please refer below section.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html
3. Subscribe Lambda function to SNS topic
https://docs.aws.amazon.com/sns/latest/dg/sns-lambda-as-subscriber.html
4. Analyzing the sample code from below document.
https://aws.amazon.com/premiumsupport/knowledge-center/sns-lambda-webhooks-chime-slack-teams/
#!/usr/bin/python3.6
import urllib3
import json
http = urllib3.PoolManager()
def lambda_handler(event, context):
url = "https://outlook.office.com/webhook/xxxxxxx"
msg = {
"text": event['Records'][0]['Sns']['Message']
}
print('Type of message', type(msg))
encoded_msg = json.dumps(msg).encode('utf-8')
resp = http.request('POST',url, body=encoded_msg)
print("=====After encode======",encoded_msg)
print({
"status_code": resp.status,
"response": resp.data
})
5. Unit Testing
6. Integration Testing
7. Debug with Cloud Watch Logs.
8. Comparison of two logs
As you can see the different of above two logs, the success encoded JSON message, child node wrapped with a double quotes.
9. Improve Lambda Function to Transform Output Message.
Now we have improved the Lambda function to transform the output message to another JSON.
#!/usr/bin/python3.6
import urllib3
import json
import ast
http = urllib3.PoolManager()
def lambda_handler(event, context):
url = "https://outlook.office.com/webhook/xxxxxxx"
alert_dict = event['Records'][0]['Sns']['Message']
if isinstance(alert_dict, str):
print('Type of variable is a string')
alert_dict = ast.literal_eval(alert_dict)
else:
print('Type is variable is not a string')
#Construct new json format
title = "Budget Exceed Alert "
limit = (alert_dict['budget']['budgetLimit']['amount'])
threshold = (alert_dict['action']['actionThreshold']['value'])
print("===========", type(limit))
print("===========", threshold)
actual_amount = (limit * threshold) /100
alert_json = dict(AlertTitle=title,TimeStamp=alert_dict['timeStamp'],BudgetName=alert_dict['budget']['budgetName'],BudgetType=alert_dict['budget']['budgetType'],BudgetAmount=alert_dict['budget']['budgetLimit']['amount'],AlertThreshold=alert_dict['action']['actionThreshold']['value'],ActualAmount=actual_amount,AlertType=alert_dict['action']['notificationType'],Message=alert_dict['message'])
summery = {
"text": json.dumps(alert_json)
}
print("===========", type(summery))
print("=====Before JSON======", summery)
json_object = json.dumps(summery)
encoded_msg = json_object.encode('utf-8')
print("=====After encode======",encoded_msg)
resp = http.request('POST',url, body=encoded_msg)
print({
"status_code": resp.status,
"response": resp.data
})
10. Unit Testing
11. Integration Testing
12. Conclusion
- Now both, unite test and integration test will generate same output JSON.
- The way create JSON is important in Python and it's bit tricky.
summery = {
"text": json.dumps(alert_json)
}
- Did you notice one difference in both testing?
3/06/2022
Learning Camunda/Zeebe by Example - SAGA Pattern
Now let's build the project with maven command : mvn clean install
https://github.com/dhanuka84/SAGA-Microservices-Zeebe/tree/main/src/zeebe-saga-spring-boot
Then Build the docker images with maven spring-boot plugin:
mvn spring-boot:build-image
- To deploy Zeebe cluster, you have to go through same steps given in previous post.
Deploy MongoDB Cluster
MONGO_REPLICASET_HOST=mongo docker-compose -f src/zeebe-saga-spring-boot/docker/docker-compose-mongo.yaml up
Deploy Microservices
docker-compose -f src/zeebe-saga-spring-boot/docker/docker-compose-micros.yaml up
Test Microservices Health with Actuator ( status, liveness, readiness)
dhanuka@dhanuka:~$ curl http://localhost:8081/actuator/health | jq '. | {status: .status, liveness: .components.livenessState.status, readiness: .components.readinessState.status,}'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 352 100 352 0 0 592 0 --:--:-- --:--:-- --:--:-- 593
{
"status": "UP",
"liveness": "UP",
"readiness": "UP"
}
dhanuka@dhanuka:~$ curl http://localhost:8083/actuator/health | jq '. | {status: .status, liveness: .components.livenessState.status, readiness: .components.readinessState.status,}'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 352 100 352 0 0 824 0 --:--:-- --:--:-- --:--:-- 822
{
"status": "UP",
"liveness": "UP",
"readiness": "UP"
}
dhanuka@dhanuka:~$ curl http://localhost:8084/actuator/health | jq '. | {status: .status, liveness: .components.livenessState.status, readiness: .components.readinessState.status,}'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 352 100 352 0 0 675 0 --:--:-- --:--:-- --:--:-- 675
{
"status": "UP",
"liveness": "UP",
"readiness": "UP"
}
dhanuka@dhanuka:~$ curl http://localhost:8082/actuator/health | jq '. | {status: .status, liveness: .components.livenessState.status, readiness: .components.readinessState.status,}'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 352 100 352 0 0 820 0 --:--:-- --:--:-- --:--:-- 822
{
"status": "UP",
"liveness": "UP",
"readiness": "UP"
}
Deploy Zeebe BPMN Process
dhanuka@dhanuka:~$ zbctl deploy workflows/saga-example.bpmn --insecure
{
"key": "2251799813703663",
"processes": [
{
"bpmnProcessId": "trip-booking",
"version": 1,
"processDefinitionKey": "2251799813703662",
"resourceName": "workflows/saga-example.bpmn"
}
]
}
Create a booking via Rest Call to Booking Microservice
dhanuka@dhanuka:~$ curl --location --request POST 'http://localhost:8081/booking/' \
--header 'Content-Type: application/json' \
--data-raw '{
"id" : "0",
"clientId":"123",
"resourceId":"987",
"fromDate":"2021-02-22T14:52:44.494264+01:00",
"toDate":"2021-03-06T14:52:44.495451+01:00",
"createdAt":"2021-02-10T14:52:44.495469+01:00",
"active":false
}'
{"id":"0","clientId":"123","houseBookingId":"642c957b-f7ed-45d9-8719-b11fa451dbfa","carBookingId":"dbb44830-2cb6-4c02-ab5d-bf0850e69bb0","flightBookingId":"b573f7ca-cab9-48ab-9bdf-cc08e5e199ae","fromDate":"2021-02-22T13:52:44.494264Z","toDate":"2021-03-06T13:52:44.495451Z","createdAt":"2021-02-10T13:52:44.495469Z","active":false}
- Once you login to Zeebe dashboard via http://localhost:8080/ , you can select the process id (trip-booking) and then version, then you can select the process instance
- Then you can see the happy path of the work flow.
Java Code Explanation
1. Where is Zeebee Process Instance Created?
- As you can see, the instance was created in line 34.
Key Points:
Line 34 : Spring WebFlux, which provides reactive programming support for web applications
https://www.baeldung.com/spring-webflux
Line 37: Creating a variable called bookingResult with Booking data.
Line 40: Usage of Completable Futures.
Line 49: Save the Trip Booking entity when received hotel booking id, car booking id and flight booking id.
So all the operations within and outside the microservice will be asynchronous and reactive.
2. Zeebee Task polling/requesting for jobs, then execute and response back to broker.
- Let’s take Hotel Booking as an example.
Key Points:
Line 41,42,43 : Access job data.
Line 45 : Validate the job based on headers
Line 50: Access Booking Info, based on requestName (bookingResult) variable value, which was created when process instance creation.
Line 64: Create Hotel Booking and reactively update Zeebe with response.
- Note that now the resultName variable value of response is bookHotelResult.
Failure Path Testing
- You have to change the BPMN configuration to enable simulateError as below.
Deploy workflow process next version
dhanuka@dhanuka:~$ zbctl deploy workflows/saga-example.bpmn --insecure
{
"key": "2251799813703663",
"processes": [
{
"bpmnProcessId": "trip-booking",
"version": 2,
"processDefinitionKey": "2251799813703662",
"resourceName": "workflows/saga-example.bpmn"
}
]
}
Create a booking via Rest Call to Booking Microservice
dhanuka@dhanuka:~$ curl --location --request POST 'http://localhost:8081/booking/' \
--header 'Content-Type: application/json' \
--data-raw '{
"id" : "0",
"clientId":"123",
"resourceId":"987",
"fromDate":"2021-02-22T14:52:44.494264+01:00",
"toDate":"2021-03-06T14:52:44.495451+01:00",
"createdAt":"2021-02-10T14:52:44.495469+01:00",
"active":false
}'
- Now you can see the failure path of the workflow
- We failed the work flow from Flight Booking task.
- The failure will be propagated to other tasks which is polling on same instance_id .
Failure Path Java Explanation
Key Points
