Learning Kubernetes by Example : Kafka Cluster Auto Scaling and Re-balancing

This post is the next episode of previous post (http://dhanuka84.blogspot.com/2020/04/learning-kubernetes-by-example-kafka.html) .
In this blog post we are mainly focus on how to trigger alerts with Prometheus and how kafka-operator react based on those alerts. Finally how Cruise-Control rebalance the kafka cluster based on the same alert.
Please note that, since this post is some kind of simple version of two references mentioned in the bottom, I have hijacked the diagram from same article :).

Key Points:

1. Once we provisioned manifests (kafkacluster-prometheus.yaml) for key Prometheus components, Prometheus operator will create kafka-alerts, kafka-prometheus and kafka-operator-alertmanager .
2. When you provision kafka cluster (Envoykafkacluster.yaml),  monitoringConfig will deploy Prometheus jmx java agents in brokers. So Prometheus can monitor kafka cluster.
3. Based on the metrics and relevant rules that has been configured, Prometheus will generate alerts to Kafka-Operator. Then Kafka-Operator will create kafka-broker pods (according to below test case).
4.   Kafka-Operator will propagate same alert to Cruise-Control, then Cruise-Control will do cluster re-balancing on Kafka brokers. 

Steps:

Follow the previous blogpost (http://dhanuka84.blogspot.com/2020/04/learning-kubernetes-by-example-kafka.html) and then follow additional steps mentioned below.


1. Deploy Prometheus operator

kubectl apply -n default -f config/samples/bundle.yaml

2. Deploy Kafka cluster related Prometheus workloads

kubectl create -n default -f config/samples/kafkacluster-prometheus.yaml

3. Port forward to both Cruise-Control WebUI and Prometheus WebUI.

kubectl port-forward -n kafka svc/kafka-cruisecontrol-svc 8090:8090 &

kubectl port-forward -n default pod/prometheus-kafka-prometheus-0 9090:9090

4. Then login to WebUIs

http://localhost:9090/alerts

http://localhost:8090/

5. Create kafka-internal pod

kubectl create -n kafka -f - <<EOF

apiVersion: v1

kind: Pod

metadata:

  name: kafka-internal

spec:

  containers:

  - name: kafka

    image: wurstmeister/kafka:2.12-2.1.0

    # Just spin & wait forever

    command: [ "/bin/bash", "-c", "--" ]

    args: [ "while true; do sleep 3000; done;" ]

EOF

6. Create kafka topic with 1001 partitions.

bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper example-zookeepercluster-client.zookeeper:2181 --create --topic alertblog --partitions 101 --replication-factor 3

Created topic "alertblog".

Observations & Explanations

In this kafka-prometheus manifest file, we have configured the rules and out of these we are testing PartitionCountHigh rule/alert.

You can see that above manifest used, one of the kafka broker metrics called, "kafka.server:type=ReplicaManager,name=PartitionCount"

Number of partitions on this broker. This should be mostly even across all brokers.

So if the broker partition count exceed 100, then a triggered will be fired and new broker will be created. Finally re-balanced will be executed.

1. Before triggering the alert

K8s dashboard

 

Cruise-Control

Prometheus Alerts

2. After alert is triggered.

Kafka-Operator Logs

Prometheus-kafka pod logs

Kafka-Cruise-Control pod logs

Prometheus alerts pending status

Prometheus alerts firing status

Cruise-Control after broker added

K8s dashboard after broker added

References:

https://medium.com/@prune998/banzaicloud-kafka-operator-and-broker-autoscaling-1c7324260de1

https://banzaicloud.com/blog/kafka-alert/

Learning Kubernetes by Example : Kafka Cluster External Access

The whole purpose of this blog post is understand fundamentals of Kubernetes by setting up kafka cluster inside kubernetes. But this blog post does not explain each Kubernetes components and its purpose. Since you can find many resources to learn those theoretical aspect of kubernetes, I didn't include those into this post.

Key points:

1. Use Banzaicloud Kafka operator.
2. Kafka operator use Pods, ConfigMaps, Persistence Volume Claims.
3. Kubernetes Loadbalancer and Envoy service proxy.
4. Headless services within cluster scope.
5. User Replica set for brokers.
6. Service proxy (envoy pods)
7. Kube-Api-Server APIs are OpenAPI based.
8. In Kubernetes all most everything declare/configure based on OpenAPI

With the Banzai Cloud Kafka operator we can:

• modify the configuration of unique Brokers
• remove specific Brokers from clusters
• use multiple Persistent Volumes for each Broker

Steps:

Follow previous blogpost (https://dhanuka84.blogspot.com/2020/03/learn-kubernetes-by-example-kafka.html) till step 9. Then execute below steps to setup kafka cluster with LoadBalancer service.

Before creating kafka cluster we need to enable load balancing capability in minikube.
Minikube does not have a load balancer implementation, thus our envoy service will not get an external IP and the operator will get stuck at this point.

kubectl run minikube-lb-patch --replicas=1 --image=elsonrodriguez/minikube-lb-patch:0.1 --namespace=kube-system

10. Install Kafka cluster

kubectl create -n kafka -f  config/samples/Envoykafkacluster.yaml

11. Start k8s dashboard

dhanuka@dhanuka:~$ minikube dashboard


http://127.0.0.1:46205/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ 

12. Creating a kafka topic

kubectl -n kafka run kafka-topics -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-topics.sh --zookeeper example-zookeepercluster-client.zookeeper:2181 --topic my-topic --create --partitions 1 --replication-factor 1

dhanuka@dhanuka:~$ minikube service envoy-loadbalancer -n kafka

13. Create a produce and generate some messages

dhanuka@dhanuka:~$ kubectl -n kafka run kafka-producer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-producer.sh --broker-list envoy-loadbalancer:19090 --topic my-topic >to >load >balancer >to >load >balancer >

kubectl -n kafka run kafka-producer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-producer.sh --broker-list 192.168.39.180:32652 --topic my-topic

If you don't see a command prompt, try pressing enter.

>>hi
>I am
>remote

>

14. Create a consumer and consume generated messages

dhanuka@dhanuka:~$kubectl -n kafka run kafka-consumer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server envoy-loadbalancer:19090 --topic my-topic --from-beginning If you don't see a command prompt, try pressing enter.

to load balancer to load balancer

 kubectl -n kafka run kafka-consumer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.39.180:32652 --topic my-topic --from-beginning
If you don't see a command prompt, try pressing enter.
to
load
balancer
to
load
balancer

hi
I am

remote

Explanation: High level Kubernetes Architecture

Dashboard

http://127.0.0.1:46205/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/#/overview?namespace=_all

Explanation: Connectivity between zookeeper and kafka brokers

Services in zookeeper Namespace

dhanuka@dhanuka:~/research/kafka/kafka-operator$ kubectl get svc -n zookeeper

NAME                                                TYPE        CLUSTER-IP      EXTERNAL-IP     PORT(S)                                   AGE
example-zookeepercluster-client        ClusterIP   10.102.39.217     <none>                   2181/TCP                                   90m
example-zookeepercluster-headless   ClusterIP    None                   <none>                   2888/TCP,3888/TCP                  90m


Services in kafka Namespace

dhanuka@dhanuka:~/research/kafka/kafka-operator$ kubectl get svc -n kafka

NAME                          TYPE                CLUSTER-IP      EXTERNAL-IP      PORT(S)                                                 AGE
envoy-loadbalancer       LoadBalancer   10.108.163.32      10.108.163.32  19090:32440/TCP,19091:32652/TCP,19092:30377/TCP                                              3h10m


kafka-headless                ClusterIP           None                    <none>          29092/TCP,29093/TCP,9020/TCP                               3h10m

Let's start with Kafka cluster configuration

Go to below configuration file

https://github.com/dhanuka84/kafka-operator/blob/master/config/samples/Envoykafkacluster.yaml

You can see that, kafka connect to zookeeper cluster using "example-zookeepercluster-client " service.

In the Dashboard go to "Custom Resource Definition"

Then click "KafkaCluster"

Then if you click Edit button as highlighted below you can see the full spec and related properties which are already initialized in  https://github.com/dhanuka84/kafka-operator/blob/master/config/samples/Envoykafkacluster.yaml  file.

References:

https://v1-17.docs.kubernetes.io/docs/concepts/overview/components/

https://banzaicloud.com/docs/supertubes/kafka-operator/features/