Grasshopper ----------> Master ( D 2 ): Learning Kubernetes by Example : Kafka Cluster External Access

The whole purpose of this blog post is understand fundamentals of Kubernetes by setting up kafka cluster inside kubernetes. But this blog post does not explain each Kubernetes components and its purpose. Since you can find many resources to learn those theoretical aspect of kubernetes, I didn't include those into this post.

Key points:

1. Use Banzaicloud Kafka operator.
2. Kafka operator use Pods, ConfigMaps, Persistence Volume Claims.
3. Kubernetes Loadbalancer and Envoy service proxy.
4. Headless services within cluster scope.
5. User Replica set for brokers.
6. Service proxy (envoy pods)
7. Kube-Api-Server APIs are OpenAPI based.
8. In Kubernetes all most everything declare/configure based on OpenAPI

With the Banzai Cloud Kafka operator we can:

modify the configuration of unique Brokers
remove specific Brokers from clusters
use multiple Persistent Volumes for each Broker

Steps:

Follow previous blogpost (https://dhanuka84.blogspot.com/2020/03/learn-kubernetes-by-example-kafka.html) till step 9. Then execute below steps to setup kafka cluster with LoadBalancer service.

Before creating kafka cluster we need to enable load balancing capability in minikube.
Minikube does not have a load balancer implementation, thus our envoy service will not get an external IP and the operator will get stuck at this point.

kubectl run minikube-lb-patch --replicas=1 --image=elsonrodriguez/minikube-lb-patch:0.1 --namespace=kube-system

10. Install Kafka cluster

kubectl create -n kafka -f config/samples/Envoykafkacluster.yaml

11. Start k8s dashboard

dhanuka@dhanuka:~$ minikube dashboard

http://127.0.0.1:46205/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/

12. Creating a kafka topic

kubectl -n kafka run kafka-topics -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-topics.sh --zookeeper example-zookeepercluster-client.zookeeper:2181 --topic my-topic --create --partitions 1 --replication-factor 1

dhanuka@dhanuka:~$ minikube service envoy-loadbalancer -n kafka

13. Create a produce and generate some messages

dhanuka@dhanuka:~$ kubectl -n kafka run kafka-producer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-producer.sh --broker-list envoy-loadbalancer:19090 --topic my-topic >to >load >balancer >to >load >balancer >

kubectl -n kafka run kafka-producer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-producer.sh --broker-list 192.168.39.180:32652 --topic my-topic

If you don't see a command prompt, try pressing enter.

>>hi
>I am
>remote

>

14. Create a consumer and consume generated messages

dhanuka@dhanuka:~$kubectl -n kafka run kafka-consumer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server envoy-loadbalancer:19090 --topic my-topic --from-beginning If you don't see a command prompt, try pressing enter.

to load balancer to load balancer

kubectl -n kafka run kafka-consumer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.39.180:32652 --topic my-topic --from-beginning
If you don't see a command prompt, try pressing enter.
to
load
balancer
to
load
balancer

hi
I am

remote

Explanation: High level Kubernetes Architecture

Dashboard

http://127.0.0.1:46205/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/#/overview?namespace=_all

Explanation: Connectivity between zookeeper and kafka brokers

Services in zookeeper Namespace

dhanuka@dhanuka:~/research/kafka/kafka-operator$ kubectl get svc -n zookeeper

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
example-zookeepercluster-client ClusterIP 10.102.39.217 <none> 2181/TCP 90m
example-zookeepercluster-headless ClusterIP None <none> 2888/TCP,3888/TCP 90m

Services in kafka Namespace

dhanuka@dhanuka:~/research/kafka/kafka-operator$ kubectl get svc -n kafka

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
envoy-loadbalancer LoadBalancer 10.108.163.32 10.108.163.32 19090:32440/TCP,19091:32652/TCP,19092:30377/TCP 3h10m

kafka-headless ClusterIP None <none> 29092/TCP,29093/TCP,9020/TCP 3h10m

Let's start with Kafka cluster configuration

Go to below configuration file

https://github.com/dhanuka84/kafka-operator/blob/master/config/samples/Envoykafkacluster.yaml

You can see that, kafka connect to zookeeper cluster using "example-zookeepercluster-client " service.

In the Dashboard go to "Custom Resource Definition"

Then click "KafkaCluster"

Then if you click Edit button as highlighted below you can see the full spec and related properties which are already initialized in https://github.com/dhanuka84/kafka-operator/blob/master/config/samples/Envoykafkacluster.yaml file.