Learning Kubernetes by Example : Kafka Cluster

In this example, we are going to setup Kafka cluster using minikube .
I am using Ubuntu 18.04.3 LTS as my host machine.

Keypoints:

1. Each pods has a ip address.
2. Kafka deployed using Helm (2.0) and Kubernetes (1.17.2) operators
3. docker version : 18.09.1
4. Machine : 24 GB RAM, 8 VCPUs
5. Using Virtualbox as a dependency for minikube virtualization.

Steps:

1. Install minikube

https://kubernetes.io/docs/tasks/tools/install-minikube/

2. Install Helm 2 

https://v2.helm.sh/docs/install/


3. Start minikube with resource allocation

minikube start --disk-size='40g' --cpus=8 --memory='16g'


4. clone https://github.com/dhanuka84/kafka-operator.git git hub repository and go to cloned directory.

5. Installing Tiller

helm init --canary-image


6. Install cert-manager and CustomResourceDefinitions

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.11.0/cert-manager.yaml

7. Install zookeeper operator

helm install --name zookeeper-operator --namespace=zookeeper banzaicloud-stable/zookeeper-operator

8. Install zookeeper pod

kubectl create --namespace zookeeper -f - <<EOF

apiVersion: zookeeper.pravega.io/v1beta1

kind: ZookeeperCluster

metadata:

  name: example-zookeepercluster

  namespace: zookeeper

spec:

  replicas: 1

EOF

9. Install Kafka operator

helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com/

helm install --name=kafka-operator --namespace=kafka banzaicloud-stable/kafka-operator.

10. Install Kafka cluster

kubectl create -n kafka -f config/samples/simplekafkacluster.yaml

11. Check Installation

dhanuka@dhanuka:~$ kubectl get pods  --all-namespaces

NAMESPACE      NAME                                       READY   STATUS    RESTARTS   AGE

cert-manager   cert-manager-69b4f77ffc-9xb89              1/1     Running   6          14d

cert-manager   cert-manager-cainjector-576978ffc8-kct5v   1/1     Running   52         14d

cert-manager   cert-manager-webhook-c67fbc858-d6w2r       1/1     Running   2          14d

kafka          kafka-0-ndqqn                              1/1     Running   0          5d17h

kafka          kafka-1-gf68m                              1/1     Running   0          5d17h

kafka          kafka-2-6v2j9                              1/1     Running   0          5d17h

kafka          kafka-cruisecontrol-64594876c-g8mzf        1/1     Running   5          14d

kafka          kafka-operator-operator-b556dddf9-kvv72    2/2     Running   45         14d

kube-system    coredns-6955765f44-29q5w                   1/1     Running   1          14d

kube-system    coredns-6955765f44-v6f8q                   1/1     Running   1          14d

kube-system    etcd-minikube                              1/1     Running   1          14d

kube-system    kube-apiserver-minikube                    1/1     Running   41         14d

kube-system    kube-controller-manager-minikube           1/1     Running   45         14d

kube-system    kube-proxy-qlhnc                           1/1     Running   1          14d

kube-system    kube-scheduler-minikube                    1/1     Running   47         14d

kube-system    storage-provisioner                        1/1     Running   3          14d

kube-system    tiller-deploy-85748c5d55-kv6tb             1/1     Running   1          14d

zookeeper      example-zookeepercluster-0                 1/1     Running   3          14d

zookeeper      zookeeper-operator-669bf74b5f-vr9dw        1/1     Running   2          14d

12. Creating a kafka topic

kubectl -n kafka run kafka-topics -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-topics.sh --zookeeper example-zookeepercluster-client.zookeeper:2181 --topic my-topic --create --partitions 1 --replication-factor 1

13. Create a produce and generate some messages

dhanuka@dhanuka:~$ kubectl -n kafka run kafka-producer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-producer.sh --broker-list kafka-headless:29092 --topic my-topic welcome to kubernetes clustering If you don't see a command prompt, try pressing enter.

14. Create a consumer and consume generated messages

dhanuka@dhanuka:~$ kubectl -n kafka run kafka-consumer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka-headless:29092 --topic my-topic --from-beginning If you don't see a command prompt, try pressing enter. welcome to kubernetes clustering

Inspecting the cluster

1. List all the services

dhanuka@dhanuka:~$ kubectl get service --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cert-manager cert-manager ClusterIP 10.110.184.5 <none> 9402/TCP 14d cert-manager cert-manager-webhook ClusterIP 10.102.227.195 <none> 443/TCP 14d default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 14d kafka kafka-cruisecontrol-svc ClusterIP 10.107.64.181 <none> 8090/TCP,9020/TCP 14d kafka kafka-headless ClusterIP None <none> 29092/TCP,29093/TCP,9020/TCP 14d kafka kafka-operator-alertmanager ClusterIP 10.98.58.117 <none> 9001/TCP 14d kafka kafka-operator-authproxy ClusterIP 10.105.190.124 <none> 8443/TCP 14d kafka kafka-operator-operator ClusterIP 10.102.160.123 <none> 443/TCP 14d kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 14d kube-system tiller-deploy ClusterIP 10.101.77.57 <none> 44134/TCP 14d zookeeper example-zookeepercluster-client ClusterIP 10.108.226.134 <none> 2181/TCP 14d zookeeper example-zookeepercluster-headless ClusterIP None <none> 2888/TCP,3888/TCP 14d


2. Login to kafka pod

dhanuka@dhanuka:~$ kubectl exec -it  kafka-1-gf68m  bash -n kafka
bash-4.4# ls
bin         dev         home        kafka-logs  lib64       mnt         proc        run         srv         tmp         var
config      etc         kafka       lib         media       opt         root        sbin        sys         usr

bash-4.4# cat config/broker-config 

advertised.listeners=INTERNAL://kafka-1.kafka-headless.kafka.svc.cluster.local:29092,CONTROLLER://kafka-1.kafka-headless.kafka.svc.cluster.local:29093
auto.create.topics.enable=false
broker.id=1
control.plane.listener.name=CONTROLLER
cruise.control.metrics.reporter.bootstrap.servers=INTERNAL://kafka-1.kafka-headless.kafka.svc.cluster.local:29092
inter.broker.listener.name=INTERNAL
listener.security.protocol.map=INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
listeners=INTERNAL://:29092,CONTROLLER://:29093
log.dirs=/kafka-logs/kafka
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
super.users=

zookeeper.connect=example-zookeepercluster-client.zookeeper:2181/
bash-4.4# 

3. Get persistence volume claims

dhanuka@dhanuka:~$ kubectl get pvc --all-namespaces
NAMESPACE   NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
kafka       kafka-0-storaget5fvg              Bound    pvc-26f3150d-0412-4f25-b1db-2b6a256871a1   10Gi       RWO            standard       14d
kafka       kafka-1-storagessscn              Bound    pvc-e4744ad6-2ae3-4b95-a68d-74d2d07148ec   10Gi       RWO            standard       14d
kafka       kafka-2-storagekp4qr              Bound    pvc-97596d2e-3b44-4570-842a-9e4a8de8f755   10Gi       RWO            standard       14d

zookeeper   data-example-zookeepercluster-0   Bound    pvc-c88627ab-b040-4362-af95-02184b64e4c0   20Gi       RWO            standard       14d


4. Describe persistence volume

dhanuka@dhanuka:~$ kubectl describe pv pvc-26f3150d-0412-4f25-b1db-2b6a256871a1 -n kafka
Name:            pvc-26f3150d-0412-4f25-b1db-2b6a256871a1
Labels:          <none>
Annotations:     hostPathProvisionerIdentity: a5f572ed-5611-11ea-929c-e06af1f704c7
                 pv.kubernetes.io/provisioned-by: k8s.io/minikube-hostpath
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    standard
Status:          Bound
Claim:           kafka/kafka-0-storaget5fvg
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        10Gi
Node Affinity:   <none>
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp/hostpath-provisioner/pvc-26f3150d-0412-4f25-b1db-2b6a256871a1
    HostPathType:  

Events:            <none>

5. Describe kafka pod

dhanuka@dhanuka:~$ kubectl -n kafka describe pod kafka-0-ndqqn Name: kafka-0-ndqqn Namespace: kafka Priority: 0 Node: minikube/192.168.39.146 Start Time: Tue, 03 Mar 2020 01:16:50 +0800 Labels: app=kafka brokerId=0 controller-tools.k8s.io=1.0 kafka_cr=kafka Annotations: banzaicloud.com/last-applied: {"metadata":{"annotations":{"prometheus.io/port":"9020","prometheus.io/scrape":"true"},"generateName":"kafka-0-","labels":{"app":"kafka","... prometheus.io/port: 9020 prometheus.io/scrape: true Status: Running IP: 172.17.0.12 IPs: IP: 172.17.0.12 Controlled By: KafkaCluster/kafka Init Containers: cruise-control-reporter: Container ID: docker://25c10ccbc4147b7a714503ce73b09223b7f266a1c5b189e74cd2aec2bf0b1a7a Image: solsson/kafka-cruise-control@sha256:658c21295a940b4c490aadfb95973b34f27fd9c944c3f11a2a9b89e5948a78bd Image ID: docker-pullable://solsson/kafka-cruise-control@sha256:658c21295a940b4c490aadfb95973b34f27fd9c944c3f11a2a9b89e5948a78bd Port: <none> Host Port: <none> Command: /bin/sh -cex cp -v /opt/cruise-control/cruise-control/build/dependant-libs/cruise-control-metrics-reporter.jar /opt/kafka/libs/extensions/cruise-control-metrics-reporter.jar State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 03 Mar 2020 01:17:09 +0800 Finished: Tue, 03 Mar 2020 01:17:09 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /opt/kafka/libs/extensions from extensions (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-z7gvc (ro) jmx-exporter: Container ID: docker://3e21801af6bcf2da0eb09ea9ec002a3b1b2a98b25d903d33de8cacbab0210256 Image: banzaicloud/jmx-javaagent:0.12.0 Image ID: docker-pullable://banzaicloud/jmx-javaagent@sha256:e61710879b0436157363498d95cc9731d3c41e214126d7305dbe4cbe99e5cedc Port: <none> Host Port: <none> Command: cp /opt/jmx_exporter/jmx_prometheus_javaagent-0.12.0.jar /opt/jmx-exporter/jmx_prometheus.jar State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 03 Mar 2020 01:17:20 +0800 Finished: Tue, 03 Mar 2020 01:17:20 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /opt/jmx-exporter/ from jmx-jar-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-z7gvc (ro) Containers: kafka: Container ID: docker://a4a71e41f6d0b70060103fc2adbc81a5c4c7e5dc616ea0bef5c3b187e6b357b0 Image: wurstmeister/kafka:2.12-2.3.0 Image ID: docker-pullable://wurstmeister/kafka@sha256:a1b4deaa037fd110d87676fc315608c556fcef98f8f1f9302e9cf85757f9d1f3 Ports: 29092/TCP, 29093/TCP, 9020/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Command: bash -c if [[ -n "$ENVOY_SIDECAR_STATUS" ]]; then COUNT=0 MAXCOUNT=${1:-30} HEALTHYSTATUSCODE="200" while true; do COUNT=$(expr $COUNT + 1) SC=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:15000/ready) echo "waiting for envoy proxy to come up"; sleep 1; if (( "$SC" == "$HEALTHYSTATUSCODE" || "$MAXCOUNT" == "$COUNT" )); then break fi done fi /opt/kafka/bin/kafka-server-start.sh /config/broker-config State: Running Started: Tue, 03 Mar 2020 01:17:29 +0800 Ready: True Restart Count: 0 Limits: cpu: 1500m memory: 3Gi Requests: cpu: 1 memory: 2Gi Environment: CLASSPATH: /opt/kafka/libs/extensions/* KAFKA_OPTS: -javaagent:/opt/jmx-exporter/jmx_prometheus.jar=9020:/etc/jmx-exporter/config.yaml KAFKA_HEAP_OPTS: -Xmx2G -Xms2G KAFKA_JVM_PERFORMANCE_OPTS: -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 ENVOY_SIDECAR_STATUS: (v1:metadata.annotations['sidecar.istio.io/status']) Mounts: /config from broker-config (rw) /etc/jmx-exporter/ from kafka-kafka-jmx-exporter (rw) /kafka-logs from kafka-data-0 (rw) /opt/jmx-exporter/ from jmx-jar-data (rw) /opt/kafka/libs/extensions from extensions (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-z7gvc (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: kafka-data-0: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: kafka-0-storaget5fvg ReadOnly: false broker-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: kafka-config-0 Optional: false extensions: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kafka-kafka-jmx-exporter: Type: ConfigMap (a volume populated by a ConfigMap) Name: kafka-kafka-jmx-exporter Optional: false jmx-jar-data: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> default-token-z7gvc: Type: Secret (a volume populated by a Secret) SecretName: default-token-z7gvc Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: <none>