Grasshopper ----------> Master ( D 2 ): March 2020

In this example, we are going to setup Kafka cluster using minikube .
I am using Ubuntu 18.04.3 LTS as my host machine.

Keypoints:

1. Each pods has a ip address.
2. Kafka deployed using Helm (2.0) and Kubernetes (1.17.2) operators
3. docker version : 18.09.1
4. Machine : 24 GB RAM, 8 VCPUs
5. Using Virtualbox as a dependency for minikube virtualization.

Steps:

1. Install minikube

https://kubernetes.io/docs/tasks/tools/install-minikube/

2. Install Helm 2

https://v2.helm.sh/docs/install/

3. Start minikube with resource allocation

minikube start --disk-size='40g' --cpus=8 --memory='16g'

4. clone https://github.com/dhanuka84/kafka-operator.git git hub repository and go to cloned directory.

5. Installing Tiller

helm init --canary-image

6. Install cert-manager and CustomResourceDefinitions

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.11.0/cert-manager.yaml

7. Install zookeeper operator

helm install --name zookeeper-operator --namespace=zookeeper banzaicloud-stable/zookeeper-operator

8. Install zookeeper pod

kubectl create --namespace zookeeper -f - <<EOF

apiVersion: zookeeper.pravega.io/v1beta1

kind: ZookeeperCluster

metadata:

name: example-zookeepercluster

namespace: zookeeper

spec:

replicas: 1

EOF

9. Install Kafka operator

helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com/

helm install --name=kafka-operator --namespace=kafka banzaicloud-stable/kafka-operator.

10. Install Kafka cluster

kubectl create -n kafka -f config/samples/simplekafkacluster.yaml

11. Check Installation

dhanuka@dhanuka:~$ kubectl get pods --all-namespaces

NAMESPACE NAME READY STATUS RESTARTS AGE

cert-manager cert-manager-69b4f77ffc-9xb89 1/1 Running 6 14d

cert-manager cert-manager-cainjector-576978ffc8-kct5v 1/1 Running 52 14d

cert-manager cert-manager-webhook-c67fbc858-d6w2r 1/1 Running 2 14d

kafka kafka-0-ndqqn 1/1 Running 0 5d17h

kafka kafka-1-gf68m 1/1 Running 0 5d17h

kafka kafka-2-6v2j9 1/1 Running 0 5d17h

kafka kafka-cruisecontrol-64594876c-g8mzf 1/1 Running 5 14d

kafka kafka-operator-operator-b556dddf9-kvv72 2/2 Running 45 14d

kube-system coredns-6955765f44-29q5w 1/1 Running 1 14d

kube-system coredns-6955765f44-v6f8q 1/1 Running 1 14d

kube-system etcd-minikube 1/1 Running 1 14d

kube-system kube-apiserver-minikube 1/1 Running 41 14d

kube-system kube-controller-manager-minikube 1/1 Running 45 14d

kube-system kube-proxy-qlhnc 1/1 Running 1 14d

kube-system kube-scheduler-minikube 1/1 Running 47 14d

kube-system storage-provisioner 1/1 Running 3 14d

kube-system tiller-deploy-85748c5d55-kv6tb 1/1 Running 1 14d

zookeeper example-zookeepercluster-0 1/1 Running 3 14d

zookeeper zookeeper-operator-669bf74b5f-vr9dw 1/1 Running 2 14d

12. Creating a kafka topic

kubectl -n kafka run kafka-topics -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-topics.sh --zookeeper example-zookeepercluster-client.zookeeper:2181 --topic my-topic --create --partitions 1 --replication-factor 1

13. Create a produce and generate some messages

dhanuka@dhanuka:~$ kubectl -n kafka run kafka-producer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-producer.sh --broker-list kafka-headless:29092 --topic my-topic welcome to kubernetes clustering If you don't see a command prompt, try pressing enter.

14. Create a consumer and consume generated messages

dhanuka@dhanuka:~$ kubectl -n kafka run kafka-consumer -it --image=banzaicloud/kafka:2.13-2.4.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka-headless:29092 --topic my-topic --from-beginning If you don't see a command prompt, try pressing enter. welcome to kubernetes clustering

Inspecting the cluster

1. List all the services

dhanuka@dhanuka:~$ kubectl get service --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cert-manager cert-manager ClusterIP 10.110.184.5 <none> 9402/TCP 14d cert-manager cert-manager-webhook ClusterIP 10.102.227.195 <none> 443/TCP 14d default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 14d kafka kafka-cruisecontrol-svc ClusterIP 10.107.64.181 <none> 8090/TCP,9020/TCP 14d kafka kafka-headless ClusterIP None <none> 29092/TCP,29093/TCP,9020/TCP 14d kafka kafka-operator-alertmanager ClusterIP 10.98.58.117 <none> 9001/TCP 14d kafka kafka-operator-authproxy ClusterIP 10.105.190.124 <none> 8443/TCP 14d kafka kafka-operator-operator ClusterIP 10.102.160.123 <none> 443/TCP 14d kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 14d kube-system tiller-deploy ClusterIP 10.101.77.57 <none> 44134/TCP 14d zookeeper example-zookeepercluster-client ClusterIP 10.108.226.134 <none> 2181/TCP 14d zookeeper example-zookeepercluster-headless ClusterIP None <none> 2888/TCP,3888/TCP 14d

2. Login to kafka pod

dhanuka@dhanuka:~$ kubectl exec -it kafka-1-gf68m bash -n kafka
bash-4.4# ls
bin dev home kafka-logs lib64 mnt proc run srv tmp var
config etc kafka lib media opt root sbin sys usr

bash-4.4# cat config/broker-config

advertised.listeners=INTERNAL://kafka-1.kafka-headless.kafka.svc.cluster.local:29092,CONTROLLER://kafka-1.kafka-headless.kafka.svc.cluster.local:29093
auto.create.topics.enable=false
broker.id=1
control.plane.listener.name=CONTROLLER
cruise.control.metrics.reporter.bootstrap.servers=INTERNAL://kafka-1.kafka-headless.kafka.svc.cluster.local:29092
inter.broker.listener.name=INTERNAL
listener.security.protocol.map=INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
listeners=INTERNAL://:29092,CONTROLLER://:29093
log.dirs=/kafka-logs/kafka
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
super.users=

zookeeper.connect=example-zookeepercluster-client.zookeeper:2181/
bash-4.4#

3. Get persistence volume claims

dhanuka@dhanuka:~$ kubectl get pvc --all-namespaces
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
kafka kafka-0-storaget5fvg Bound pvc-26f3150d-0412-4f25-b1db-2b6a256871a1 10Gi RWO standard 14d
kafka kafka-1-storagessscn Bound pvc-e4744ad6-2ae3-4b95-a68d-74d2d07148ec 10Gi RWO standard 14d
kafka kafka-2-storagekp4qr Bound pvc-97596d2e-3b44-4570-842a-9e4a8de8f755 10Gi RWO standard 14d

zookeeper data-example-zookeepercluster-0 Bound pvc-c88627ab-b040-4362-af95-02184b64e4c0 20Gi RWO standard 14d

4. Describe persistence volume

dhanuka@dhanuka:~$ kubectl describe pv pvc-26f3150d-0412-4f25-b1db-2b6a256871a1 -n kafka
Name: pvc-26f3150d-0412-4f25-b1db-2b6a256871a1
Labels: <none>
Annotations: hostPathProvisionerIdentity: a5f572ed-5611-11ea-929c-e06af1f704c7
pv.kubernetes.io/provisioned-by: k8s.io/minikube-hostpath
Finalizers: [kubernetes.io/pv-protection]
StorageClass: standard
Status: Bound
Claim: kafka/kafka-0-storaget5fvg
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity: <none>
Message:
Source:
Type: HostPath (bare host directory volume)
Path: /tmp/hostpath-provisioner/pvc-26f3150d-0412-4f25-b1db-2b6a256871a1
HostPathType:

Events: <none>

5. Describe kafka pod

dhanuka@dhanuka:~$ kubectl -n kafka describe pod kafka-0-ndqqn Name: kafka-0-ndqqn Namespace: kafka Priority: 0 Node: minikube/192.168.39.146 Start Time: Tue, 03 Mar 2020 01:16:50 +0800 Labels: app=kafka brokerId=0 controller-tools.k8s.io=1.0 kafka_cr=kafka Annotations: banzaicloud.com/last-applied: {"metadata":{"annotations":{"prometheus.io/port":"9020","prometheus.io/scrape":"true"},"generateName":"kafka-0-","labels":{"app":"kafka","... prometheus.io/port: 9020 prometheus.io/scrape: true Status: Running IP: 172.17.0.12 IPs: IP: 172.17.0.12 Controlled By: KafkaCluster/kafka Init Containers: cruise-control-reporter: Container ID: docker://25c10ccbc4147b7a714503ce73b09223b7f266a1c5b189e74cd2aec2bf0b1a7a Image: solsson/kafka-cruise-control@sha256:658c21295a940b4c490aadfb95973b34f27fd9c944c3f11a2a9b89e5948a78bd Image ID: docker-pullable://solsson/kafka-cruise-control@sha256:658c21295a940b4c490aadfb95973b34f27fd9c944c3f11a2a9b89e5948a78bd Port: <none> Host Port: <none> Command: /bin/sh -cex cp -v /opt/cruise-control/cruise-control/build/dependant-libs/cruise-control-metrics-reporter.jar /opt/kafka/libs/extensions/cruise-control-metrics-reporter.jar State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 03 Mar 2020 01:17:09 +0800 Finished: Tue, 03 Mar 2020 01:17:09 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /opt/kafka/libs/extensions from extensions (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-z7gvc (ro) jmx-exporter: Container ID: docker://3e21801af6bcf2da0eb09ea9ec002a3b1b2a98b25d903d33de8cacbab0210256 Image: banzaicloud/jmx-javaagent:0.12.0 Image ID: docker-pullable://banzaicloud/jmx-javaagent@sha256:e61710879b0436157363498d95cc9731d3c41e214126d7305dbe4cbe99e5cedc Port: <none> Host Port: <none> Command: cp /opt/jmx_exporter/jmx_prometheus_javaagent-0.12.0.jar /opt/jmx-exporter/jmx_prometheus.jar State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 03 Mar 2020 01:17:20 +0800 Finished: Tue, 03 Mar 2020 01:17:20 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /opt/jmx-exporter/ from jmx-jar-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-z7gvc (ro) Containers: kafka: Container ID: docker://a4a71e41f6d0b70060103fc2adbc81a5c4c7e5dc616ea0bef5c3b187e6b357b0 Image: wurstmeister/kafka:2.12-2.3.0 Image ID: docker-pullable://wurstmeister/kafka@sha256:a1b4deaa037fd110d87676fc315608c556fcef98f8f1f9302e9cf85757f9d1f3 Ports: 29092/TCP, 29093/TCP, 9020/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Command: bash -c if [[ -n "$ENVOY_SIDECAR_STATUS" ]]; then COUNT=0 MAXCOUNT=${1:-30} HEALTHYSTATUSCODE="200" while true; do COUNT=$(expr $COUNT + 1) SC=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:15000/ready) echo "waiting for envoy proxy to come up"; sleep 1; if (( "$SC" == "$HEALTHYSTATUSCODE" || "$MAXCOUNT" == "$COUNT" )); then break fi done fi /opt/kafka/bin/kafka-server-start.sh /config/broker-config State: Running Started: Tue, 03 Mar 2020 01:17:29 +0800 Ready: True Restart Count: 0 Limits: cpu: 1500m memory: 3Gi Requests: cpu: 1 memory: 2Gi Environment: CLASSPATH: /opt/kafka/libs/extensions/* KAFKA_OPTS: -javaagent:/opt/jmx-exporter/jmx_prometheus.jar=9020:/etc/jmx-exporter/config.yaml KAFKA_HEAP_OPTS: -Xmx2G -Xms2G KAFKA_JVM_PERFORMANCE_OPTS: -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 ENVOY_SIDECAR_STATUS: (v1:metadata.annotations['sidecar.istio.io/status']) Mounts: /config from broker-config (rw) /etc/jmx-exporter/ from kafka-kafka-jmx-exporter (rw) /kafka-logs from kafka-data-0 (rw) /opt/jmx-exporter/ from jmx-jar-data (rw) /opt/kafka/libs/extensions from extensions (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-z7gvc (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: kafka-data-0: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: kafka-0-storaget5fvg ReadOnly: false broker-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: kafka-config-0 Optional: false extensions: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kafka-kafka-jmx-exporter: Type: ConfigMap (a volume populated by a ConfigMap) Name: kafka-kafka-jmx-exporter Optional: false jmx-jar-data: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> default-token-z7gvc: Type: Secret (a volume populated by a Secret) SecretName: default-token-z7gvc Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: <none>

Grasshopper ----------> Master ( D 2 )

3/08/2020

Learning Kubernetes by Example : Kafka Cluster

Keypoints:

Steps:

Inspecting the cluster