12/02/2019

Setup Kubernetes Cluster with Virtualbox








Steps


1. Install Oracle Virtualbox & Extension pack


Download from here - https://www.virtualbox.org/wiki/Download_Old_Builds_5_2

2. Download Ubuntu ISO( ubuntu-18.04.3-desktop-amd64.iso )  image from below link


https://ubuntu.com/download/desktop

3. Create virtual machine in virtual box.  

Minimum Requirement is : 2 GB RAM, 1 VCPU

You can follow below blog for more details.

https://www.wikihow.com/Install-Ubuntu-on-VirtualBox


4. Sample virtual machine




After successfully setup virtual machine, you need to install guest additions as described below.
https://www.virtualbox.org/manual/ch04.html

5. Key points when creating a virtual machine


Network Settings:

  • You need to connect to internet so we have to use NAT network adapter
  • Also we are using bridged adapter as the cluster communication network, and it's IP as static IP
  • Configure the adapters as mentioned in the image

6. Once you setup virtual machine you need to install Docker and disable swap


7. Setup the virtual machine. Execute following commands in a terminal


sudo apt-get update

sudo apt install net-tools

Change hostname

sudo hostnamectl set-hostname "k8s-master"

exec bash

8. Install Docker


sudo dpkg --configure -a

Install relevant packages

sudo apt install apt-transport-https ca-certificates curl software-properties-common


sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"

sudo apt update

apt-cache policy docker-ce

sudo apt install docker-ce


Use Docker as normal user

sudo usermod -aG docker ${USER}

su - ${USER}

sudo systemctl start docker

sudo systemctl enable docker


9. Create a static IP


Type below command and select enp0s9 entry

ifconfig -a


enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.166  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::a00:27ff:fe7c:faba  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:7c:fa:ba  txqueuelen 1000  (Ethernet)
        RX packets 21115  bytes 8269767 (8.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14770  bytes 1610853 (1.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

sudo vim /etc/network/interfaces

insert below code block to above file


auto enp0s9

iface enp0s9 inet static

address 192.168.0.166
netmask 255.255.255.0

Change /etc/hosts file with below IP entries

sudo vim /etc/hosts

192.168.0.166      k8s-master
192.168.0.178      k8s-worker-node2


10. Install Kubernetes


Install relevant packages

sudo apt install apt-transport-https ca-certificates curl software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add

sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"

sudo apt update


Disable swap

sudo swapoff -a

vim /etc/fstab

comment out lines which indicate about swap
example :

#/swapfile                                 none            swap    sw              0       0

sudo apt-get install kubeadm -y


11. Cloning virtual machine as worker/slave node


Use below link but clone this as FULL clone


12. Setup Kubernetes Master



sudo kubeadm init  --apiserver-advertise-address=192.168.0.166 --pod-network-cidr=192.168.0.0/24

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

Save below out put to a file

kubeadm join 192.168.0.166:6443 --token pa36ca.1e073aktjf9rac6m \
    --discovery-token-ca-cert-hash sha256:b850d754741005a6d7f2a096441d3c229dd23bc3a0078157c5d815a22216bc34 

13. Setup Kubernetes in Slave node


Do step 9 and  10. Then execute below command to join slave node to cluster
Then change hostname

sudo kubeadm join 192.168.0.166:6443 --token pa36ca.1e073aktjf9rac6m \
    --discovery-token-ca-cert-hash sha256:b850d754741005a6d7f2a096441d3c229dd23bc3a0078157c5d815a22216bc34

14. Deploy Flannel as Pod Network in both master and slave nodes


sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml


15. Verify the cluster from master node


kubanetes@k8s-master:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-5644d7b6d9-vwnfx             0/1     Running   0          8m37s
kube-system   coredns-5644d7b6d9-zwgvn             0/1     Running   0          8m37s
kube-system   etcd-k8s-master                      1/1     Running   0          7m59s
kube-system   kube-apiserver-k8s-master            1/1     Running   0          8m1s
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          7m51s
kube-system   kube-flannel-ds-amd64-wk5s6          1/1     Running   0          27s
kube-system   kube-proxy-xgcjh                     1/1     Running   0          8m37s
kube-system   kube-scheduler-k8s-master            1/1     Running   0          7m56s



kubanetes@k8s-master:~$ kubectl get nodes
NAME               STATUS   ROLES    AGE    VERSION
k8s-master         Ready    master   26m    v1.16.3
k8s-worker-node2   Ready    <none>   114s   v1.16.3


References:







                 

6/09/2019

Flume with Docker





Use case

Send file through TCP to Flume and log content in the console.
  • Assumptions: Docker already installed

Steps


1. Clone below github project.

https://github.com/dhanuka84/docker-flume

2. Build docker image using following command

docker build -t my-flume-image .

3. Change configuration in following folder according to your local machine.

config/*
run-fl.sh

4. Execute bash script

sh run-fl.sh

5. Send file through TCP tunnel

netcat localhost 4444 < README.md

6. Output from docker container




References:


1. https://flume.apache.org/releases/content/1.8.0/FlumeUserGuide.html
2. https://github.com/mrwilson/docker-flume
3. https://blog.probablyfine.co.uk/2014/08/24/using-docker-with-apache-flume-2.html



2/09/2019

Rule Execution as Streaming Process with Flink


As explained in the above diagram, rule creator (Desktop) will create JSON based rule and push them to Kafka (rule topic). Event Source will send events to Kafka  (testin topic). Finally Flink will consume both rules and events as streams and process rules based on key (Driver Id). Rules will be stored in Flink as in-memory collection and the rules also can be updated in same manner. Finally out put result will be send to Kafka (testout topic).



Setup Flink



  • Download Apache Flink 1.6.3 from below location and extract archive file.

          https://www.apache.org/dyn/closer.lua/flink/flink-1.6.3/flink-1.6.3-bin-scala_2.11.tgz


  • Download below dependancy jar files and place in flink-1.6.3/lib folder.





  • Configure Flink with flink-1.6.3/conf/flink-conf.yaml .


         Change job-manager and task-manager heap size to much smaller size.

         jobmanager.heap.size: 1g
         taskmanager.heap.size: 2g
         taskmanager.numberOfTaskSlots: 20

  • Change State Backend to RocksDB

          Create folder from you home location

         ~$ mkdir -p data/flink/checkpoints

          Edit configuration

          state.backend: rocksdb
          state.checkpoints.dir: file:///home/dhanuka/data/flink/checkpoints


  • Start Flink cluster in standalone mode.

           :~/software/flink-1.6.3$ ./bin/start-cluster.sh

Setup Kafka & Zookeeper


  • In here I am using docker and docker compose so you can follow below blogspost to install docker & docker-compose in ubuntu.

          http://dhanuka84.blogspot.com/2019/02/install-docker-in-ubuntu.html


  • Please checkout below docker project from github.

          https://github.com/dhanuka84/my-docker.git 


  • Change IP address to your machine IP address.

         https://github.com/dhanuka84/my-docker/blob/master/kafka/kafka-hazelcast.yml


  • Bootup Kafka and Zookeeper with docker-compose
          my-docker/kafka$ sudo docker-compose -f kafka-hazelcast.yml up


  • Check docker containers





  • Create Kafka Topics

         Download Confluent Platform - https://www.confluent.io/download/

  • Got to confluent platform extracted location and run below commands



bin/kafka-topics  --create --zookeeper localhost:2181 --replication-factor 1 --partitions 6 --topic  testin
bin/kafka-topics  --create --zookeeper localhost:2181 --replication-factor 1 --partitions 6 --topic  testout
bin/kafka-topics  --create --zookeeper localhost:2181 --replication-factor 1 --partitions 6 --topic  rule

Create Java based Rule Job


  • Checkout the project from github and build the project with maven.

          https://github.com/dhanuka84/stream-analytics
       
          stream-analytics$ mvn clean install -Dmaven.test.skip=true


          Please note that both rules and events filter by key.
         
          KeyedStream<Event, String> filteredEvents = events.keyBy(Event::getDriverID);
          rules.broadcast().keyBy(Rule::getDriverID)

          Store rules in ListState using flatMap1 method.
          Execute rule against  each relevant event using flatMap2 method.

          Finally results transform to JSON string.
  • Prepare flat jar to upload, using below command within checked out project home.

stream-analytics$ jar uf target/stream-analytics-0.0.1-SNAPSHOT.jar application.properties consumer.properties producer.properties

  • Upload Flink Job Jar file
         copy stream-analytics-0.0.1-SNAPSHOT.jar Flink home folder and run below command

         bin/flink run stream-analytics-0.0.1-SNAPSHOT.jar

Testing

  • Run Producer.java class using your favorite IDE. This will generate both events and rules then publish to Kafka.
  • To verify from Kafka level use below commands within confluent home location.


Get offset

bin/kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic testout  --time -1

 Read from Kafka topic
bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic testout

2/08/2019

Install docker in Ubuntu

1. Go to below link
https://download.docker.com/linux/ubuntu/dists/xenial/pool/stable/amd64/

2. Download below debian installations
docker-ce-cli_18.09.1_3-0_ubuntu-xenial_amd64.deb
containerd.io_1.2.2-1_amd64.deb
docker-ce_18.09.1_3-0_ubuntu-xenial_amd64.deb


3. Install using below command
sudo dpkg -i /path/to/package.deb

4. Download docker-compose

curl -L https://github.com/docker/compose/releases/download/1.23.2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose


5. Make docker-compose executable
sudo chmod +x /usr/local/bin/docker-compose

6. Check Versions
docker --version
docker-compose --version

7. Check whether docker running

service docker status


8. Use Docker as normal user

sudo systemctl stop docker
sudo usermod -aG docker ${USER}
su - ${USER}
sudo systemctl start docker
sudo systemctl enable docker









11/24/2018

Elasticsearch : Factors Need To Consider To Get Rid Of From OOM







We used Elasticsearch as centralize logging system for multiple products.

Observations:

1. 8283 primary shards
2. 10 Billion documents
3. 8.06 TB of data in each data node
4. 4365 indices
5. Five primary shards per index
6. Four Data Nodes
7. Three out of four data nodes heap memory in critical state.


Identified drawbacks

  1.  Use daily index creation for even 1GB size indexes. 
  2. The default primary shard count for every index is 5. 
  3. For 12 different indices, number of primary shards will be as follows.


(primary shard per index) * 12 (number of indexes/ products) * 30 (days per month) = 1800


(3 months) 1800 * 3 = 5400
(6 months) 1800 * 6 = 10800


Improvements

  1. Use monthly index creation or create index more configurable manner based on the primary shard size.

Note: According to Elastic , recommended size of a primary shard is 30GB - 40GB (depend on network bandwidth)
       number of shards per node below 20 to 25 per GB (roughly each primary shard cost 40 MB of heap memory)


5 * 6 * 11 + 5 * 30  = 480 (No of primary shards count)


Improvement = 10800/480 = 22.5 times


Clear some misunderstanding



According to Elastic these are the ratio between RAM & Storage for two main purposes (reading & writing).

 For memory-intensive search workloads, more RAM (less storage) can improve performance. Use high-performance SSDs drives
 and a RAM-to-storage ratio for users of 1:16 (or even 1:8). For example, if you use a ratio of 1:16, a cluster with 4GB of RAM will 
get 64GB of storage allocated to it.

For logging workloads, more storage space can be more cost effective. Use a RAM-to-storage ratio of 1:48 to 1:96, as data sizes are 
typically much larger compared to the RAM needed for logging. A cost effective solution might be to step down from SSDs to spinning
 media, such as high-performance server disks. For example, if you use a ratio of 1:96, a cluster with 4GB of RAM will get 384 of 
storage allocated to it.
 
If we take one of our ES based centralize Logs cluster in production it’s working fine for 8.5 TB data just for 25GB RAM in data nodes.
 So according to Logs Cluster, RAM to Disk ratio is as 1:348. Also when we check number of primary shards it’s > 7000 .
How can this happen, so Logs even work with less RAM??
Highest contribution made by one of the product logs, roughly daily index size is 30 GB. Most of other products generate less than
 2 GB size of daily indexes.
As we all know , the Logs Cluster query load is really low compared to logs writing frequency.Having said that , If we take biggest 
logs contribution product logs writing frequency it’s 
something around 600 per second.
This is really low log writing load. According to my understanding normal workload should be 10000 per second
So Elastic give recommendation keeping these workloads in mind, where our case this is really low situation and that’s why we still
 can live with current  RAM to Storage ratio.
References: