Elasticsearch Hot Warm Cold Setup

We are going to deploy one master node, one  hot data node, warm data node, cold data node and one kibana node in same local machine. Then we play with Index Life Cycle Management UI.

Steps:

1. Clone below github project and go to cloned directory

https://github.com/dhanuka84/es-hot-warm-cold.git

2. You need to change my local machine configurations according to your machine

Local IP: 192.168.0.114
Project Folder: /home/dhanuka/research/elastic/docker2/es-hot-warm-cold/

How to change:  Use below linux command to replace all

find ./ -type f -exec sed -i 's/192.168.0.114/new_ip/g' {} \;

find ./ -type f -exec sed -i 's#home/dhanuka/research/elastic/docker2#new_folder#g' {} \;

Install Docker :

http://dhanuka84.blogspot.com/2019/02/install-docker-in-ubuntu.html

Change system settings: Run below command in terminal

sudo echo "vm.max_map_count=262144"  >> /etc/sysctl.conf

3. Run run-all.sh script as below

sh run-all.sh

You can check terminal output with nohup.out file

4. Then login to kibana as below.

http://localhost:5601/

5. Then go to monitoring tab

You can see number of Elasticsearch and Kibana nodes with their status.

6. Now let's create ILCM policy

Go to management tab and click index lifecycle policies as below.

7. Let's create policy as shown above and below.

In here we will create hot, warm , cold and delete phases of an index life cycle 

8. Then you need to create a index template as below.

PUT _template/my_template
{
  "index_patterns": ["kibana_sample_data*"],
  "settings": {
    "index.routing.allocation.require.my_node_type": "hot"
  }
}

9. Finally you need to add previously created ILCM policy to index template as below

10. Now when ever you create new index based on that template , ILCM policywill be applied to that index. You can verify template as below.

Request : 

GET _template/my_template

Response:

{
  "my_template" : {
    "order" : 0,
    "index_patterns" : [
      "kibana_sample_data*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "hotwarm-policy",
          "rollover_alias" : "my_index_alias"
        },
        "routing" : {
          "allocation" : {
            "require" : {
              "my_node_type" : "hot"
            }
          }
        }
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}