Create Elasticsearch Cluster In AWS

What Is Elasticsearch?

Elasticsearch is a search engine based on the Lucene library which is developed in Java.

You can send the data in the form of JSON documents to elasticsearch using any API or ingestion tools such as Logstash.

It is used to store, manage and retrieve document oriented information.

In this tutorial , We will learn how to create AWS managed elasticsearch cluster using AWS console.

What Is Amazon Elasticsearch?

Amazon ES is a AWS managed service which is a easy method to deploy elasticsearch clusters and manage the JSON based documents.

It’s a log analytics tool which helps in real time application monitoring and analysis.

Benefits Of Using Amazon ES

  • It will automatically replaces the fault nodes in the cluster.
  • Reduces the overhead of managing infrastructure.

Features Of Amazon Elasticsearch

  • Scales up to 3 PB of storage , cost-effective storage and variety of Instance types.
  • Secure by encrypting data at Rest , IAM Access control , managed within VPC.
  • Provides stability with dedicated master node , automated backup ,Multi-AZ in the same region.
  • Ability to integrate with Kibana , S3 , Kinesis , DynamoDB , Cloudtrail.

Lets go ahead and create a Amazon Elasticsearch cluster using AWS Console.

Creating Elasticsearch Cluster

To create Amazon ES cluster , Login to AWS Console ,

Under Analytics , Choose Elasticsearch Service

You will get the following page,

Click Create a new domain

You will asked to choose the deployment type , And you can change this anytime after creation.

For this tutorial , Choose Development and testing

Next Under Version , select the version of elasticsearch for you domain ,For latest features choose the latest updated version of elasticsearch

and click Next

Configure Domain

We need to configure a domain for the elasticsearch.

Domain is nothing but a collection of resources such as Instance type , Storage , snapshot configuration , master node setup.

Provide a name for Elasticsearch domain name

Next we have to choose the instance type based on the application.While creating these data nodes consider instance parameter such as CPU and Memory.

And mention the Number of Data nodes you want.

Next we have to configure the storage for the Data nodes.

The Data nodes storage will be calculated based on the total number of data nodes.

Choose the Data nodes storage type as EBS

Select the EBS volume type , and provide the storage size.

For the High stability of the Elasticsearch domain cluster , It is recommened to have a Dedicated master node.

You can choose the different instance type for the master node.

Enable Encryption , if you want to Encrypt the data at rest and to have a secure communications between the nodes and the applications.

If you want to access the elasticsearch domain endpoint only through HTTPS , Enable it.

Node-to-node encryption ,Provides added security , If you want the nodes within VPC to communicate over the TLS encrypted channel.

Enable encryption of data at rest , To secure all the indexes and snapshots of the elasticsearch service.

Amazon ES will take a automated snapshot everyday. You can configure the backup time for the Elasticsearch to take backup automatically.

It is recommened to take backup during non-production or less traffic hours.

Click Next,

Setup Access

If you want the elasticsearch domain to be accessed from within VPC and Using security group you can allow elasticsearch domain to be accessed within certain IP Addresses.

Under Network configuration ,

You have an option to select Internet or VPC.

If you choose Internet endpoint , The elasticsearch domain endpoint is accessible from anywhere in the world.

If you want to restrict it , You have to choose VPC access Which provides an additional layer of security to the Elasticsearch cluster.

Select VPC access and Choose the VPC and Subnets where you want the ES domain to be launched

and choose the Security group.

By default , A role will be attached with the Elasticsearch service to access VPC.

You can enable Amazon Cognito authentication , If you want the kibana to be authenticated by Cognito.We will ignore this for now.

Next , Under Access policy,

Amazon ES supports 3 types of Policies,

  • Resource-based policies
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:user/test-user"
        ]
      },
      "Action": [
        "es:*"
      ],
      "Resource": "arn:aws:es:us-west-1:987654321098:domain/domainname/*"
    }
  ]
}

 

  • Identity-based policies

 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "es:Describe*",
        "es:List*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
  • IP-based policies
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "es:*"
      ],
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "10.0.2.0/24"
          ]
        }
      },
      "Resource": "arn:aws:es:us-west-1:987654321098:domain/domainname/*"
    }
  ]
}

For now , Choose Do not require signing request with IAM credential,

Choose Next and Review the configurations and Click Confirm.

It will take up to 10 minutes for the Domain to be initialize.Once it is ready you can upload the data to the elasticsearch domain and store it as indexes.

Conclusion

We have successfully created a Elasticsearch service in AWS using Console and we have gone through each configuration in detail.

Hope you have liked it, Thanks for reading this article

Please do checkout my other publications