In this tutorial I will show you how to install and configure ELK stack on Ubuntu which will be used as a Centralized Log management system.

ELK stands for Elasticsearch , Logstash and Kibana.

Before proceeding further , Refer this article , To create an EC2 Instance using AWS Console

ELASTICSEARCH

ElasticSearch is an open source, RESTful search engine built on top of Apache Lucene and released under an Apache license. It is Java-based and can search and index document files in diverse formats.

Elasticsearch is a scalable search engine that can be used to search for all kind of text documents, including log files

The data is queried, retrieved and stored in a JSON document scheme.

LOGSTASH

Logstash is a tool based on the filter/pipes patterns for gathering, processing and generating the logs or events. It helps in centralizing and making real time analysis of logs and events from different sources.

Logstash is written on JRuby programming language that runs on the JVM

Logstash can collect data from different sources and send to multiple destinations.

KIBANA

Kibana is a visualization tool, which accesses the logs from Elasticsearch and is able to display to the user in the form of line graph, bar graph, pie charts.

The flow of ELK stack is shown in the image below.

ELK

Input data source can be application or Nginx logs which will be collected by the filebeat and sent to the logstash.

The logstash then collect and parse and transform the data and it will be sent to Elasticsearch as an Index.

The indexes stored in the Elasticsearch will be used by Kibana and it will be represented to the user in the form of Barcharts , Graphs etc.

INSTALLING ELASTICSEARCH:

In this setup we will install and configure elasticsearch on Ubuntu server.

But before installing the Elasticsearch , We need to install JAVA on the system , As the elasticsearch requires Java.

To install Java 8 use the below commands.

sudo apt install openjdk-8-jdk openjdk-8-jre

Now that we have installed java on the system.To verify the version and installation of Java use the below command.

openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

Then we need to install dependencies such as software-properties-common’ & ‘apt-transport-https’ packages.

sudo apt install software-properties-common apt-transport-https -y

After install Java and the dependent packages on the system.

We will install Elasticsearch , Following the below procedures.

Add the elastic stack key and the elastic stack repository to the system.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

Once the repository is added to the system ,update the repository and install the elasticsearch package in the system.

sudo apt-get update
sudo apt-get install elasticsearch -y

Once the installation is completed , Go the /etc/elasticsearch folder and configure the elasticsearch on elasticsearch.yml

Uncomment the network.host and change the value to localhost.By doing this , elasticsearch port will listen only to localhost.

uncomment http.port on which the elasticsearch will run.

Hence the elasticsearch running on the port 9200 will listen only on localhost.

network.host: localhost
http.port: 9200

Save the configuration and exit the file.

Nnow start the elasticsearch service and enable it to run on system boot.

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

To check the status of the elasticsearch , use the below command.

sudo systemctl status elasticsearch

Now the elasticsearch service should be UP and running.

We can check the version of running elasticsearch , Use the below command,

curl http://localhost:9200/?pretty
ELK

INSTALLING KIBANA

In this step we will install and configure Kibana dashboard.

It is recommended to install kibana after setting up elasticsearch to ensure that all the required components are already in place.

As we have already added the repository , It comes with the kibana package.

To install kibana use the below command.

sudo apt-get install kibana

To edit the kibana configuration file , Go to /etc/kibana folder and open kibana.yml file.

Uncomment the following lines.

server.port: 5601

5601 is the port in which kibana runs.

server.host: 127.0.0.1

Once done , Save and exit.

To start the kibana service,

sudo systemctl start kibana

To enable the service to start on the system boot.

sudo systemctl enable kibana

To check the status of the kibana service

sudo systemctl status kibana

Now that we have successfully installed and configure Kibana dashboard which is running on default port 5601 and listening on any host.

INSTALL AND CONFIGURE NGINX FOR KIBANA

In this step We will configure Nginx web server as a reverse proxy for the kibana dashboard.

Lets Install Nginx package to the system.

sudo apt-get install nginx

After the nginx is installed , Go to /etc/nginx/sites-available folder and then create a virtual host file named “kibana”.

sudo /etc/nginx/sites-available
sudo vi kibana

And paste the below nginx configuration in the kibana virtual host file.

server {
listen 80;
server_name fitdevops.in;
auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;
location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}

In the server_name You should replace with your domain name. Save and exit the file.

Next We have to setup a basic authentication for the Kibana dashboard.

use the below commands to create basic authentication.

sudo htpasswd -c /etc/nginx/.kibana-user elastic
Type the elastic user password

Provide the password for the elastic user and confirm the password for authentication.

We should enable the virtual host configuration.

ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/
To validate the nginx configuration.
nginx -t

You should get the below response.

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Start the nginx service and enable the service to start on the system boot.

sudo systemctl start nginx
sudo systemctl enable nginx

Now We have successfully installed and configure kibana and nginx reverse proxy for the Kibana Dashboard.

INSTALLING LOGSTASH

We will install and configure logstash which will collect the logs from filebeat and then it will transform the logs and sent it to elasticsearch.

Install logstash using the below command.

sudo apt-get install logstash

Then we will generate the SSL certificate key to secure the log transfer from Filebeat to Logstash.By doing so the data will be secure during transactions.

once the logstash packge in installed to the system , Go to /etc/logstash folder and then create a folder named ssl

sudo cd /etc/logstash
sudo mkdir ssl

Before generating SSL certificate , Edit the /etc/hosts file and add the below configuration.

192.168.10.3                        elk-master

We will generate the SSL certificate using openssl command.

openssl req -subj '/CN=elk-master/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout ssl/logstash-forwarder.key -out ssl/logstash-forwarder.crt

The SSL certificate will be created and stored in the ssl directory.

We have to create new configuration files for collecting inputs from filebeat , processing syslog files and then sending them to elasticsearch.

Go to /etc/logstash/conf.d folder and then create the below mentioned files.

The first file filebeat-input.conf file is to receive inputs from the incoming sources such as filebeat.

sudo vi filebeat-input.conf
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/ssl/logstash-forwarder.crt"
ssl_key => "/etc/logstash/ssl/logstash-forwarder.key"
}
}

Save and close the file.

Then to parse the syslog files and to process them , we are going to use a grok expression which is a filtering mechanism.

sudo vi syslog-filter.conf
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

Save and exit the file.

Finally for the elasticsearch output , we will create a confiuration file named output-elasticsearch.conf

output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

Save and close the file.

Once all the required configurations are implemented , We will start the logstash service.

sudo systemctl start logstash
sudo systemctl enable logstash

Now the Logstash service is running and by default it runs on the port 5443.

The ELK stack setup is completed.

CONCLUSION

We have installed and configured the Centralized Log management tool using Elasticsearch , Logstash and Kibana.

Hope you find it helpful.

Thanks for reading this article.Please do check out my other publications.

Monitor AWS Resources using Cloudwatch Alarms

Attach Elastic IP Address to EC2 Instance

Scan EC2 Instances using ClamAV in AWS

Create AMI from existing EC2 Instances

How to Create and Restore RDS snapshots