In my previous blogs , I have explained in detail on How to

Install and configure Prometheus

Monitoring Linux nodes using Node Exporter

From the above articles , We did a setup of Node exporter on the target Linux hosts and Prometheus scraps the metrics from the Node exporters.

Also we have verified by checking the data of the target Linux hosts from the Prometheus web interface.

In this guide ,We will see how to setup Grafana and intergrate prometheus as a datasource and setup a monitoring.

What is Grafana?

Grafana is an open source monitoring tool with Dashboards and Visualizations.

Prometheus

It can integrated with several other services such as Prometheus , Elasticsearch , Cloudwatch , Loki , InfluxDB , Graphite etc.

Prometheus

Grafana also helps us to alert to several channels such as Email , Slack , Opsgenie , Webhook , Telegram etc.

Installing Grafana

Make sure you have the root access to perform the below actions.

Adding the stable version of grafana to the repository.

sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"

Now we have to update the repository and install grafana packages.

apt-get install -y apt-transport-https
apt-get install -y software-properties-common wget
apt-get update
apt-get install grafana -y
Prometheus

Once the grafana packages are installed.But default the systemd service will be created for the same.

The file is stored as /usr/lib/systemd/system/grafana-server.service

Prometheus

Lets start the grafana service using the below command.

systemctl start grafana-server

To check the status of the grafana,

systemctl status grafana-server
Prometheus

Enable the grafana service to start on the system boot.

systemctl enable grafana-server
Prometheus

Grafana listens on the port 3000 by default , We can verify the same using below command.

netstat -nltp
Prometheus

Accessing the Grafana Web Interface

We have installed the grafana service , Now its time to access the Grafana web interface.

To open the dashboard , The url as follows,

http://publicipaddress:3000

Note : If you’re hosting grafana in EC2 instance , Make sure port 3000 is whitelisted in the Security group.

You should see the following page.

Prometheus

The default username and the password to login to Grafana web interface is

Username : admin

Password : admin

Then it will ask us to set the new password for the admin user.

provide the password and click change.

Prometheus

We have successfully implemented the Grafana Monitoring tool , The dashboard is as shown below.

Prometheus

Integrating Prometheus with Grafana

To check the node exporters metrics which is alread stored in the prometheus server , We need to integrate prometheus with grafana.

To integrate , Hover to Settings icon and choose Datasources

Under Configuration , You can add the datasources here.

Prometheus

Grafana can query data from these datasource to plot the graphs.

For this tutorial , We are going to use Prometheus as the datasource.

Click Add Data source and then choose Prometheus

Prometheus

Here we will asked to provide few information such as Prometheus URL.

Provide a name for the datasource , and then provide the ip address and the port in which Prometheus is Running.

In our case , It is

http://privateipaddress:9090

Prometheus

Grafana should be able to access port 9090 and it should be whitelisted in the Firewall.

Now click Save and Test , It should response as Data source is working

We have now integrated both the services.

Now Its time to setup alerting mechanism

Setup Alerting in Grafana

For the Grafana to alert to a specific channel or medium, We need to add / configure the same.

Choose Alerting and then click Notification channels

Prometheus

Click Add channel , From the lists of available notification channels , Choose the one where you will be active.

Provide a name for the Notification Channel and then choose Type.

Prometheus

Once done , Click Save.

Now We have datasource and the alerting mechanism.

Lets go ahead and setup dashboard and configure the alerting for the same.

Setting Up Grafana Dashboards

We can setup dashboard by ourselves from the scratch or we can import the existing dashboard from Grafana.

To setup a dashboard for the Node Exporter metrics.

In the left side , Hover to + icon and choose Import

Prometheus

It will ask us the id of the dashboard. The id to import the Node exporter graph is 9096.

Provide the id and then click Load

Give a name for the dashboard and then choose the data source as Prometheus.

Prometheus

and then click Import.

It will collect all the metrics from the Prometheus data source and plot the graph as per the metrics such as CPU usage , Disk usage , Memory usage etc.

Make sure the dashboard id’s are chose as per the version of the Node exporter installed , Else It won’t show up any data in the graphs.

Using the dashboard we have created , We can check the resources used by the servers.

Setup Centralized Monitoring and Alerting Dashboard

We can setup a dashboard where we can check the metrics of the servers and at the same time we can check the alert status.

The id to import the dashboard is 5984

Once the dashboard is imported , You can see Disk usage , Memory and CPU usages.

To setup alerting for this, Lets say If the CPU usage is above 80% (for example) alert it to the Notification channel we have configured.

From the drop down of the CPU usage graph , Choose Edit

Prometheus

From the left navigation pane , Choose the Alerting icon

Provide a name for the alert , and configure how often the data should be evaluated & based on that If the usage if above the threshold we set , It will alert us.

Prometheus

Scroll down and under Notifications , Click + and select the Notification channel

And provide a message for the alert ,For better Understanding.

Prometheus

and then click the Save icon on top right corner of the dashboard.

Prometheus

Repeat the same procedures for the other metrics / graphs.

We can also configure multiple conditions while configuring alerting by various levels such as Warning , High and Critical.

Conclusion

We have successfully implemented a complete Monitoring and Alerting setup for the Linux servers using Node exporter , Prometheus and Grafana.

I hope you really find it helpful.Please do check out my other blogs.