Setup Monitoring And Alerting Using Prometheus & Grafana
In my previous blogs , I have explained in detail on How to
Install and configure Prometheus
Monitoring Linux nodes using Node Exporter
From the above articles , We did a setup of Node exporter on the target Linux hosts and Prometheus scraps the metrics from the Node exporters.
Also we have verified by checking the data of the target Linux hosts from the Prometheus web interface.
In this guide ,We will see how to setup Grafana and intergrate prometheus as a datasource and setup a monitoring.
What Is Grafana?
Grafana is an open source monitoring tool with Dashboards and Visualizations.
It can integrated with several other services such as Prometheus , Elasticsearch , Cloudwatch , Loki , InfluxDB , Graphite etc.
Grafana also helps us to alert to several channels such as Email , Slack , Opsgenie , Webhook , Telegram etc.
Installing Grafana
Make sure you have the root access to perform the below actions.
Adding the stable version of grafana to the repository.
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
Now we have to update the repository and install grafana packages.
apt-get install -y apt-transport-https
apt-get install -y software-properties-common wget
apt-get update
apt-get install grafana -y
Once the grafana packages are installed.But default the systemd service will be created for the same.
The file is stored as /usr/lib/systemd/system/grafana-server.service
Lets start the grafana service using the below command.
systemctl start grafana-server
To check the status of the grafana,
systemctl status grafana-server
Enable the grafana service to start on the system boot.
systemctl enable grafana-server
Grafana listens on the port 3000 by default , We can verify the same using below command.
netstat -nltp
Accessing The Grafana Web Interface
We have installed the grafana service , Now its time to access the Grafana web interface.
To open the dashboard , The url as follows,
http://publicipaddress:3000
Note : If you’re hosting grafana in EC2 instance , Make sure port 3000 is whitelisted in the Security group.
You should see the following page.
The default username and the password to login to Grafana web interface is
Username : admin
Password : admin
Then it will ask us to set the new password for the admin user.
provide the password and click change.
We have successfully implemented the Grafana Monitoring tool , The dashboard is as shown below.
Integrating Prometheus With Grafana
To check the node exporters metrics which is alread stored in the prometheus server , We need to integrate prometheus with grafana.
To integrate , Hover to Settings icon and choose Datasources
Under Configuration , You can add the datasources here.
Grafana can query data from these datasource to plot the graphs.
For this tutorial , We are going to use Prometheus as the datasource.
Click Add Data source and then choose Prometheus
Here we will asked to provide few information such as Prometheus URL.
Provide a name for the datasource , and then provide the ip address and the port in which Prometheus is Running.
In our case , It is
http://privateipaddress:9090
Grafana should be able to access port 9090 and it should be whitelisted in the Firewall.
Now click Save and Test , It should response as Data source is working
We have now integrated both the services.
Now Its time to setup alerting mechanism
Setup Alerting In Grafana
For the Grafana to alert to a specific channel or medium, We need to add / configure the same.
Choose Alerting and then click Notification channels
Click Add channel , From the lists of available notification channels , Choose the one where you will be active.
Provide a name for the Notification Channel and then choose Type.
Once done , Click Save.
Now We have datasource and the alerting mechanism.
Lets go ahead and setup dashboard and configure the alerting for the same.
Setting Up Grafana Dashboards
We can setup dashboard by ourselves from the scratch or we can import the existing dashboard from Grafana.
To setup a dashboard for the Node Exporter metrics.
In the left side , Hover to + icon and choose Import
It will ask us the id of the dashboard. The id to import the Node exporter graph is 9096.
Provide the id and then click Load
Give a name for the dashboard and then choose the data source as Prometheus.
and then click Import.
It will collect all the metrics from the Prometheus data source and plot the graph as per the metrics such as CPU usage , Disk usage , Memory usage etc.
Make sure the dashboard id’s are chose as per the version of the Node exporter installed , Else It won’t show up any data in the graphs.
Using the dashboard we have created , We can check the resources used by the servers.
Setup Centralized Monitoring And Alerting Dashboard
We can setup a dashboard where we can check the metrics of the servers and at the same time we can check the alert status.
The id to import the dashboard is 5984
Once the dashboard is imported , You can see Disk usage , Memory and CPU usages.
To setup alerting for this, Lets say If the CPU usage is above 80% (for example) alert it to the Notification channel we have configured.
From the drop down of the CPU usage graph , Choose Edit
From the left navigation pane , Choose the Alerting icon
Provide a name for the alert , and configure how often the data should be evaluated & based on that If the usage if above the threshold we set , It will alert us.
Scroll down and under Notifications , Click + and select the Notification channel
And provide a message for the alert ,For better Understanding.
and then click the Save icon on top right corner of the dashboard.
Repeat the same procedures for the other metrics / graphs.
We can also configure multiple conditions while configuring alerting by various levels such as Warning , High and Critical.
Conclusion
We have successfully implemented a complete Monitoring and Alerting setup for the Linux servers using Node exporter , Prometheus and Grafana.
I hope you really find it helpful.Please do check out my other blogs.