Monitor Linux Processes Using Pushgateway & Prometheus
There are certain cases where we cannot push the custom metrics to the prometheus.
In this case , We can make use of the Pushgateway.
In this blog post , We will see how to implement pushgateway on Linux servers to Monitor Linux processes and then configure the prometheus to scrape the custom metrics collected by pushgateway.
Pre-Requisites
Before implementing pushgateway , We need to have a prometheus server setup.
If you have not setup prometheus yet , Check these blogs related to prometheus.
Install and configure prometheus on Ubuntu
Monitor HTTP endpoints using Blackbox exporter & Prometheus
Monitor Linux Nodes using Node Exporter & Prometheus
Setup Monitoring & Alerting using Prometheus & Grafana
What Is Pushgateway?
The Prometheus Pushgateway collects custom metrics such as batch jobs , Linux processes to expose their metrics to Prometheus.
It is a intermediary service which allows us to push to prometheus as these metrics cannot be scraped.
Installing Pushgateway
Pushgateway can be installed in two ways.
1)Using Docker
We can implement pushgateway using the prom/pushgateway docker image.
If you havn’t installed docker yet , Refer this article
Run the below command to pull pushgateway image and run them as docker container.
docker pull prom/pushgateway
We have download the docker images , We can check the docker images using the below command.
docker images prom/pushgateway
Tehn with the help of the docker image , We can run the pushgateway as a docker container on port 9091.
docker run -d -p 9091:9091 prom/pushgateway
You can verify the actively running docker container using the command,
docker ps
To verify , whether the pushgateway is working and started collecting the metrics , Run the below command.
curl http://localhost:9091/metrics
2)Using Binaries
We can get the latest version of pushgateway from the below link.
https://prometheus.io/download/
Download the package and extract it.
wget https://github.com/prometheus/pushgateway/releases/download/v1.2.0/pushgateway-1.2.0.linux-amd64.tar.gz
tar -xvzf pushgateway-1.2.0.linux-amd64.tar.gz
You will get the pushgateway binary within the pushgateway-1.2.0.linux-amd64 folder.
We need to create a user for pushgateway service.
useradd --no-create-home --shell /bin/false pushgateway
Copy or move the pushgateway binary (executables) to the /usr/local/bin folder.
cp pushgateway-1.2.0.linux-amd64/pushgateway /usr/local/bin/
chown pushgateway: /usr/local/bin/pushgateway
To manage the service , We need to create systemd unit file for the pushgateway service.
Go to /etc/systemd/system folder and create a file named pushgateway.service and Add the below contents.
[Unit] Description=Pushgateway Wants=network-online.target After=network-online.target [Service] User=pushgateway Group=pushgateway Type=simple ExecStart=/usr/local/bin/pushgateway --web.listen-address=":9091" --web.telemetry-path="/metrics" --persistence.file="/tmp/metric.store" --persistence.interval=5m [Install] WantedBy=multi-user.target
Save and close the file.
Reload systemd and start the pushgateway service.
systemctl daemon-reload
systemctl start pushgateway
Check the status of the pushgateway service using the below command.
systemctl status pushgateway
We can also verify the installation using the command.
netstat -nltp |grep 9091
Now that the installation of pushgateway is completed ,Lets push the metrics to Prometheus.
Pushing Metrics To Prometheus
We have the custom metrics collected by the pushgateway service.
Now we have to configure the prometheus to scrape the pushgateway for the metrics.
Open the prometheus configuration file.
vi /etc/prometheus/prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'Prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
- job_name: 'NodeExporter'
scrape_interval: 5s
static_configs:
targets: ['localhost:9100']
- job_name: 'Pushgateway'
scrape_interval: 5s
static_configs:
targets: ['localhost:9100']
From the above configuration , I have added a separate job for the pushgateway.
Dont forget to replace localhost with the IP address of the server where pushgateway service is configured.
Save and close the file.
Restart and check the status of prometheus service.
systemctl restart prometheus
systemctl status prometheus
For this tutorial , Lets collect the CPU and memory usage of the top linux processes and push those metrics to the pushgateway service.
Here is the bash script to monitor the cpu and memory consumption by the Linux processses
TOP CPU USAGE
Create a file named cpu-top and add the below contents.
!/bin/bash
z=$(ps aux)
while read -r z
do
var=$var$(awk '{print "cpu_usage{process=\""$11"\", pid=\""$2"\"}", $3z}');
done <<< "$z"
curl -X POST -H "Content-Type: text/plain" --data "$var
" http://localhost:9091/metrics/job/top/instance/machineTOP MEMORY USAGE
Create a file named memory-top and add the below contents.
!/bin/bash
z=$(ps aux)
while read -r z
do
var=$var$(awk '{print "memory_usage{process=\""$11"\", pid=\""$2"\"}", $4z}');
done <<< "$z"
curl -X POST -H "Content-Type: text/plain" --data "$var
" http://localhost:9091/metrics/job/top/instance/machineThe above scripts will run and collect the high cpu and memory usage by the linux processes.
But also we need to configure in such a way that this bash scripts keeps running in the background and push the metrics to prometheus.
For that we are going to install and configured supervisor on the instance where pushgateway is installed.
Installing Supervisor
To install supervisor , Run the below commands
sudo apt-get update && sudo apt-get install python-setuptools
We can install supervisor now.
sudo easy_install supervisor
Then we need to setup directories for supervisor.
sudo mkdir /etc/supervisor
Add the supervisor config.
echo_supervisord_conf > /etc/supervisor/supervisord.conf
Open the /etc/supervisor/supervisord.conf file and add the below line in the [include] section
files=conf.d/*.conf
Save and close the file.
To setup programs under supervisord , We need to create conf.d directory.
mkdir /etc/supervisor/conf.d
Lets setup systemd file for the supervisor.
Create a file supervisord.service and add the below contents.
vi /etc/systemd/system/supervisord.service
[Unit] Description=Supervisor daemon Documentation=http://supervisord.org After=network.target [Service] ExecStart=/usr/local/bin/supervisord -n -c /etc/supervisor/supervisord.conf ExecStop=/usr/local/bin/supervisorctl $OPTIONS shutdown ExecReload=/usr/local/bin/supervisorctl $OPTIONS reload KillMode=process Restart=on-failure RestartSec=42s [Install] WantedBy=multi-user.target Alias=supervisord.service
Save and close the file.
Lets start the supervisor service.
systemctl start supervisord.service
Check the status of the service and enable it to start on system boot,
systemctl status supervisord.service
systemctl enable supervisord.service
Once we have supervisor service UP and running , Lets configure the bash script to be managed by supervisord service.
We already have two file named memory-top and cpu-top
Lets keep this files under /opt directory.
We are going to create another 2 files named memory.sh and cpu.sh in the /opt folder.
Create a file memory.sh and add the below contents
cd /opt while sleep 1; do ./memory-top; done;
Create a file named cpu.sh and add the below contents
cd /opt while sleep 1; do ./cpu-top; done;
make sure the bash scripts are executable by running,
chmod +x cpu.sh && chmod +x memory.sh
By this time , We will have four files under /opt directory
Now we need to setup a program under supervisor to manage these bash scripts.
cd /etc/supervisor/conf.d
Create a file named cpu.conf and add the below configurations.
[program:cpu] command=sh /opt/cpu.sh autostart=true autorestart=true stderr_logfile=/var/log/long.err.log stdout_logfile=/var/log/long.out.log
Create a file named memory.conf and add the below configurations.
[program:memory] command=sh /opt/memory.sh autostart=true autorestart=true stderr_logfile=/var/log/long.err.log stdout_logfile=/var/log/long.out.log
Save and close the file.
We need to run the below commands to let the supervisor to reread the configuration file and apply the latest changes.
supervisorctl reread
supervisorctl update
To run the supervisor client , Run
supervisorctl
To check the status of the process , Run
status
You should get the below response.
We can also check the status of the programs from the linux bash by running,
supervisorctl status
We can manage each programs from the supervisor client.
Now the bash scripts will keep collecting the cpu and memory usage of top running linux processes and sends the metrics to prometheus.
These scripts will be managed by supervisor service.
Setup Dashboard On Grafana
We now have the data collected by the pushgateway are sent to the prometheus.
We are going to setup a dashboard on grafana using the prometheus data source.
In this blog , I have explained how you can add prometheus as data source in Grafana.
Login to Grafana , Hover to + icon and Click Dashboard
You will see the following screen.
Click Add Query , Queries to , Choose Prometheus
Add this prom query , topk(10, memory_usage) for monitoring memory usage
Add this prom query , topk(10, cpu_usage) for monitoring cpu usage
You should be able to find the linux processes and the memory and cpu usage of them.
We can setup an alerting with the help of these metrics from Grafana.
Conclusion
We have successfully implemented a pushgateway to monitor linux processes and monitor them using prometheus and grafana.
Hope you find it helpful.
Please check out my other articles.