There are certain cases where we cannot push the custom metrics to the prometheus.

In this case , We can make use of the Pushgateway.

In this blog post , We will see how to implement pushgateway on Linux servers to Monitor Linux processes and then configure the prometheus to scrape the custom metrics collected by pushgateway.

Pre-Requisites

Before implementing pushgateway , We need to have a prometheus server setup.

If you have not setup prometheus yet , Check these blogs related to prometheus.

Install and configure prometheus on Ubuntu

Monitor HTTP endpoints using Blackbox exporter & Prometheus

Monitor Linux Nodes using Node Exporter & Prometheus

Setup Monitoring & Alerting using Prometheus & Grafana

What is Pushgateway?

The Prometheus Pushgateway collects custom metrics such as batch jobs , Linux processes to expose their metrics to Prometheus.

It is a intermediary service which allows us to push to prometheus as these metrics cannot be scraped.

Installing Pushgateway

Pushgateway can be installed in two ways.

1)Using Docker

We can implement pushgateway using the prom/pushgateway docker image.

If you havn’t installed docker yet , Refer this article

Run the below command to pull pushgateway image and run them as docker container.

docker pull prom/pushgateway
Monitor Linux Processes

We have download the docker images , We can check the docker images using the below command.

docker images prom/pushgateway
Monitor Linux Processes

Tehn with the help of the docker image , We can run the pushgateway as a docker container on port 9091.

docker run -d -p 9091:9091 prom/pushgateway
Monitor Linux Processes

You can verify the actively running docker container using the command,

docker ps
Monitor Linux Processes

To verify , whether the pushgateway is working and started collecting the metrics , Run the below command.

curl http://localhost:9091/metrics
Monitor Linux Processes
Monitor Linux Processes

2)Using Binaries

We can get the latest version of pushgateway from the below link.

https://prometheus.io/download/

Download the package and extract it.

wget https://github.com/prometheus/pushgateway/releases/download/v1.2.0/pushgateway-1.2.0.linux-amd64.tar.gz
Monitor Linux Processes
tar -xvzf pushgateway-1.2.0.linux-amd64.tar.gz
Monitor Linux Processes

You will get the pushgateway binary within the pushgateway-1.2.0.linux-amd64 folder.

We need to create a user for pushgateway service.

useradd --no-create-home --shell /bin/false pushgateway

Copy or move the pushgateway binary (executables) to the /usr/local/bin folder.

cp pushgateway-1.2.0.linux-amd64/pushgateway /usr/local/bin/
chown pushgateway: /usr/local/bin/pushgateway

To manage the service , We need to create systemd unit file for the pushgateway service.

Go to /etc/systemd/system folder and create a file named pushgateway.service and Add the below contents.

[Unit]
Description=Pushgateway
Wants=network-online.target
After=network-online.target

[Service]
User=pushgateway
Group=pushgateway
Type=simple
ExecStart=/usr/local/bin/pushgateway --web.listen-address=":9091" --web.telemetry-path="/metrics" --persistence.file="/tmp/metric.store" --persistence.interval=5m

[Install]
WantedBy=multi-user.target

Save and close the file.

Reload systemd and start the pushgateway service.

systemctl daemon-reload
systemctl start pushgateway

Check the status of the pushgateway service using the below command.

systemctl status pushgateway
Monitor Linux Processes

We can also verify the installation using the command.

netstat -nltp |grep 9091
Monitor Linux Processes

Now that the installation of pushgateway is completed ,Lets push the metrics to Prometheus.

Pushing metrics to Prometheus

We have the custom metrics collected by the pushgateway service.

Now we have to configure the prometheus to scrape the pushgateway for the metrics.

Open the prometheus configuration file.

vi /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'Prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'NodeExporter'
    scrape_interval: 5s
    static_configs:
    targets: ['localhost:9100']

  - job_name: 'Pushgateway'
    scrape_interval: 5s
    static_configs:
    targets: ['localhost:9100']

From the above configuration , I have added a separate job for the pushgateway.

Dont forget to replace localhost with the IP address of the server where pushgateway service is configured.

Save and close the file.

Restart and check the status of prometheus service.

systemctl restart prometheus
systemctl status prometheus

For this tutorial , Lets collect the CPU and memory usage of the top linux processes and push those metrics to the pushgateway service.

Here is the bash script to monitor the cpu and memory consumption by the Linux processses

TOP CPU USAGE

Create a file named cpu-top and add the below contents.

!/bin/bash
z=$(ps aux)
while read -r z
do
var=$var$(awk '{print "cpu_usage{process=\""$11"\", pid=\""$2"\"}", $3z}');
done <<< "$z"
curl -X POST -H "Content-Type: text/plain" --data "$var
" http://localhost:9091/metrics/job/top/instance/machine

TOP MEMORY USAGE

Create a file named memory-top and add the below contents.

!/bin/bash
z=$(ps aux)
while read -r z
do
var=$var$(awk '{print "memory_usage{process=\""$11"\", pid=\""$2"\"}", $4z}');
done <<< "$z"
curl -X POST -H "Content-Type: text/plain" --data "$var
" http://localhost:9091/metrics/job/top/instance/machine

The above scripts will run and collect the high cpu and memory usage by the linux processes.

But also we need to configure in such a way that this bash scripts keeps running in the background and push the metrics to prometheus.

For that we are going to install and configured supervisor on the instance where pushgateway is installed.

Installing Supervisor

To install supervisor , Run the below commands

sudo apt-get update && sudo apt-get install python-setuptools

We can install supervisor now.

sudo easy_install supervisor

Then we need to setup directories for supervisor.

sudo mkdir /etc/supervisor

Add the supervisor config.

echo_supervisord_conf > /etc/supervisor/supervisord.conf

Open the /etc/supervisor/supervisord.conf file and add the below line in the [include] section

 files=conf.d/*.conf
Monitor Linux Processes

Save and close the file.

To setup programs under supervisord , We need to create conf.d directory.

mkdir /etc/supervisor/conf.d

Lets setup systemd file for the supervisor.

Create a file supervisord.service and add the below contents.

vi /etc/systemd/system/supervisord.service
[Unit]
Description=Supervisor daemon
Documentation=http://supervisord.org
After=network.target

[Service]
ExecStart=/usr/local/bin/supervisord -n -c /etc/supervisor/supervisord.conf
ExecStop=/usr/local/bin/supervisorctl $OPTIONS shutdown
ExecReload=/usr/local/bin/supervisorctl $OPTIONS reload
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target
Alias=supervisord.service

Save and close the file.

Lets start the supervisor service.

systemctl start supervisord.service

Check the status of the service and enable it to start on system boot,

systemctl status supervisord.service
systemctl enable supervisord.service

Once we have supervisor service UP and running , Lets configure the bash script to be managed by supervisord service.

We already have two file named memory-top and cpu-top

Lets keep this files under /opt directory.

We are going to create another 2 files named memory.sh and cpu.sh in the /opt folder.

Create a file memory.sh and add the below contents

cd /opt
while sleep 1; do ./memory-top; done;

Create a file named cpu.sh and add the below contents

cd /opt
while sleep 1; do ./cpu-top; done;

make sure the bash scripts are executable by running,

chmod +x cpu.sh && chmod +x memory.sh

By this time , We will have four files under /opt directory

Now we need to setup a program under supervisor to manage these bash scripts.

cd /etc/supervisor/conf.d

Create a file named cpu.conf and add the below configurations.

[program:cpu]
command=sh /opt/cpu.sh
autostart=true
autorestart=true
stderr_logfile=/var/log/long.err.log
stdout_logfile=/var/log/long.out.log

Create a file named memory.conf and add the below configurations.

[program:memory]
command=sh /opt/memory.sh
autostart=true
autorestart=true
stderr_logfile=/var/log/long.err.log
stdout_logfile=/var/log/long.out.log

Save and close the file.

We need to run the below commands to let the supervisor to reread the configuration file and apply the latest changes.

supervisorctl reread
supervisorctl update

To run the supervisor client , Run

supervisorctl

To check the status of the process , Run

status

You should get the below response.

Monitor Linux Processes

We can also check the status of the programs from the linux bash by running,

supervisorctl status

We can manage each programs from the supervisor client.

Now the bash scripts will keep collecting the cpu and memory usage of top running linux processes and sends the metrics to prometheus.

These scripts will be managed by supervisor service.

Setup Dashboard on Grafana

We now have the data collected by the pushgateway are sent to the prometheus.

We are going to setup a dashboard on grafana using the prometheus data source.

In this blog , I have explained how you can add prometheus as data source in Grafana.

Login to Grafana , Hover to + icon and Click Dashboard

You will see the following screen.

Monitor Linux Processes

Click Add Query , Queries to , Choose Prometheus

Add this prom query , topk(10, memory_usage) for monitoring memory usage

Monitor Linux Processes

Add this prom query , topk(10, cpu_usage) for monitoring cpu usage

Monitor Linux Processes

You should be able to find the linux processes and the memory and cpu usage of them.

We can setup an alerting with the help of these metrics from Grafana.

Conclusion

We have successfully implemented a pushgateway to monitor linux processes and monitor them using prometheus and grafana.

Hope you find it helpful.

Please check out my other articles.