Monitor HTTP Endpoints Using Blackbox Exporter & Prometheus

In this blog post , We will see how to monitor the health of the HTTP endpoints with the help of Prometheus and Blackbox exporter.

And also how we can use grafana to alert in case of HTTP endpoints failure status codes.

In my previous posts , I have explained on,

• How to Install and Configure Prometheus
• Monitoring Linux nodes using Prometheus & Node Exporter
• Setup Monitoring & Alerting using Prometheus & Grafana

Once you have the Prometheus setup in place , Lets go ahead and setup blackbox exporter on the target servers where we are running http endpoints.

What Is Blackbox Exporter?

Blackbox exporter is a (probing exporter) tool to monitor HTTP , DNS , TCP and ICMP endpoints.

With the help of blackbox exporter we can scrape the details of all the endpoints on the target instances with the details such as response time , Status code , SSL certificate expiry , DNS lookup latencies .

Creating User For Exporter

We will create a user account blackbox_exporter , We will run the blackbox exporter service using this user for the security concerns.

Also the files and folder of the blackbox exporter service should be owned by blackbox_exporter

sudo useradd --no-create-home --shell /bin/false blackbox_exporter

Using the above command we have created blackbox_exporter user.

Configuring Blackbox Exporter

The configuration file of the blackbox exporter is configured in YAML format.

The configuration file consists of lists of modules.Each module is the probing configuration in the blackbox exporter.

As we are going to check the Status code of the HTTP endpoint , We are going to use the HTTP prober

First We need to download the latest version of blackbox_exporter from the official respository.

https://prometheus.io/download/#blackbox_exporter

Depending on the Operating system we use , The package can be downloaded.

I will be downloading latest binaries for the Linux operating system.

Download the archive to the target server where the blackbox_exporter should be configured.

We will use wget to download the package directly to the server.

wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz

After the package is downloaded to the server , We need to extract it.

Extract the blackbox_exporter package using the below command.

tar -xvzf blackbox_exporter-0.16.0.linux-amd64.tar.gz

We can find the binary (executable file) and the configuration file of the blackbox exporter.

The binary file will be used to start the blackbox exporter service using the systemd unit file.We will configure it in the next step.

blackbox.yml , this is the file where we will configure the modules and probers.

Lets copy the binary file (blackbox_exporter) to the local user account.

sudo cp blackbox_exporter /usr/local/bin/

Also we should change the ownership of the blackbox exporter executables.

sudo chown blackbox_exporter: /usr/local/bin/blackbox_exporter

Next , We need to setup folder for the blackbox exporter configuration files.

sudo mkdir /etc/blackbox_exporter

And copy the blackbox.yml from the downloaded location to /etc/blackbox_exporter

sudo cp blackbox.yml /etc/blackbox_exporter/

The ownership of the blackbox folder should be changed as follows.

sudo chown -R blackbox_exporter: /etc/blackbox_exporter

By default , The blackbox.yml configuration file will be coming with multiple probers (http , tcp , icmp) enabled.

As our intent in this blog is to setup health check monitoring for http endpoints. Remove the existing configuration of blackbox.yml file and add the below configuration.

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []
      method: GET

Save and close the file.

Now Its time to setup systemd unit file for blackbox exporter service.

cd /etc/systemd/system

Create a file blackbox_exporter.service and add the below contents.

[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml

[Install]
WantedBy=multi-user.target

After adding , Save and close the file.

Lets start the blackbox exporter service using the below command.

systemctl start blackbox_exporter

Enable it to start on system restart / boot.

systemctl enable blackbox_exporter

To check the status of the blackbox exporter service , Run the below command.

systemctl status blackbox_exporter

We can see that the service is UP and running.

By default , blackbox exporter will be listening on the port 9115.

We can check the same by running the below command.

netstat -nltp |grep blackbox

To verify the installation , We are going to scrape the details collected by the blackbox exporter service.

curl http://localhost:9115/metrics

The blackbox exporter is started collecting the metrics and we can confirm that the service is working well.

Configuring Prometheus For Blackbox Exporter

We are going to configure prometheus to collect the metrics gathered by the blackbox exporter service.

For that , We need to add scrape target in the configuration file of the prometheus.

The prometheus configuration file will be stored under /etc/prometheus folder as prometheus.yml file.

Login to the server where the prometheus is configured.

Open the prometheus.yml file.

vi /etc/prometheus/prometheus.yml

Under scrape_configs , We are going to add a new job for the blackbox exporter running on the target servers.

Currently the configuration file of the prometheus should be as shown below.

global:
   scrape_interval: 15s

scrape_configs:
   -  job_name: 'prometheus' 
      scrape_interval: 5s 
      static_configs: 
         - targets: ['127.0.0.1:9090']

Below this we are going to add another job_name as shown below.

   - job_name: 'blackbox_exporter' 
     metrics_path: /probe 
     params: 
       module: [http_2xx] 
     static_configs: 
       - targets: 
           - https://fitdevops.in
     relabel_configs:
       - source_labels: [__address__]
         target_label: __param_target
       - source_labels: [__param_target]
         target_label: instance
       - target_label: __address__
         replacement: localhost:9115

The blackbox exporter running on the port 9115 will be having all the metrics on the /probe endpoint.

For other exporters , In the static_configs we will add the IP address or the hostname of the target instance as the targets.

But for blackbox exporter , We will add the http endpoints that should be monitored for health checks.Lets say , If you are running an nginx web server , with the domain name https://fitdevops.in , We should add it in the configuration appropriately.

We will add the target host separately using the __address__ labels.

For replacement: , Provide the IP address or hostname of the blackbox exporter server.

Once the configurations are added.Save and close the file.

For that changes to take effect , We need to start the prometheus service.

systemctl restart prometheus

Also check the status of the service , If any invalid syntax in the prometheus configuration file , The prometheus service will fail to start.

systemctl status prometheus

Setup Health Check Alerting Using Grafana

Now we have all the probed metrics collected by the blackbox exporter service in the Prometheus data source.

Using this details Lets us build a dashboard in Grafana and setup alerting based on the status code.

Before that , Lets check the target added to prometheus using the Prometheus Web UI.

Using the below url in the browser to access the Prometheus console.

http://ipaddress:9090

You will see the following page.

To check the targets, Under Status Click Targets

We can see the target hosts and the endpoints added for monitoring.

OKay , Lets goahead and Open the Grafana console , If you havn’t configured grafana yet , Check here.

By default , Grafana runs on the port 3000.We can use the IP address and port to access the Grafana admin portal.

http://ipaddress:3000

Login to the Grafana admin portal.

We can use the dashboards already shared by the Grafana for monitoring http endpoints.

The Id of the Grafana dashboard is 4859

But before that , We have to add the prometheus as data source in Grafana.

Hover to Settings icon and click Data sources , Choose Add data source

Select Prometheus , Provide the IP address and the port of Prometheus.

Click Save and Test , You should get a response as Data source is working.

To setup a dashboard, Hover to + icon and Click Import , Provide the id and click Load

Provide a name for the dashboard , Choose the Prometheus Data source and click Import.

You can find the dashboard with the endpoints and their status codes.

We can check the status of the each endpoints added for monitoring in the prometheus configuration file.

Before configuring alerting , We need to set a channel where the alerts will be sent.

Hover to Bell icon , and click Notification Channels

Click New channel , Provide a name and from the lists of available altering types , Select one , Provide the details.

For eg: Choose Email.

For Email addressess , Type the email addresses that should be notified with alerts.

Test and Save it.

To setup a monitoring for the endpoints , Go to the dashboard , Click Timeline: All and Choose Edit

Click the Alarm icon ,

Provide a name for the Alert , And we will specify how often the evaluation should happen for the condition we set.

We will evaluate for every 60 sec.

And the condition is , If the Status code of the HTTP endpoint is above 200 , We will be alerted.

For Notifications , Click + icon and choose the notification channel.Add a message for better understanding about the alerts.

And finally click Save Dashboard.

Conclusion

We have successfully implemented a HTTP endpoint monitoring and alerting using Blackbox Exporter , Prometheus & Grafana.

Going forward , If the HTTP status code of the endpoint is above 200 , We will be alerted immediately.

Hope you find it helpful.

Please do check out my other publications.