How To Install and Configure Blackbox Exporter for Prometheus

17.7K

This guide details all the steps needed to install and configure the Blackbox Exporter with Prometheus.

As a system administrator, or as an application developer, you are often issuing ping commands in order to check the availability of your services.

You may want to check your website uptime, if your website is accessible and if it is loading fast for your users.

If you are a network engineer, you may want to resolve DNS response times to diagnose network latency issues.

You may also want to have an extensive monitoring of your ICMP requests in order to gather more information about your network health.

In this tutorial, we are taking a look at the Blackbox exporter for Prometheus.

The Blackbox exporter is a tool that allows engineers to monitor HTTP, DNS, TCP and ICMP endpoints. Results can be visualized in modern dashboard tools such as Grafana.

Ready?

1 – What You Will Learn

By following this tutorial until the end, here are the concepts that you are going to learn about.

• How to install Prometheus securely using authentication and TLS encryption;
• What is the Blackbox exporter and how it differs from application instrumenting;
• How to install the Blackbox exporter as a service;
• How to bind the Blackbox exporter with Prometheus;
• How to monitor your first HTTP endpoint.

That’s quite a long program, let’s head to it.

2 – Installing Prometheus securely

In our previous tutorials, we already discovered how to install Prometheus with Grafana on Linux operating systems.

If you carefully followed this tutorial, your Prometheus server is currently sitting behind a reverse proxy with authentication enabled.

In this tutorial, we are going to use the https://localhost:1234 URL in order to reach Prometheus with the Blackbox exporter.

However, if you configured Prometheus on another URL, you will need to change the configuration options provided in this tutorial.

3 – What is the Blackbox exporter with Prometheus?

The Blackbox exporter is a probing exporter used to monitor network endpoints such as HTTP, HTTPS, DNS, ICMP or TCP endpoints.

a – Blackbox general concepts

The Blackbox exporter provides metrics about HTTP latencies, DNS lookups latencies as well as statistics about SSL certificates expiration.

The Blackbox exporter is mainly used to measure response times.

As a consequence, if you are looking for more detailed metrics for your application (for example a Python application), you will have to instrument your application.

It is noteworthy to say that the Blackbox exporter can be bound with the AlertManager and Prometheus in order to have detailed alerts when one endpoint goes down.

When running, the Blackbox exporter is going to expose a HTTP endpoint that can be used in order to monitor targets over the network. By default, the Blackbox exporter exposes the /probe endpoint that is used to retrieve those metrics.

For example, if my Blackbox exporter is running on port 9115, and if I query metrics for google.com, this is the endpoint that I can query from the exporter.

$ http://localhost:9115/probe?target=https://google.com&module=https_2xx

As you probably understand, the Blackbox exporter is a standalone tool, it does not need any other tools to run.

On the other hand, Prometheus binds to the exporter. You are going to define ‘targets’ in a dedicated Blackbox configuration section, and Prometheus will issue requests to the probe endpoint we saw earlier.

Prometheus is acting as a way to automate requests and as a way to store them for long-term storage.

b – What are the Blackbox modules?

As most of the Prometheus ecosystem tools, the blackbox exporter is configured with a YAML configuration file.

The Blackbox exporter configuration file is made of modules.

A module can be seen as one probing configuration for the Blackbox exporter.

As a consequence, if you choose to have a HTTP prober checking for successful HTTPS responses (2xx HTTP codes for example), the configuration will be summarized within the same module.

The documentation for modules is available here. As you can see, you have many options for all the probers that are available to you.

Monitoring endpoints availability is either a success of a failure.

As a consequence, the Blackbox exporter will report if it successfully probed the targets it was assigned (0 for a failure and 1 for a success).

Given the value of the probing, you can choose to define alerts in order to be notified (on Slack for example) when an endpoint goes down.

c – How does the Blackbox exporter differs from application instrumenting?

As a reminder, application instrumenting means that you are adding client libraries to an application in order to expose metrics to Prometheus.

Client libraries are available for most programming languages and frameworks such as Go, Python, Java, Javascript or Ruby.

The main difference between the Blackbox exporter and application instrumenting is that the Blackbox exporter only focuses on availability while instrumentations can go more into details about performance.

With instrumentation, you can choose to monitor the performance of a single SQL query for example, or a single server function.

The Blackbox exporter will not be able to expose this granularity, only a request failure or a request success.

Now that you have some concepts about the Blackbox exporter, let’s start installing it as a service on our host.

4 – Installing the Blackbox exporter for Prometheus

First of all, you are going to download the latest version of the Blackbox exporter available for Prometheus.

a – Downloading the Blackbox exporter

To download the Blackbox exporter, head over to Prometheus downloads page.

Filter your results by choosing Linux as the current operating system.

Scroll down a bit, and find the blackbox exporter executable, right below the AlertManager section.

As you can see, at the time of this tutorial, the Blackbox exporter is available on the 0.14.0 version.

Click on the archive to download it. If you are more familiar with wget, copy the link and run this command.

$ wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.14.0/blackbox_exporter-0.14.0.linux-amd64.tar.gz

The archive should be correctly downloaded on your host. To extract the archive, run the following command

$ tar xvzf blackbox_exporter-0.14.0.linux-amd64.tar.gz

Besides the license and the notice files, the archive contains two important files:

• blackbox_exporter: the executable for the Blackbox exporter. This is the executable that you are going to launch via your service file in order to probe targets;
• blackbox.yml: configuration file for the Blackbox exporter. This is where you are going to define your modules and probers.

Documentation for the blackbox_exporter executable is available when running the following command

$ cd blackbox_exporter-0.14.0.linux-amd64
$ ./blackbox_exporter -h

As you can see, the configuration for the Blackbox exporter is pretty straightforward. Those are the flags that we are going to use when defining our systemd service file.

b – Create a service file for the Blackbox exporter

As a meticulous system administrator, you are not going to launch executables from your home directory.

Instead, you are going to define service files and define start and restart policies for it.

First, make your executables accessible for your user local account.

$ sudo mv blackbox_exporter /usr/local/bin

Note : by moving files to the /usr/local/bin path, all users may have access to the blackbox exporter binary. For binaries to be restricted to your own user account, you need to add the path to your own PATH environment variable.

Next, create configuration folders for your blackbox exporter.

$ sudo mkdir -p /etc/blackbox
$ sudo mv blackbox.yml /etc/blackbox

For safety purposes, the Blackbox exporter is going to be run by its own user account (named blackbox here)

As a consequence, we need to create a user account for the Blackbox exporter.

$ sudo useradd -rs /bin/false blackbox

Then, make sure that the blackbox binary can be run by your newly created user.

$ sudo chown blackbox:blackbox /usr/local/bin/blackbox_exporter

Give the correct permissions to your configuration folders recursively.

$ sudo chown -R blackbox:blackbox /etc/blackbox/*

Now that everything is set, it is time for you to create your service file.

To create the Blackbox exporter service, head over to the /lib/systemd/system folder and create a service named blackbox.service

$ cd /lib/systemd/system
$ sudo touch blackbox.service

Edit your service file, and paste the following content in it.

$ sudo nano blackbox.service

[Unit]
Description=Blackbox Exporter Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=blackbox
Group=blackbox
ExecStart=/usr/local/bin/blackbox_exporter \
  --config.file=/etc/blackbox/blackbox.yml \
  --web.listen-address=":9115"

Restart=always

[Install]
WantedBy=multi-user.target

Note: as you can see, the Blackbox exporter service has no dependencies to other tools of the Prometheus ecosystem or to Prometheus itself.

Save your service file, and make sure that your service is enabled at boot time.

$ sudo systemctl enable blackbox.service
$ sudo systemctl start blackbox.service

For now the Blackbox exporter is not configured to scrape any targets, but we are going to add a few ones in the next section.

If your service is correctly running, you can check the metrics gathered by issuing a request to the HTTP API.

$ curl http://localhost:9115/metrics

If you navigate to the Blackbox exporter URL with a Web Browser, this is what you should see on your screen.

Now that your Blackbox exporter is gathering metrics, it is time to bind it to Prometheus.

c – Binding the Blackbox exporter with Prometheus

Important note: in this section, Prometheus is going to scrape the Blackbox Exporter to gather metrics about the exporter itself.

To configure Prometheus to scrape HTTP targets, head over to the next sections.

To bind the Blackbox exporter with Prometheus, you need to add it as a scrape target in Prometheus configuration file.

If you follow the Prometheus setup tutorial, your configuration file is stored at /etc/prometheus/prometheus.yml

Edit this configuration file, and amend the following changes

$ sudo nano /etc/prometheus/prometheus.yml

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090', 'localhost:9115']

With Prometheus, you don’t need to restart the systemd service for Prometheus to update.

You can also send a signal to the process for it to restart.

In order to send a SIGHUP signal to your Prometheus process, identify the PID of your Prometheus server.

$ ps aux | grep prometheus
schkn  4431  0.0  0.0  14856  1136 pts/0    S+   09:34   0:00 /bin/prometheus --config.file=...

The PID of my Prometheus process is 4431. Send a SIGHUP signal to this process for the configuration to restart.

$ sudo kill -HUP 4431

Head over to your Prometheus target configuration (http://localhost:9090/targets), and check that you are correctly scrapping your Blackbox exporter.

Great!

Now that the Blackbox exporter is configured with Prometheus, it is time to add our first target.

As a matter of simplicity, we are going to monitor the HTTP endpoint of Prometheus itself with the Blackbox exporter.

5 – Monitoring HTTPS endpoints with the Blackbox Exporter

In our setup, Prometheus is now currently sitting behind a reverse proxy (NGINX) configured with self signed certificates.

This is the endpoint we are going to monitor with the Blackbox Exporter.

a – Creating a Blackbox module

To monitor Prometheus, we are going to use the HTTP prober.

Head over to your Blackbox configuration file, erase its content and paste the following configuration.

modules:
  http_prometheus:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      method: GET
      fail_if_ssl: false
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: true
      basic_auth:
        username: "username"
        password: "password"

Here are the details of the parameters we chose.

• fail_if_not_ssl: as we are actively monitoring a HTTPS endpoint, we need to make sure that we are retrieving the page with SSL encryption. Otherwise, we count it as a failure;
• insecure_skip_verify: if you followed our previous tutorial, we generated our certificates with self-signed certificates. As a consequence, you are not able to verify it with a certificate authority;
• basic_auth: the reverse proxy endpoint is configured with a basic username/password authentication. The Blackbox exporter needs to be aware of those to probe the Prometheus server.

Save your file, and check your configuration file with the blackbox binary itself.

$ blackbox_exporter --config.check

Great!

Our configuration file seems to be correctly formatted.

Again, there is no need to restart the Blackbox Exporter service. Instead, send a simple SIGHUP signal to the process.

Similarly to the Prometheus process, identify the Blackbox exporter PID.

$ ps aux | grep blackbox
devconnected 574  0.0  0.0  14856  1136 pts/0    S+   09:34   0:00 /usr/local/bin/blackbox_exporter --config.file=...

The PID of my Prometheus process is 574.

Send a SIGHUP signal to this process for the configuration to restart.

$ sudo kill -HUP 574

b – Binding the Blackbox Exporter Module in Prometheus

Now that the module is defined, it is time for Prometheus to start actively using it to monitor our target.

To do so, head over to Prometheus configuration file, and paste the following changes.

scrape_configs:

...

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_prometheus] 
    static_configs:
      - targets:
        - https://127.0.0.1:1234    # Target to probe with https.
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115  # The blackbox exporter's real hostname:port.

Save your changes, and send a SIGHUP signal to Prometheus for it to restart.

When your server has restarted, head over to https://localhost:1234/config and make sure that your changes were correctly saved.

Great!

Prometheus will start scrapping our target now.

To verify it, head over to the /graph endpoint, and issue the following PromQL request.

probe_success{instance="https://127.0.0.1:1234", job="blackbox"}

If your target is up, the probe success should return 1.

Now that your Blackbox exporter is all set, let’s have a quick Grafana dashboard in order to visualize your results.

6 – Visualizing HTTP metrics on Grafana

a – Installing Grafana on Linux

As always, if you need to install Grafana on your Linux host, make sure to read the dedicated tutorial.

We are not going to create the dashboard by ourselves, instead we are going to import an existing one that answers our needs.

b – Importing a Grafana dashboard

In order to import a Grafana dashboard, click on the “Plus” icon on the left menu, and click on “Import”.

On the next window, select the Grafana.com dashboard option, and type the following dashboard ID : 7587.

From there, Grafana should automatically detect your dashboard.

Select Prometheus as a datasource, and click on “Import“.

That’s it! By clicking on “Import”, your dashboard was automatically created on Grafana.

This is what you should now see on your screen.

As you can see, this dashboard focuses on HTTP(s) metrics.

You can have metrics about the up status of your website, the current SSL status as well as the SSL expiry date.

You also have graphs showing the current latencies of your HTTP requests, as well as the average DNS lookup time.

It is noteworthy to say that you can choose the target that you want to monitor by clicking on the “Target” dropdown at the top of the dashboard. In our case, we have only one target, but it can become quite handy when you are monitoring several hosts.

7 – Conclusion

Congratulations, you did it!

You successfully installed the Blackbox Exporter with Prometheus.

You also learnt how you can install Grafana and import your first dashboard.

This tutorial is only the beginning of your learning path on becoming a monitoring expert.

If you are looking for more resources, we have a complete section dedicated to it.

Click the image below to be redirected to our monitoring section.

I hope that you learnt something new today.

Until then, have fun, as always.