Home MonitoringGrafana How To Setup Telegraf InfluxDB and Grafana on Linux

How To Setup Telegraf InfluxDB and Grafana on Linux

by schkn

From all the existing modern monitoring tools, the TIG (Telegraf, InfluxDB and Grafana) stack is probably one of the most popular ones.

This stack can be used to monitor a wide panel of different datasources: from operating systems (such as Linux or Windows performance metrics), to databases (such as MongoDB or MySQL), the possibilities are endless.

The principle of the TIG stack is easy to understand.

Telegraf is an agent responsible for gathering and aggregating data, like the current CPU usage for example.

InfluxDB will store data, and expose it to Grafana, which is a modern dashboarding solution.

In this tutorial, we are going to learn how to setup Telegraf, InfluxDB and Grafana.

We are also going to secure our instances with HTTPS via secure certificates.

Setup Telegraf, InfluxDB and Grafana for Linux metrics

This tutorial is going to cover steps for Influx 1.7.x, but I will link to the InfluxDB 2.x setup as soon as it is written.

Before starting, make sure that you have sudo privileges on the system, otherwise you won’t be able to install any packages.

Ready?

I – Installing InfluxDB

The complete InfluxDB installation has already been covered in one of our previous articles.

II – Installing Telegraf

Telegraf is an agent that collects metrics related to a wide panel of different targets.

It can also be used as a tool to process, aggregate, split or group data.

The whole list of available targets (also called inputs) is available here.

In our case, we are going to use InfluxDB as an output.

a – Getting packages on Ubuntu distributions

To download packages on Ubuntu 18.04+, run the following commands:

$ wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
$ source /etc/lsb-release
$ echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

This is the output what you should see.

Adding InfluxDB packages to Ubuntu

b – Getting packages on Debian distributions.

To install Telegraf on Debian 10+ distributions, run the following commands:

First, update your apt packages and install the apt-transport-https package.

$ sudo apt-get update
$ sudo apt-get install apt-transport-https

Finally, add the InfluxData keys on your instance.

Given your Debian version, you have to choose for the corresponding packages.

$ cat /etc/debian_version
10.0
Checking Debian version
$ wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
$ source /etc/os-release


# Debian 7 Wheezy
$ test $VERSION_ID = "7" && echo "deb https://repos.influxdata.com/debian wheezy stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

# Debian 8 Jessie
$ test $VERSION_ID = "8" && echo "deb https://repos.influxdata.com/debian jessie stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

# Debian 9 Stretch
$ test $VERSION_ID = "9" && echo "deb https://repos.influxdata.com/debian stretch stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

# Debian 10 Buster
$ test $VERSION_ID = "10" && echo "deb https://repos.influxdata.com/debian buster stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
Adding InfluxDB packages on Debian 10 Buster

c – Install Telegraf as a service

Now that all the packages are available, it is time for you to install them.

Update your package list and install Telegraf as a service.

$ sudo apt-get update
$ sudo apt-get install telegraf

d – Verify your Telegraf installation

Right now, Telegraf should run as a service on your server.

To verify it, run the following command:

$ sudo systemctl status telegraf

Telegraf should run automatically, but if this is not the case, make sure to start it.

$ sudo systemctl start telegraf
Checking Telegraf service health

However, even if your service is running, it does not guarantee that it is correctly sending data to InfluxDB.

To verify it, check your journal logs.

$ sudo journalctl -f -u telegraf.service
Checking Telegraf service through journalctl

If you are having error messages in this section, please refer to the troubleshooting section at the end.

III – Configure InfluxDB Authentication

In order to have a correct TIG stack setup, we are going to setup InfluxDB authentication for users to be logged in when accessing the InfluxDB server.

a – Create an admin account on your InfluxDB server

Before enabled HTTP authentication, you are going to need an admin account.

To do so, head over to the InfluxDB CLI.

$ influx
Connected to http://localhost:8086 version 1.7.7
InfluxDB shell version: 1.7.7

> CREATE USER admin WITH PASSWORD 'password' WITH ALL PRIVILEGES
> SHOW USERS

user   admin
----   -----
admin  true

b – Create a user account for Telegraf

Now that you have an admin account, create an account for Telegraf

> CREATE USER telegraf WITH PASSWORD 'password' WITH ALL PRIVILEGES
> SHOW USERS

user      admin
----      -----
admin     true
telegraf  true

c – Enable HTTP authentication on your InfluxDB server

HTTP authentication needs to be enabled in the InfluxDB configuration file.

Head over to /etc/influxdb/influxdb.conf and edit the following lines.

[http]
  # Determines whether HTTP endpoint is enabled.
  enabled = true
  
  # The bind address used by the HTTP service.
  bind-address = ":8086"

  # Determines whether user authentication is enabled over HTTP/HTTPS.
  auth-enabled = true

d – Configure HTTP authentication on Telegraf

Now that a user account is created for Telegraf, we are going to make sure that it uses it to write data.

Head over to the configuration file of Telegraf, located at /etc/telegraf/telegraf.conf.

Modify the following lines :

## HTTP Basic Auth
  username = "telegraf"
  password = "password"

Restart the Telegraf service, as well as the InfluxDB service.

$ sudo systemctl restart influxdb
$ sudo systemctl restart telegraf

Again, check that you are not getting any errors when restarting the service.

$ sudo journalctl -f -u telegraf.service

Awesome, our requests are now authenticated.

Time to encrypt them.

IV – Configure HTTPS on InfluxDB

Configuring secure protocols between Telegraf and InfluxDB is a very important step.

You would not want anyone to be able to sniff data you are sending to your InfluxDB server.

If your Telegraf instances are running remotely (on a Raspberry Pi for example), securing data transfer is a mandatory step as there is a very high chance that somebody will be able to read the data you are sending.

a – Create a private key for your InfluxDB server

First, install the gnutls-utils package that might come as gnutls-bin on Debian distributions for example.

$ sudo apt-get install gnutls-utils
(or)
$ sudo apt-get install gnutls-bin

Now that you have the certtool installed, generate a private key for your InfluxDB server.

Head over to the /etc/ssl folder of your Linux distribution and create a new folder for InfluxDB.

$ sudo mkdir influxdb && cd influxdb
$ sudo certtool --generate-privkey --outfile server-key.pem --bits 2048

b – Create a public key for your InfluxDB server

$ sudo certtool --generate-self-signed --load-privkey server-key.prm --outfile server-cert.pem

Great! You now have a key pair for your InfluxDB server.

Do not forget to set permissions for the InfluxDB user and group.

$ sudo chown influxdb:influxdb server-key.pem server-cert.pem

c – Enable HTTPS on your InfluxDB server

Now that your certificates are created, it is time to tweak our InfluxDB configuration file to enable HTTPS.

Head over to /etc/influxdb/influxdb.conf and modify the following lines.

# Determines whether HTTPS is enabled.
  https-enabled = true

# The SSL certificate to use when HTTPS is enabled.
https-certificate = "/etc/ssl/influxdb/server-cert.pem"

# Use a separate private key location.
https-private-key = "/etc/ssl/influxdb/server-key.pem"

Restart the InfluxDB service and make sure that you are not getting any errors.

$ sudo systemctl restart influxdb
$ sudo journalctl -f -u influxdb.service

d – Configure Telegraf for HTTPS

Now that HTTPS is available on the InfluxDB server, it is time for Telegraf to reach InfluxDB via HTTPS.

Head over to /etc/telegraf/telegraf.conf and modify the following lines.

# Configuration for sending metrics to InfluxDB
[[outputs.influxdb]]

# https, not http!
urls = ["https://127.0.0.1:8086"]

## Use TLS but skip chain & host verification
insecure_skip_verify = true

Why are we enabling the insecure_skip_verify parameter?

Because we are using a self-signed certificate.

As a consequence, the InfluxDB server identify is not certified by a certificate authority. If you want an example of what a full-TLS authentication looks like, make sure to read the guide to centralized logging on Linux.

Restart Telegraf, and again make sure that you are not getting any errors.

$ sudo systemctl restart telegraf
$ sudo journalctl -f -u telegraf.service

IV – Exploring your metrics on InfluxDB

Before installing Grafana and creating our first Telegraf dashboard, let’s have a quick look at how Telegraf aggregates our metrics.

By default, for Linux systems, Telegraf will start gathering related to the performance of your system via plugins named cpu, disk, diskio, kernel, mem, processes, swap and system.

Names are pretty self-explanatory, those plugins gather some metrics on the CPU usage, the memory usage as well as the current disk read and write IO operations.

Looking for a tutorial dedicated to Disk I/O? Here’s how to setup Grafana and Prometheus to monitor Disk I/O in real time.

Let’s have a quick look at one of the measurements.

To do this, use the InfluxDB CLI with the following parameters.

Data is stored in the “telegraf” database, each measurement being named as the name of the input plugin.

$ influx -ssl -unsafeSsl -username 'admin' -password 'password'
Connected to http://localhost:8086 version 1.7.7
InfluxDB shell version: 1.7.7

> USE telegraf
> SELECT * FROM cpu WHERE time > now() - 30s
Example of an InfluxDB query

Great!

Data is correctly being aggregated on the InfluxDB server.

It is time to setup Grafana and build our first system dashboard.

V – Installing Grafana

The installation of Grafana has already been covered extensively in our previous tutorials.

You can follow the instructions detailed here in order to install it.

a – Add InfluxDB as a datasource on Grafana

In the left menu, click on the Configuration > Data sources section.

Selecting the data sources in Grafana menu

In the next window, click on “Add datasource“.

Add data source button in Grafana

In the datasource selection panel, choose InfluxDB as a datasource.

InfluxDB datasource option in Grafana

Here is the configuration you have to match to configure InfluxDB on Grafana.

Complete InfluxDB configuration in Grafana

Click on “Save and Test”, and make sure that you are not getting any errors.

Data source is working confirmation box

Getting a 502 Bad Gateway error? Make sure that your URL field is set to HTTPS and not HTTP.

If everything is okay, it is time to create our Telegraf dashboard.

b – Importing a Grafana dashboard

We are not going to create a Grafana dashboard for Telegraf, we are going to use a pre-existing one already developed by the community.

If in the future you want to develop your own dashboard, feel free to do it.

To import a Grafana dashboard, select the Import option in the left menu, under the Plus icon.

Import a dashboard option in Grafana

On the next screen, import the dashboard with the 8451 ID.

This is a dashboard created by Gabriel Sagnard that displays system metrics collected by Telegraf.

Specifying a dashboard id in Grafana import

From there, Grafana should automatically try to import this dashboard.

Add the previous configured InfluxDB as the dashboard datasource and click on “Import“.

Selecting InfluxDB as a datasource in Grafana

Great!

We now have our first Grafana dashboard displaying Telegraf metrics.

This is what you should now see on your screen.

Grafana dashboard displaying Telegraf metrics

c – Modifying InfluxQL queries in Grafana query explorer

When designing this dashboard, the creator specified the hostname as “nagisa”, which is obviously different from one host to another (mine is for example named “debian-10”)

To modify it, head over to the query explorer by hovering the panel title, and clicking on “Edit”.

Edit panel in Grafana

In the “queries” panel, change the host, and the panel should starting displaying data.

InfluxDB query in Grafana

Go back to the dashboard, and this is what you should see.

CPU panel in Grafana

VI – Conclusion

In this tutorial, you learned how to setup a complete Telegraf, InfluxDB and Grafana stack on your server.

So where should you go from there?

The first thing would be to connect Telegraf to different inputs, look for existing dashboards in Grafana or design your own ones.

Also, we already made a lot of examples using Grafana and InfluxDB, you could maybe find some inspiration reading those tutorials.

Did you know for example that you could monitor SSH hackers in real time with the TIG stack?

Troubleshooting

  • Error writing to output [influxdb] : could not write any address

Possible solution: make sure that InfluxDB is correctly running on the port 8086.

$ sudo lsof -i -P -n | grep influxdb
influxd   17737        influxdb  128u  IPv6 1177009213      0t0  TCP *:8086 (LISTEN)

If you are having a different port, change your Telegraf configuration to forward metrics to the custom port that your InfluxDB server was assigned.


  • [outputs.influxdb] when writing to [http://localhost:8086] : 401 Unauthorized: authorization failed

Possible solution: make sure that the credentials are correctly set in your Telegraf configuration. Make sure also that you created an account for Telegraf on your InfluxDB server.


  • http: server gave HTTP response to HTTPS client

Possible solution: make sure that you enabled the https-authentication parameter in the InfluxDB configuration file. It is set by default to false.


  • x509: cannot validate certificate for 127.0.0.1 because it does not contain any IP SANs

Possible solution: your TLS verification is set, you need to enable the insecure_skip_verify parameter as the server identify cannot be verified for self-signed certificates.

You may also like

18 comments

Links 11/8/2019: DragonFly 5.6.2 and OpenBSD 6.6 Beta | Techrights August 11, 2019 - 5:18 pm

[…] How To Setup Telegraf InfluxDB and Grafana on Linux […]

Reply
How To Check SSL Certificate Expiration with Grafana – devconnected August 24, 2019 - 9:41 am

[…] and Prometheus Complete Setup on Linux How to Setup Grafana and Prometheus on Linux How To Setup Telegraf InfluxDB and Grafana on… Monitoring Linux Processes using Prometheus and Grafana How To Install Grafana on Windows 8/10 […]

Reply
Michele October 18, 2019 - 7:13 am

Hi, I wanted to ask you, in my case I have a Raspberry Pi 4B 4GB (I installed the operating system on an SSD, currently I only put Pi-Hole in it) with Raspbian Buster on board; I wanted to install Telegraf, InfluxDB and Grafana to monitor my LAN network; can I follow the whole procedure of your posts, or do I have to change something? Thanks in advance for your reply.

Reply
schkn October 19, 2019 - 1:06 pm

It is indeed possible, you can follow the instructions in the tutorial.

To monitor your network, it is possible to use ntopng which is a very powerful tool for networking monitoring and it has an InfluxDB driver.

Reply
How To Install Grafana on Windows 8/10 – devconnected October 24, 2019 - 10:09 pm

[…] Configure your first datasource for Grafana : Grafana, InfluxDB and Telegraf setup […]

Reply
MikeB November 11, 2019 - 2:55 pm

Grafana doesn’t support InfluxDB 2.0. Seems like something you should tell your readers, right?

Reply
Kokanee January 7, 2020 - 9:01 pm

Thanks! This was a great HOWTO guide!

Reply
schkn January 11, 2020 - 8:44 am

Thank you!

If you loved the content, make sure to check our other articles on the subject 🙂

Reply
finnlugg May 24, 2020 - 10:17 pm

Probably a stupid question. You say:”In the “queries” panel, change the host”. But what is my host name? Windows 10(computername?)

Reply
Sam May 31, 2020 - 5:22 pm

Great right up. I was fine until it came to key pairs, then this:

@grafana:/etc/ssl/influxdb$ sudo certtool –generate-self-signed –load-privkey server-key.prm –outfile server-cert.pem
Generating a self signed certificate…
error reading –load-privkey: server-key.prm

I rode it out and was then face with this

@grafana:/etc/ssl/influxdb$ ls -a
. .. server-cert.p server-cert.pem server-key.pem
taff@grafana:/etc/ssl/influxdb$ sudo chown influxdb:influxdb server-key.pem server-cert.pem
taff@grafana:/etc/ssl/influxdb$ sudo nano /etc/influxdb/influxdb.conf
taff@grafana:/etc/ssl/influxdb$ sudo systemctl restart influxdb
taff@grafana:/etc/ssl/influxdb$ sudo journalctl -f -u influxdb.service
— Logs begin at Sun 2020-05-31 15:56:49 UTC. —
May 31 16:54:10 grafana influxd[3773]: ts=2020-05-31T16:54:10.698246Z lvl=info msg=”opened HTTP access log” log_id=0N6vKGiG000 service=httpd path=stderr
May 31 16:54:10 grafana influxd[3773]: run: open server: open service: tls: failed to find any PEM data in certificate input
May 31 16:54:10 grafana systemd[1]: influxdb.service: Main process exited, code=exited, status=1/FAILURE
May 31 16:54:10 grafana systemd[1]: influxdb.service: Failed with result ‘exit-code’.
May 31 16:54:11 grafana systemd[1]: influxdb.service: Service hold-off time over, scheduling restart.
May 31 16:54:11 grafana systemd[1]: influxdb.service: Scheduled restart job, restart counter is at 5.
May 31 16:54:11 grafana systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
May 31 16:54:11 grafana systemd[1]: influxdb.service: Start request repeated too quickly.
May 31 16:54:11 grafana systemd[1]: influxdb.service: Failed with result ‘exit-code’.
May 31 16:54:11 grafana systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database.

clearly the link, any help (newbie level) much appreciated.

Reply
CarbonR June 17, 2020 - 6:36 pm

Hi,
Is it possible that in section “b – Create a public key for your InfluxDB server”
$ sudo certtool –generate-self-signed –load-privkey server-key.prm –outfile server-cert.pem
the extension of the ‘server-key’ has to be ‘.pem’ instead of ‘.prm’?

Thanks,

Reply
Shan July 23, 2020 - 10:15 am

Hi @SCHKN

Can you provide the configuration for `certtool` in this demo? There are so many questions when setting up certificates and would be great if you can point out which fields in the subject should be filled out. You seem to have skipped that part completely.

Maybe it might be worth using a `.conf` file along with `certtool` to make sure basic information for generating certificates is maintained.

Reply
Paras Nath August 3, 2020 - 12:04 pm

influx -ssl -unsafeSsl -username ‘admin’ -password ‘password’
Failed to connect to https://localhost:8086: Get https://localhost:8086/ping: dial tcp 127.0.0.1:8086: connect: connection refused
Please check your connection settings and ensure ‘influxd’ is running.

Reply
Adrian September 7, 2020 - 10:13 pm

Correct to: sudo certtool –generate-self-signed –load-privkey server-key.pem –outfile server-cert.pem

Reply
schkn September 16, 2020 - 4:58 pm

Thank you for the correction!

Reply
Stefan November 20, 2020 - 11:43 am

Just a question, is there a guide comming soon for influxdb 2.0? Been trying to configure it for a coulpe of days now but…

Reply
Alan May 14, 2021 - 10:03 am

thanks :thumbsup

Reply
siva February 21, 2023 - 8:20 am

Is there a guide comming soon for influxdb upgradation from 1.8 to 2.6 version

Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.