Overview

Because I am familiar with the Prometheus model, some server software and small applications are usually monitored through Prometheus. So recently I had a new environment that needed to be rebuilt, so I took the opportunity to document the process of installing and configuring prometheus and grafana.

The build involves installing and deploying prometheus and grafana using Docker, mapping the local disk directory (prometheus data can be quite large, so I am using an additional mounted disk), then also including a reverse proxy for Nginx, and a simple configuration of prometheus. service discovery service, provided by replacing the configuration file, because my crawl target is an internal container platform and the target’s ip changes frequently, so I need a simple dynamic change management.

If you just want to learn how to install and deploy Grafana, you can refer to a previous guide I wrote: Installing and deploying Grafana, which covers deploying a systemd-managed of Grafana, as well as steps for using the Nginx proxy.

compose run prometheus & grafana

Install Docker and compose

  1. [[email protected].io]# swapoff -a
  2. [[email protected].io]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  3. [[email protected].io]# yum install -y docker-ce
  4. [[email protected].io]# systemctl start docker
  5. [[email protected].io]# systemctl enable docker
  6. [[email protected].io]# sudo curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/sbin/docker-compose
  7. [[email protected].io]# sudo chmod +x /usr/local/sbin/docker-compose

Environment prepare

  1. [[email protected].io]# mkdir -p /data/monitoring/{software,configs,grafana,prometheus}
  2. [[email protected].io]# mkdir -p /data/monitoring/configs/{grafana,prometheus}
  3. [[email protected].io]# cd /data/monitoring/software && tar -xvf prometheus-2.42.0.linux-amd64.tar.gz && ln -s prometheus-2.42.0.linux-amd64 prometheus
  4. [[email protected].io]# cp /data/monitoring/software/prometheus/prometheus.yml /data/monitoring/configs/prometheus/prometheus.yml
  5. [[email protected].io]# cat > /data/monitoring/configs/grafana/grafana.ini <<EOF
  6. [server]
  7. protocol = http
  8. http_addr = 0.0.0.0
  9. http_port = 3000
  10. domain = grafana.liqiang.io
  11. enforce_domain = true
  12. serve_from_sub_path = true
  13. root_url = http://%(domain)s:%(http_port)s/grafana
  14. EOF

Run prometheus & grafana

  1. [[email protected].io]# cat <<EOF >/data/monitoring/software/docker-compose.yaml
  2. version: "3.7"
  3. services:
  4. prometheus:
  5. image: prom/prometheus:latest
  6. user: "${UID}:${GID}"
  7. volumes:
  8. - /data/monitoring/configs/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
  9. - /data/monitoring/prometheus:/prometheus
  10. command:
  11. - '--web.enable-lifecycle'
  12. - '--config.file=/etc/prometheus/prometheus.yml'
  13. - '--storage.tsdb.path=/prometheus'
  14. - '--web.console.libraries=/usr/share/prometheus/console_libraries'
  15. - '--web.console.templates=/usr/share/prometheus/consoles'
  16. ports:
  17. - 9090:9090
  18. grafana:
  19. image: grafana/grafana:latest
  20. user: "${UID}:${GID}"
  21. volumes:
  22. - /data/monitoring/configs/grafana/grafana.ini:/etc/grafana/grafana.ini
  23. - /data/monitoring/grafana:/var/lib/grafana
  24. ports:
  25. - 3000:3000
  26. links:
  27. - prometheus
  28. EOF
  29. [[email protected].io]# cd /data/monitoring/software/ && docker-compose up -d

At this point, both Prometheus and Grafana have been installed and deployed, and you can access Grafana via http://localhost:3000. However, I used to use Nginx as a proxy so that I didn’t have to specify an additional port.

Configuring Nginx

Since I already have Nginx installed, I don’t need to install it again. If you don’t have it installed, the easy option is to run another Nginx with docker-compose, or install one directly with Linux’s package management software, for example, I installed it directly with Yum by

  1. [[email protected].io]# yum install -y nginx
  2. [[email protected].io]# mkdir -p /etc/nginx/grafana
  3. [[email protected].io]# cat> /etc/nginx/grafana/grafana.conf << EOF
  4. location /grafana {
  5. proxy_pass http://127.0.0.1:3000;
  6. proxy_set_header X-Real-IP $remote_addr;
  7. proxy_set_header Host $host;
  8. proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  9. proxy_http_version 1.1;
  10. proxy_set_header Upgrade $http_upgrade;
  11. proxy_set_header Connection "upgrade";
  12. proxy_redirect off;
  13. }
  14. EOF
  15. [[email protected].io]# echo "Please add 'include /etc/nginx/grafana/*.conf;' to your nginx.conf"
  16. [[email protected].io]# nginx -t
  17. [[email protected].io]# nginx -s reload

Configurare Grafana

Open the browser and use this URL:http://grafana.liqiang.io/grafana,I think you can see this page:

Figure 1:Grafana Login Page

The default username and password for grafana is:

You will then be asked to change your password, it is best to do so.

Service discovery procedures

Prometheus supports a very wide range of service discovery methods, for example:

However, these service discovery features are that the URL Path of the Target you need to collect is fixed and can only be a list of service discovery Endpoints, meaning that you can only use the number of Containers for a specific service discovery and cannot dynamically increase or decrease the type of service.

For example, if you have two services, Order and Account, then Prometheus’ service discovery supports you to dynamically change the number of containers for these two services, for example from 100 to 150. However, if you add a new service, Item, then this service discovery provided by Prometheus is not supported.

So what do we do to dynamically scale the service, here I have chosen to write a program that then dynamically does the service metadata work and then, depending on the actual situation (e.g. the deployment platform of the service), extend this program to do the dynamic discovery of the Endpoint:.

Service Metadata Discovery

There are two key points in the implementation of service metadata discovery.

Since the code is quite long, I won’t post it, but if you are interested you can just open the Github Repo: devops/prometheus/service_discovery to see for yourself~, and in practice, add a service to the end of docker-compose.yaml:

  1. [[email protected].io]# cat docker-compose.yaml
  2. ... ...
  3. service-discovery:
  4. image: liqiangliu/prometheus-service-discovery:0.3.0
  5. user: "${UID}:${GID}"
  6. environment:
  7. - PROMETHEUS_ADDR=http://prometheus:9090
  8. volumes:
  9. - /data/monitoring/configs/prometheus:/etc/prometheus
  10. ports:
  11. - 5555:5555

Then try to add a service:

  1. [[email protected].io]# curl --location --request POST 'http://127.0.0.1:5555/services' --header 'Content-Type: application/json' --data-raw '{
  2. "name": "prometheus-default",
  3. "endpoints": [
  4. {
  5. "host": "prometheus",
  6. "port": 9090
  7. }
  8. ]
  9. }'

Summary

This completes our docker compose based prometheus and grafana monitoring system, and supports a simple service discovery feature, which is simple to apply, so you can extend it yourself.