0. 概述

最近有个统计 Nginx 的需求,按照以前 ETL 的操作步骤来说,可能处理方式是写一个 Python 项目,然后解析一下 Nginx 的访问日志,然后暴露统计数据接口就可以了。但是,本着不作不舒服的原则,我想让 nginx 自己暴露这个功能,于是就有了这篇文章。

要实现这个目标,第一步还是要先能编译 Nginx,然后是在编译的时候加入 Lua 支持,最后,就是插入 prometheus 监控的 lua 模块了,具体的操作流程如下。

1. 前提安装

Ubuntu

  1. [root@liqiang.io]# apt install -y wget make build-essential libssl-dev

CentOS

  1. [root@liqiang.io]#

2. 安装 LuaJIT

  1. [root@liqiang.io]# wget http://luajit.org/download/LuaJIT-2.0.5.tar.gz
  2. [root@liqiang.io]# tar zxf LuaJIT-2.0.5.tar.gz
  3. [root@liqiang.io]# cd LuaJIT-2.0.5
  4. [root@liqiang.io]# make install PREFIX=/usr/local/LuaJIT
  5. [root@liqiang.io]# export LUAJIT_LIB=/usr/local/LuaJIT/lib
  6. [root@liqiang.io]# export LUAJIT_INC=/usr/local/LuaJIT/include/luajit-2.0
  7. [root@liqiang.io]# export LD_LIBRARY_PATH=/usr/local/LuaJIT/lib/:$LD_LIBRARY_PATH

3. 安装各种扩展支持

偷懒做法

  1. [root@liqiang.io]# apt-get install libpcre3-dev \
  2. libssl-dev perl make build-essential curl

麻烦做法

  1. [root@liqiang.io]# wget https://github.com/openresty/echo-nginx-module/archive/v0.61.tar.gz
  2. [root@liqiang.io]# tar zxf v0.61.tar.gz
  3. [root@liqiang.io]# wget https://onboardcloud.dl.sourceforge.net/project/pcre/pcre/8.43/pcre-8.43.tar.gz
  4. [root@liqiang.io]# tar zxf pcre-8.43.tar.gz
  5. [root@liqiang.io]# cd pcre-8.43
  6. [root@liqiang.io]# ./configure && make && make install
  7. [root@liqiang.io]# cd ..
  8. [root@liqiang.io]# wget https://github.com/simplresty/ngx_devel_kit/archive/v0.3.1.tar.gz
  9. [root@liqiang.io]# tar zxf v0.3.1.tar.gz
  10. [root@liqiang.io]# wget https://github.com/openresty/lua-nginx-module/archive/v0.10.15.tar.gz
  11. [root@liqiang.io]# tar zxf v0.10.15.tar.gz
  12. [root@liqiang.io]# wget http://zlib.net/zlib-1.3.1.tar.gz
  13. [root@liqiang.io]# tar zxf zlib-1.3.1.tar.gz
  14. [root@liqiang.io]# cd zlib-1.3.1
  15. [root@liqiang.io]# ./configure && make && make install
  16. [root@liqiang.io]# cd ..

4. 编译 Nginx

  1. [root@liqiang.io]# wget http://nginx.org/download/nginx-1.21.2.tar.gz
  2. [root@liqiang.io]# tar zxf nginx-1.21.2.tar.gz
  3. [root@liqiang.io]# cd nginx-1.21.2
  4. [root@liqiang.io]# ./configure \
  5. --prefix=/usr/share/nginx \
  6. --sbin-path=/usr/sbin/nginx \
  7. --modules-path=/usr/lib64/nginx/modules \
  8. --conf-path=/etc/nginx/nginx.conf \
  9. --error-log-path=/var/log/nginx/error.log \
  10. --http-log-path=/var/log/nginx/access.log \
  11. --pid-path=/run/nginx.pid \
  12. --with-file-aio --with-http_ssl_module \
  13. --user=nginx --group=nginx \
  14. --with-http_stub_status_module \
  15. --with-http_gzip_static_module \
  16. --with-http_realip_module \
  17. --with-pcre=/root/nginx/pcre-8.43 \
  18. --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module \
  19. --with-http_auth_request_module \
  20. --add-module=/root/nginx/echo-nginx-module-0.61 \
  21. --add-module=/root/nginx/ngx_devel_kit-0.3.1 \
  22. --add-module=/root/nginx/lua-nginx-module-0.10.15
  23. ... ...
  24. Configuration summary
  25. + using PCRE library: /root/nginx/pcre-8.43
  26. + using system OpenSSL library
  27. + using system zlib library
  28. nginx path prefix: "/usr/local/nginx"
  29. nginx binary file: "/usr/local/nginx/sbin/nginx"
  30. nginx modules path: "/usr/local/nginx/modules"
  31. nginx configuration prefix: "/usr/local/nginx/conf"
  32. nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  33. nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  34. nginx error log file: "/usr/local/nginx/logs/error.log"
  35. nginx http access log file: "/usr/local/nginx/logs/access.log"
  36. nginx http client request body temporary files: "client_body_temp"
  37. nginx http proxy temporary files: "proxy_temp"
  38. nginx http fastcgi temporary files: "fastcgi_temp"
  39. nginx http uwsgi temporary files: "uwsgi_temp"
  40. nginx http scgi temporary files: "scgi_temp"
  41. [root@liqiang.io]# make -j 4

至此,Nginx 就算是编译完毕了。这里省略了 N 多内容,请注意上面的各个细节,错过一点可能都会有问题。这里的编译参数可以根据个人的进行调整,如果你不知道需要什么选项,有一些方法可以帮助你选择,一个是:

  1. [root@liqiang.io]# ./configure --help

不过这种方法选择太多,而且专业性也比较强,会让你迷失,我更推荐的一种方式是找到官方编译的,你正在使用中的版本,然后安装它,再通过这条命令来查看:

  1. [root@liqiang.io]# /usr/sbin/nginx -V
  2. nginx version: nginx/1.16.1
  3. built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)
  4. built with OpenSSL 1.0.2k-fips 26 Jan 2017
  5. TLS SNI support enabled
  6. configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-stream_ssl_preread_module --with-http_addition_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-http_perl_module=dynamic --with-http_auth_request_module --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-google_perftools_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'

直接复制这里的 Configure Arguments 就好了,因为你已经是适配了这个编译选项了,所以最终编译出来的 Nginx 用得也会更加顺手一些。

5. 添加 prometheus 监控

接着就是添加 Prometheus 的 Lua 支持了。这里大概有三个要点需要注意,分别是:

4.1 添加 Nginx 设置

在 Nginx 的 http 配置段中加入这么一段:

  1. [root@liqiang.io]# cat nginx.conf
  2. ...
  3. http {
  4. ...
  5. lua_shared_dict prometheus_metrics 10M;
  6. lua_package_path "/usr/local/nginx/lua/prometheus.lua";
  7. init_by_lua '
  8. prometheus = require("prometheus").init("prometheus_metrics")
  9. metric_requests = prometheus:counter(
  10. "nginx_http_requests_total", "Number of HTTP requests", {"host", "status"})
  11. metric_latency = prometheus:histogram(
  12. "nginx_http_request_duration_seconds", "HTTP request latency", {"host"})
  13. metric_connections = prometheus:gauge(
  14. "nginx_http_connections", "Number of HTTP connections", {"state"})
  15. ';
  16. log_by_lua '
  17. metric_requests:inc(1, {ngx.var.server_name, ngx.var.status})
  18. metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.server_name})
  19. ';
  20. ...
  21. }

4.2 添加 Lua 模块

  1. [root@liqiang.io]# wget https://github.com/knyar/nginx-lua-prometheus/archive/0.20181120.tar.gz
  2. [root@liqiang.io]# tar zxf 0.20181120.tar.gz
  3. [root@liqiang.io]# cd nginx-lua-prometheus-0.20181120
  4. [root@liqiang.io]# mkdir -p /usr/local/nginx/lua/
  5. [root@liqiang.io]# cp prometheus.lua /usr/local/nginx/lua/

4.3 添加 Metric 路由

  1. [root@liqiang.io]# cat prometheus.conf
  2. server {
  3. listen 9145;
  4. allow 192.168.0.0/16;
  5. deny all;
  6. location /metrics {
  7. content_by_lua '
  8. metric_connections:set(ngx.var.connections_reading, {"reading"})
  9. metric_connections:set(ngx.var.connections_waiting, {"waiting"})
  10. metric_connections:set(ngx.var.connections_writing, {"writing"})
  11. prometheus:collect()
  12. ';
  13. }
  14. }

5. 验证效果

最后,就是验收一下效果啦:

  1. [root@liqiang.io]# curl http://127.0.0.1:9145/metrics
  2. # HELP nginx_http_connections Number of HTTP connections
  3. # TYPE nginx_http_connections gauge
  4. nginx_http_connections{state="reading"} 0
  5. nginx_http_connections{state="waiting"} 0
  6. nginx_http_connections{state="writing"} 1
  7. # HELP nginx_http_request_duration_seconds HTTP request latency
  8. # TYPE nginx_http_request_duration_seconds histogram
  9. nginx_http_request_duration_seconds_bucket{host="",le="00.005"} 2
  10. nginx_http_request_duration_seconds_bucket{host="",le="00.010"} 2
  11. nginx_http_request_duration_seconds_bucket{host="",le="00.020"} 2
  12. nginx_http_request_duration_seconds_bucket{host="",le="00.030"} 2
  13. nginx_http_request_duration_seconds_bucket{host="",le="00.050"} 2
  14. nginx_http_request_duration_seconds_bucket{host="",le="00.075"} 2
  15. nginx_http_request_duration_seconds_bucket{host="",le="00.100"} 2
  16. nginx_http_request_duration_seconds_bucket{host="",le="00.200"} 2

6. Ref