openresty定制prometheus metrics监控nginx(1)

之前总喜欢看服务器日志，看有多少google爬虫访问过我的博客。时间长了就觉得太low了,肉眼搜索google的user agent,还要看ip是不是google的.觉得low就撸点代码搞定

nginx-lua-prometheus

代码很简单，参考nginx-lua-prometheus就行了

http{
    server{
        init_by_lua_file /root/lua/init.lua;
        location / {
            log_by_lua_file /root/lua/stat.lua;
        }
    }
    server{
        #export port for prometheus pulling
        listen 9145;
        location /metrics {
            content_by_lua '
              prometheus:collect()
            ';
        }
    }
}

init.lua

prometheus = require("resty.prometheus").init("prometheus_metrics")
requests = prometheus:counter("nginx_http_requests_total", "Number of HTTP requests", {"host", "status"})
google_requests=prometheus:counter("nginx_http_google_requests_total", "Number of HTTP requests from google", {"status"})

stat.lua

local forwarded_for=ngx.var.http_x_forwarded_for
local cidr = require("resty.libcidr-ffi")
local is_from_google=cidr.contains(cidr.from_str('66.249.0.0/16'), cidr.from_str(forwarded_for))
ngx.log(ngx.ERR,'is_from_google: ',is_from_google,' ',forwarded_for)

local user_agent=ngx.var.http_user_agent
local is_google_bot=string.match(user_agent,'Googlebot/2.1;')
ngx.log(ngx.ERR,'is_google_bot: ',is_google_bot,' ',user_agent)

if is_from_google and is_google_bot then
  google_requests:inc(1, {ngx.var.status})
end

requests:inc(1, {ngx.var.server_name, ngx.var.status})

说明

• libcidr-ffi安装可以参考openresty设置访问白名单
• 66.249.0.0/16范围有点大，google不会公布爬虫ip,只能自己观察
• exporter没有用Nginx VTS exporter,因为想要可定制性更强的exporter

效果

监控提醒

监控好了，可是还是要主动看监控，才能知道有没有google爬虫访问.如果能'提醒'我就更好了,可以定义prometheus的alerting rules,提醒的内容是google爬虫访问的路径(path)

alerting rules

rule.yml

groups:
  - name: google request
    rules:
    - alert: google request
      expr: sum(increase(nginx_http_google_requests_total[2m])) by (path) >0
      for: 5s
      annotations:
        description: "path:{{$labels.path}}"

• {{$labels.path}}可以获取对应的label中的path
• {{$value}}可以获取提醒的时间点对应的值

当然前面定义的counter也要改一下

init.lua添加label-path

google_requests=prometheus:counter("nginx_http_google_requests_total", "Number of HTTP requests from google", {"path"})

stat.lua对应的写入path

if is_from_google and is_google_bot then
  local path=ngx.re.gsub(ngx.var.request_uri, "[.|/]html", "")
  google_requests:inc(1, {path})
end

Alertmanager

global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/your_path'

route:
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
  slack_configs:
    - title: 'google request'
      text: "{{ .CommonAnnotations.description }}"

{{ .CommonAnnotations.description }}获取前面rule.yml定义的annotations description

最后配置prometheus.yml

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 127.0.0.1:9093

rule_files:
  - "rule.yml"

效果

然而这里有个问题，导致无法触发告警，参见下一篇

---

不定期更新