Prometheus监控系统搭建与实践

前言

在微服务时代，监控系统是保障服务稳定性的基石。Prometheus作为CNCF毕业项目，已经成为云原生监控的事实标准。本文将带你从零搭建Prometheus监控系统，掌握企业级监控方案的核心要点。

一、Prometheus架构概览

1.1 核心组件

Prometheus Server：

  • 数据采集和存储
  • 提供PromQL查询语言
  • 内置时序数据库

Exporters：

  • Node Exporter：采集主机指标
  • MySQL Exporter：采集数据库指标
  • Redis Exporter：采集缓存指标

Alertmanager：

  • 告警路由和分组
  • 告警抑制和静默
  • 支持多种通知渠道

Grafana：

  • 数据可视化
  • 丰富的面板类型
  • 告警展示

1.2 工作流程

1
2
3
4
5
6
7
8
9
10
11
目标服务(Metrics) 
    ↓
Exporters暴露指标
    ↓
Prometheus拉取(Pull)
    ↓
存储到时序数据库
    ↓
PromQL查询 → Grafana展示
    ↓
触发告警规则 → Alertmanager → 通知

二、Prometheus安装部署

2.1 Docker方式部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# 创建配置目录
mkdir -p /opt/prometheus/{config,data}

# 创建配置文件
cat > /opt/prometheus/config/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s      # 采集间隔
  evaluation_interval: 15s  # 规则评估间隔
  
# 告警管理器配置
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

# 告警规则文件
rule_files:
  - '/etc/prometheus/rules/*.yml'

# 采集配置
scrape_configs:
  # Prometheus自监控
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  
  # Node Exporter
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          instance: 'server-01'
EOF

# 启动Prometheus
docker run -d \
  --name prometheus \
  -p 9090:9090 \
  -v /opt/prometheus/config:/etc/prometheus \
  -v /opt/prometheus/data:/prometheus \
  prom/prometheus:latest

2.2 二进制方式部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# 下载Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz

# 解压
tar xvf prometheus-2.45.0.linux-amd64.tar.gz
cd prometheus-2.45.0.linux-amd64

# 启动
./prometheus --config.file=prometheus.yml \
  --storage.tsdb.path=/data/prometheus \
  --web.listen-address=:9090

# 配置系统服务
cat > /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/prometheus/prometheus \
  --config.file=/usr/local/prometheus/prometheus.yml \
  --storage.tsdb.path=/data/prometheus
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

# 启动服务
systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus

三、监控指标采集

3.1 Node Exporter（主机监控）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Docker方式部署
docker run -d \
  --name node-exporter \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  prom/node-exporter:latest \
  --path.rootfs=/host

# 二进制方式
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.0/node_exporter-1.6.0.linux-amd64.tar.gz
tar xvf node_exporter-1.6.0.linux-amd64.tar.gz
cd node_exporter-1.6.0.linux-amd64
./node_exporter

# 访问指标
curl http://localhost:9100/metrics

主要指标：

  • CPU使用率：node_cpu_seconds_total
  • 内存使用：node_memory_MemAvailable_bytes
  • 磁盘IO：node_disk_io_time_seconds_total
  • 网络流量：node_network_receive_bytes_total

3.2 MySQL Exporter

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 创建监控用户
CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'password';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
FLUSH PRIVILEGES;

# 部署MySQL Exporter
docker run -d \
  --name mysql-exporter \
  -p 9104:9104 \
  -e DATA_SOURCE_NAME="exporter:password@(mysql:3306)/" \
  prom/mysqld-exporter:latest

# Prometheus配置
scrape_configs:
  - job_name: 'mysql'
    static_configs:
      - targets: ['mysql-exporter:9104']

3.3 Redis Exporter

1
2
3
4
5
6
docker run -d \
  --name redis-exporter \
  -p 9121:9121 \
  oliver006/redis_exporter:latest \
  --redis.addr=redis://redis:6379 \
  --redis.password=your_password

3.4 应用程序监控

Spring Boot集成：

1
2
3
4
5
<!-- pom.xml -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
1
2
3
4
5
6
7
8
9
# application.yml
management:
  endpoints:
    web:
      exposure:
        include: prometheus
  metrics:
    tags:
      application: ${spring.application.name}

Node.js集成：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
const client = require('prom-client');
const express = require('express');

// 创建Registry
const register = new client.Registry();

// 默认指标
client.collectDefaultMetrics({ register });

// 自定义指标
const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  registers: [register]
});

// 暴露metrics端点
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

四、PromQL查询语言

4.1 基础查询

1
2
3
4
5
6
7
8
9
10
11
# 即时查询
http_requests_total

# 范围查询（过去5分钟）
http_requests_total[5m]

# 标签过滤
http_requests_total{method="GET", status="200"}

# 正则匹配
http_requests_total{path=~"/api/.*"}

4.2 聚合操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 求和
sum(http_requests_total)

# 按标签分组
sum by (method) (http_requests_total)

# 平均值
avg(node_cpu_seconds_total)

# 最大值
max(node_memory_MemTotal_bytes)

# 计数
count(up == 1)

4.3 速率计算

1
2
3
4
5
6
7
8
# 每秒请求数（QPS）
rate(http_requests_total[5m])

# 每秒增量
irate(http_requests_total[5m])

# 增长量
increase(http_requests_total[1h])

4.4 实用查询示例

CPU使用率：

1
100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

内存使用率：

1
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

磁盘使用率：

1
(1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100

API响应时间P99：

1
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

五、告警规则配置

5.1 告警规则文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# /etc/prometheus/rules/alerts.yml
groups:
  - name: node_alerts
    interval: 30s
    rules:
      # CPU使用率告警
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 80% (current: {{ $value }}%)"
      
      # 内存使用率告警
      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is above 85% (current: {{ $value }}%)"
      
      # 磁盘使用率告警
      - alert: HighDiskUsage
        expr: (1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100 > 90
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "High disk usage on {{ $labels.instance }}"
          description: "Disk usage is above 90% (current: {{ $value }}%)"
      
      # 服务宕机告警
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.job }} is down"
          description: "{{ $labels.instance }} has been down for more than 1 minute"

5.2 应用告警规则

API告警：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
- name: api_alerts
  rules:
    # 高错误率
    - alert: HighErrorRate
      expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "High error rate on {{ $labels.instance }}"
        description: "Error rate is {{ $value | humanizePercentage }}"
    
    # 慢查询
    - alert: SlowAPI
      expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Slow API response on {{ $labels.instance }}"
        description: "P99 latency is {{ $value }}s"

六、Alertmanager配置

6.1 安装Alertmanager

1
2
3
4
5
docker run -d \
  --name alertmanager \
  -p 9093:9093 \
  -v /opt/alertmanager:/etc/alertmanager \
  prom/alertmanager:latest

6.2 配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
# /opt/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
  
# 路由配置
route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'default'
  routes:
    # 严重告警立即发送
    - match:
        severity: critical
      receiver: 'critical'
      continue: true
    
    # 警告级别告警
    - match:
        severity: warning
      receiver: 'warning'

# 接收器配置
receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://webhook-server:8080/alerts'
  
  - name: 'critical'
    email_configs:
      - to: 'ops-team@company.com'
        from: 'alert@company.com'
        smarthost: 'smtp.company.com:587'
        auth_username: 'alert@company.com'
        auth_password: 'password'
        headers:
          Subject: '【严重】{{ .GroupLabels.alertname }}'
    
    webhook_configs:
      - url: 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=YOUR_KEY'
        send_resolved: true
  
  - name: 'warning'
    webhook_configs:
      - url: 'http://webhook-server:8080/warnings'

# 抑制规则
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

6.3 企业微信告警

1
2
3
4
5
6
7
8
9
# 企业微信机器人webhook
curl 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=YOUR_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "msgtype": "markdown",
    "markdown": {
      "content": "## 告警通知\n> **告警名称**: CPU使用率过高\n> **告警级别**: <font color=\"warning\">warning</font>\n> **告警主机**: server-01\n> **当前值**: 85%\n> **告警时间**: 2024-07-15 16:45:00"
    }
  }'

七、Grafana可视化

7.1 安装Grafana

1
2
3
4
5
6
7
docker run -d \
  --name grafana \
  -p 3000:3000 \
  -v grafana-storage:/var/lib/grafana \
  grafana/grafana:latest

# 默认账号：admin/admin

7.2 添加数据源

1
2
Configuration → Data Sources → Add data source → Prometheus
URL: http://prometheus:9090

7.3 导入Dashboard

推荐Dashboard：

  • Node Exporter Full：ID 1860
  • MySQL Overview：ID 7362
  • Redis Dashboard：ID 11835
  • Spring Boot：ID 12900
1
Dashboards → Import → 输入ID → Load

7.4 自定义Panel

CPU使用率面板：

1
2
3
4
5
6
7
{
  "title": "CPU Usage",
  "targets": [{
    "expr": "100 - (avg by (instance) (irate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)"
  }],
  "type": "graph"
}

八、最佳实践

8.1 监控指标设计

四大黄金指标：

  1. 延迟（Latency）：请求响应时间
  2. 流量（Traffic）：每秒请求数
  3. 错误（Errors）：错误率
  4. 饱和度（Saturation）：资源使用率

8.2 告警策略

告警分级：

  • P0（严重）：服务完全不可用
  • P1（重要）：核心功能受影响
  • P2（一般）：性能下降
  • P3（提示）：可能的潜在问题

避免告警疲劳：

  • 合理设置阈值
  • 使用for参数避免抖动
  • 配置告警抑制规则
  • 定期Review告警规则

8.3 数据保留策略

1
2
3
4
5
# Prometheus配置
storage:
  tsdb:
    retention.time: 15d  # 保留15天
    retention.size: 50GB  # 或50GB

8.4 高可用部署

Prometheus联邦：

1
2
3
4
5
6
7
8
9
10
11
scrape_configs:
  - job_name: 'federate'
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="node"}'
    static_configs:
      - targets:
          - 'prometheus-1:9090'
          - 'prometheus-2:9090'

九、故障排查

9.1 常见问题

指标采集失败：

1
2
3
4
5
6
7
8
# 检查target状态
http://prometheus:9090/targets

# 检查exporter是否正常
curl http://node-exporter:9100/metrics

# 检查网络连通性
telnet node-exporter 9100

告警不触发：

1
2
3
4
5
6
7
8
9
# 检查告警规则
http://prometheus:9090/rules

# 检查Alertmanager状态
http://alertmanager:9093/#/status

# 查看日志
docker logs prometheus
docker logs alertmanager

9.2 性能优化

1
2
3
4
5
6
7
8
9
10
11
# 减少采集频率
global:
  scrape_interval: 30s  # 从15s改为30s

# 减少保留时间
storage:
  tsdb:
    retention.time: 7d  # 从15d改为7d

# 启用压缩
--storage.tsdb.wal-compression

十、总结

通过本文的学习，我们完整搭建了Prometheus监控系统，实现了：

  1. 多维度监控：主机、中间件、应用全覆盖
  2. 灵活告警：多级别、多渠道告警通知
  3. 可视化展示：Grafana丰富的图表
  4. 高可用架构：联邦部署，保障监控系统自身稳定性

下一步：

  • 探索ServiceMonitor（Kubernetes环境）
  • 集成分布式追踪（Jaeger/Zipkin）
  • 日志聚合（ELK/Loki）
  • AIOps智能告警

参考资料