中间件监控实战 —— MySQL / Redis / Kafka / ElasticSearch 一网打尽

中间件监控实战 —— MySQL / Redis / Kafka / ElasticSearch 一网打尽系列博客第 5 篇。四大中间件的黄金指标、Exporter 部署、关键 PromQL、告警规则。一、MySQL 监控ecs-0002: 1.94.204.2151.1 黄金指标映射黄金指标MySQL 对应流量QPSQuestions速率错误连接失败、复制错误延迟慢查询数、查询耗时饱和度连接数、线程占用、Buffer Pool1.2 部署 mysqld_exporter# 创建只读监控账号mysql-eCREATE USER exporter% IDENTIFIED BY Export3r#2024;mysql-eGRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO exporter%;# 启动 exportermysqld_exporter --config.my-cnf/etc/.my.cnf# 监听 :91041.3 关键查询# QPS rate(mysql_global_status_queries[5m]) # 慢查询速率 rate(mysql_global_status_slow_queries[5m]) # 连接数使用率 mysql_global_status_threads_connected / mysql_global_variables_max_connections * 100 # 缓冲池命中率越高越好99% 为佳 1 - (rate(mysql_global_status_innodb_buffer_pool_reads[5m]) / rate(mysql_global_status_innodb_buffer_pool_read_requests[5m]))1.4 告警规则-alert:MySQLSlowQueriesexpr:rate(mysql_global_status_slow_queries[5m])0.5for:5mlabels:{severity:warning,category:mysql}-alert:MySQLConnectionsHighexpr:mysql_global_status_threads_connected / mysql_global_variables_max_connections * 10080for:5mlabels:{severity:warning,category:mysql}二、Redis 监控ecs-00022.1 关键指标关注点指标内存redis_memory_used_bytes/redis_memory_max_bytes命中率redis_keyspace_hits/ (hitsmisses)延迟redis_up、命令处理耗时连接redis_connected_clients2.2 部署 redis_exporterredis_exporter--redis.addrlocalhost:6379# 监听 :91212.3 关键查询# 内存使用率 redis_memory_used_bytes / redis_memory_max_bytes * 100 # 命中率95% 健康 redis_keyspace_hits_total / (redis_keyspace_hits_total redis_keyspace_misses_total) * 100三、Kafka 监控ecs-0003: 124.71.228.1773.1 KRaft 模式部署新一代 Kafka 3.7 用KRaft替代 ZooKeeper更轻量process.rolesbroker,controller node.id1 controller.quorum.voters1localhost:9093 listenersPLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 advertised.listenersPLAINTEXT://124.71.228.177:90923.2 JMX Exporter 采集Kafka 通过 JMX 暴露指标用 Java Agent 转成 Prometheus 格式# 启动参数注入 agentKAFKA_OPTS-javaagent:/opt/kafka/jmx_prometheus_javaagent.jar7071:/opt/kafka/jmx_exporter.ymlkafka-server-start.sh server.properties3.3 关键指标# 消费组积压最重要 kafka_consumergroup_lag # 入站消息速率 rate(kafka_server_brokertopicmetrics_messagesin_total[5m]) # 分区数 kafka_cluster_partition_count⚠️消费滞后 (Lag) 是 Kafka 最该盯的指标——它直接反映下游处理是否跟得上。四、ElasticSearch 监控ecs-00034.1 关键指标关注点指标集群健康elasticsearch_cluster_health_status绿/黄/红JVM 堆elasticsearch_jvm_memory_used_bytes索引速率elasticsearch_indices_indexing_rate查询延迟elasticsearch_indices_search_fetch_latency4.2 部署 es_exporterelasticsearch_exporter--es.urihttp://localhost:9200--es.all# 监听 :91144.3 关键查询# 集群状态1绿 2黄 3红 elasticsearch_cluster_health_status # JVM 堆使用率 elasticsearch_jvm_memory_used_bytes{areaheap} / elasticsearch_jvm_memory_max_bytes{areaheap} * 100五、四大中间件部署命令汇总# ecs-0002bashscripts/install_middleware.sh mysqlbashscripts/install_middleware.sh redis# ecs-0003bashscripts/install_middleware.sh kafkabashscripts/install_middleware.sh es实战结果4 台机器上的 6 个 Exporter 4 个中间件全部 UP ✅目标地址状态MySQL1.94.204.215:9104UPRedis1.94.204.215:9121UPKafka JMX124.71.228.177:7071UPES124.71.228.177:9114UP六、下一篇预告第 6 篇应用埋点与告警闭环 —— 用 Flask Demo 演示 RED 方法埋点把业务异常感知做到极致。本文为《运维监控实战》系列第 5 篇。完整配置见 Gitee 仓库 deploy/。