首页私人日誌Zabbix对Kafka topic积压数据监控的问题(bug优化)

Zabbix对Kafka topic积压数据监控的问题(bug优化)

admin 10-19 16:52 327次浏览

一 自动分区


1.1 优化前计算方式


寻找配置文件


vim consumer-groups.conf


写入配置文件


test-group|test


执行脚本


bash consumer-groups.sh discovery
{
 data : [
{  {#GROUP} : test-group ,  {#TOPICP} : test ,  {#PARTITION} : 0  },
{  {#GROUP} : test-group ,  {#TOPICP} : test ,  {#PARTITION} : 1  },
{  {#GROUP} : test-group ,  {#TOPICP} : test ,  {#PARTITION} : 3  },
{  {#GROUP} : test-group ,  {#TOPICP} : test ,  {#PARTITION} : 2  }
]
}


由上可知,我们只有test-group|test这一个自动发现配置文件是没有问题的。然后接入test-group|test1


1.2 未优化前计算方式


寻找配置文件


vim consumer-groups.conf


写入配置文件


test-group|test
test-group|test1


执行脚本


bash consumer-groups.sh discovery
{
 data : [
{  {#GROUP} : test-group ,  {#TOPICP} : test ,  {#PARTITION} : 0  },
{  {#GROUP} : test-group ,  {#TOPICP} : test ,  {#PARTITION} : 1  },
{  {#GROUP} : test-group ,  {#TOPICP} : test ,  {#PARTITION} : 3  },
{  {#GROUP} : test-group ,  {#TOPICP} : test ,  {#PARTITION} : 2  }
{  {#GROUP} : test-group ,  {#TOPICP} : test1 ,  {#PARTITION} : 0  },
{  {#GROUP} : test-group ,  {#TOPICP} : test2 ,  {#PARTITION} : 1  },
{  {#GROUP} : test-group ,  {#TOPICP} : test3 ,  {#PARTITION} : 2  }
]
}


执行完我们发现,上面这种种格式是不对的,会导致我们的监控项会出现问题


1.3 优化计算方式


寻找配置文件


vim consumer-groups.conf


写入配置文件


test-group|test
test-group|test1


执行脚本


bash consumer-groups.sh discovery
{
 data : [
{  {#GROUP} : test-group ,  {#TOPICP} : test ,  {#PARTITION} : 0  },
{  {#GROUP} : test-group ,  {#TOPICP} : test ,  {#PARTITION} : 1  },
{  {#GROUP} : test-group ,  {#TOPICP} : test ,  {#PARTITION} : 3  },
{  {#GROUP} : test-group ,  {#TOPICP} : test ,  {#PARTITION} : 2  },
{  {#GROUP} : test-group ,  {#TOPICP} : test1 ,  {#PARTITION} : 0  },
{  {#GROUP} : test-group ,  {#TOPICP} : test1 ,  {#PARTITION} : 1  },
{  {#GROUP} : test-group ,  {#TOPICP} : test1 ,  {#PARTITION} : 2  }
]
}


1.4 lag分区


优化后计算方式


# test-group test分区0 lag
bash consumer-groups.sh lag test-group test 0


# test-group test分区1 lag
bash consumer-groups.sh lag test-group test 1


# test-group test1分区0 lag
bash consumer-groups.sh lag test-group test1 0


我们与未优化的计算方式对比下


优化前计算方式


# 获取分区0 lag
bash consumer-groups.sh lag 0


# 获取分区1 lag
bash consumer-groups.sh lag 1


# 获取分区2 lag
bash consumer-groups.sh lag 2


# 获取分区3 lag
bash consumer-groups.sh lag 3


最终优化后脚本


vim consumer-groups.conf
test-group|test
test-group|test1
vim consumer-groups.sh
cal_topic() {
if [ $# -ne 2 ]; then
echo  parameter num error, 读取topic信息失败 
exit 1
else
/usr/local/kafka/bin/./kafka-consumer-groups.sh --bootstrap-server 192.168.3.55:9092 --describe --group $1 |grep -w $2|grep -v none
fi
}
topic_discovery() {
printf  {\n 
printf  \t\ data\ : [\n 
m=0
num=`cat /etc/zabbix/monitor_scripts/consumer-groups.conf|wc -l`
for line in `cat /etc/zabbix/monitor_scripts/consumer-groups.conf`
do
m=`expr $m + 1`
group=`echo ${line} | awk -F  39;|  39;   39;{print $1}  39;`
topic=`echo ${line} | awk -F  39;|  39;   39;{print $2}  39;`
cal_topic $group $topic   /tmp/consumer-group-tmp
count=`cat /tmp/consumer-group-tmp|wc -l`
n=0
while read line
do
n=`expr $n + 1`
if [ $n -eq $count ]      [ $m -eq $num ]; then
topicp=`echo $line | awk   39;{print $1}  39;`
partition=`echo $line | awk   39;{print $2}  39;`
printf  \t\t{ \ {#GROUP}\ :\ ${group}\ , \ {#TOPICP}\ :\ ${topicp}\ , \ {#PARTITION}\ :\ ${partition}\  }\n 
else
topicp=`echo $line | awk   39;{print $1}  39;`
partition=`echo $line | awk   39;{print $2}  39;`
printf  \t\t{ \ {#GROUP}\ :\ ${group}\ , \ {#TOPICP}\ :\ ${topicp}\ , \ {#PARTITION}\ :\ ${partition}\  },\n 
fi
done < /tmp/consumer-group-tmp
done
printf  \t]\n 
printf  }\n 
}
if [ $1 ==  discovery  ]; then
topic_discovery
elif [ $1 ==  lag  ];then
cal_topic $2 $3   /tmp/consumer-group
cat /tmp/consumer-group |awk -v t=$3 -v p=$4   39;{if($1==t      $2==p ){print $5}}  39;
else
echo  Usage: /data/scripts/consumer-group.sh discovery | lag 
fi
bash consumer-groups.sh discovery
## test-group test分区0 lag
bash consumer-groups.sh lag test-group test 0


二 Zabbix接入


2.1 Zabbix配置文件


vim userparameter_kafka.conf
UserParameter=topic_discovery,bash /data/scripts/consumer-groups.sh discovery
UserParameter=topic_log[*],bash /data/scripts/consumer-groups.sh lag  $1   $2   $3 


2.2 Zabbix



2.3 配置监控项



2.4 告警信息


告警主机:Kafka_192.168.3.55
主机IP:192.168.3.55
主机组:Kafka
告警时间:2022.03.21 00:23:10
告警等级:Average
告警信息:test-group/test/分区1:数据积压100
告警项目:topic_lag[test-group,test,1]



Zabbix对Kafka topic积压数据监控的问题(bug优化)
快三技巧准确率100_快三稳赚不赔 快三赚钱平台推荐_快三稳赚计划
相关内容