使用Telegraf+GrayLog實作Linux業務系統服務異常時自動推送釘釘告警
實作過程參考Telegraf官方文檔
https://docs.influxdata.com/telegraf/v1.24/get_started/
https://github.com/influxdata/telegraf/blob/release-1.24/plugins/inputs/exec/README.md
https://sbcode.net/grafana/telegraf-inputs-exec-monitor-ssh-sessions/
一、GrayLog上配置Telegraf的GELF UDP方式接入Input和Stream
步驟較簡單，下面隻展示配置時的一些截圖
(圖檔點選放大檢視)
(圖檔點選放大檢視)
(圖檔點選放大檢視)
(圖檔點選放大檢視)
(圖檔點選放大檢視)
記得GrayLog上開放input設定的端口
firewall-cmd --permanent --zone=public --add-port=12201/udp
firewall-cmd --reload
(圖檔點選放大檢視)
二、業務伺服器上安裝Telegraf并配置telegraf.conf
1、業務伺服器上建立一個XX服務檢測腳本
(圖檔點選放大檢視)
vim /opt/service_check.sh
#!/bin/sh
status=$(/usr/bin/systemctl status sshd | grep Active | awk -F "since" '{print $1}')
echo $status
chmod 777 /opt/service_check.sh
- 2、telegraf.conf配置檔案生成并修改
rpm -ivh telegraf-1.24.3-1.x86_64.rpm
telegraf --sample-config --input-filter exec --output-filter graylog > telegraf.conf
vim telegraf.conf
(圖檔點選放大檢視)
最終的telegraf.conf 配置檔案如下
#cat telegraf.conf | grep -v ^# | grep -v ^$ | grep -v ^.*## | grep -v ^.*#
[global_tags]
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = "0s"
hostname = ""
omit_hostname = false
[[outputs.graylog]]
servers = ["udp://192.168.31.170:12201"]
[[inputs.exec]]
commands = [ "sh /opt/service_check.sh" ]
timeout = "10s"
name_override = "sshd_service_status_check"
data_format = "value"
data_type = "string"
interval = "45s"
cd /etc/telegraf/
mv telegraf.conf telegraf.conf_default
cp /root/telegraf.conf ./
chmod 644 telegraf.conf
systemctl start telegraf
啟動報錯，原因為telegraf.conf的權根
(圖檔點選放大檢視)
(圖檔點選放大檢視)
三、GrayLog上檢視telegraf日志并配置告警
(圖檔點選放大檢視)
其中 PrometheusAlert告警模闆
(圖檔點選放大檢視)
## [Graylog告警資訊](.check_result.Event.Source)
### <font color=#FF0000>告警描述：{{.event_definition_description}}</font>
{{ range $k,$v:=.backlog }}
##### <font color="#FF0000">告警時間</font>：{{GetCSTtime $v.timestamp}} </br>
##### <font color="#FF0000">告警伺服器名稱</font>：{{$v.source}} </br>
##### <font color="#FF0000">告警伺服器IP位址</font>：{{$v.fields.gl2_remote_ip}} </br>
##### <font color="#FF0000">服務目前狀态</font>：{{$v.fields.value}} </br>
{{end}}
GrayLog告警配置過程截圖
(圖檔點選放大檢視)
(圖檔點選放大檢視)
(圖檔點選放大檢視)
(圖檔點選放大檢視)
(圖檔點選放大檢視)
四、最終釘釘上告警效果
(圖檔點選放大檢視)
(圖檔點選放大檢視)