Linux小型環境下通過網絡實作rsyslog日志收集、MySQL儲存及loganalyzer圖形展示
任務内容：收集Linux體系下主機rsyslog類的日志（SyslogClient-IP07 192.168.250.7 和SyslogClient-IP08 192.168.250.8），先以目錄檔案方式（Rsyslog-server-IP18 192.168.250.18）集中記錄，再轉錄到MySQL資料庫（Syslog-MySQL-IP28 192.168.250.28）内集中記錄，再用LogAnalyzer（LogAnalyzer-IP38 192.168.250.38）對記錄在MySQL内的日志進行圖形展示。
1. 架構及主機
五台伺服器
1 被收集日志主機A ：
主機名：SyslogClient-IP07
CentOS 7.9
IP: 192.168.250.7
rsyslog 8.24.0
2 被收集日志主機B ：
主機名：SyslogClient-IP08
CentOS 8.4
IP: 192.168.250.8
rsyslog 8.1911.0
3 日志伺服器 (目錄檔案) ：
主機名：Rsyslog-server-IP18
CentOS 8.4
IP: 192.168.250.18/24
4. 日志伺服器(MySQ庫) ：
主機名：Syslog-MySQL-IP28
CentOS 8.4
IP: 192.168.250.28/24
5.日志圖形展示伺服器 ：
主機名：LogAnalyzer-IP38
CentOS 8.4
IP: 192.168.250.38/24
# 說明：按照上面的架構圖，準備好五台主機，将以此為基礎環境完成日志的收集、集中存放、日志展示的實驗過程 2. 開啟網絡日志服務記錄
任務内容：啟用被收集的主機的網絡日志服務記錄功能，可以将多個遠端主機的日志，發送到集中的日志伺服器，集中日志伺服器可以是以目錄内日志檔案格式或者資料庫方式，這樣友善日志的統一管理和處理。
2.1. 認識rsyslog并開啟主機網絡日志記錄功能
2.1.1. CentOS8主機 192.168.250.8 開啟日志記錄功能
## 被收集日志主機B SyslogClient-IP08 CentOS8.4 IP: 192.168.250.8 啟用網絡日志記錄功能
# 修改主機名及驗證伺服器時間同步
[root@CentOS84 ]#
[root@CentOS84 ]#hostnamectl set-hostname rsyslog-IP08
[root@CentOS84 ]#exit
[root@SyslogClient-IP08 ]#systemctl enable --now chronyd.service
[root@SyslogClient-IP08 ]#
[root@SyslogClient-IP08 ]#date
[root@SyslogClient-IP08 ]#
[root@SyslogClient-IP08 ]#rpm -qi rsyslog
Name : rsyslog
Version : 8.1911.0
...........
[root@SyslogClient-IP08 ]#rpm -ql rsyslog
# 主要的配置檔案
/etc/logrotate.d/syslog
/etc/pki/rsyslog
/etc/rsyslog.conf
/etc/rsyslog.d
/etc/sysconfig/rsyslog
...........
# 很多SO子產品
/usr/lib64/rsyslog/omhttp.so
/usr/lib64/rsyslog/omjournal.so
/usr/lib64/rsyslog/ommail.so
/usr/lib64/rsyslog/omprog.so
/usr/lib64/rsyslog/omstdout.so
/usr/lib64/rsyslog/omtesting.so
/usr/lib64/rsyslog/omuxsock.so
/usr/lib64/rsyslog/pmaixforwardedfrom.so
/usr/lib64/rsyslog/pmcisconames.so
/usr/lib64/rsyslog/pmlastmsg.so
/usr/lib64/rsyslog/pmsnare.so
/usr/sbin/rsyslogd
/var/lib/rsyslog
[root@SyslogClient-IP08 ]#vim /etc/rsyslog.conf
[root@SyslogClient-IP08 ]#cat /etc/rsyslog.conf
.............
#### RULES ####
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# 保留上面行，日志可儲存本地；增加下面行，日志通過UDP514端口傳送到 192.168.250.18 伺服器記錄
*.info;mail.none;authpriv.none;cron.none @192.168.250.18:514
.................
[root@SyslogClient-IP08 ]#systemctl restart rsyslog
## 提示說明 @ 表示UDP514協定傳輸日志 @@ 表示TCP514協定傳輸日志 2.1.2. CentOS7主機 192.168.250.7 開啟日志記錄功能
[root@CentOS79 ]#hostnamectl set-hostname SyslogClient-IP07
[root@CentOS79 ]#exit
[root@SyslogClient-IP07 ]#vim /etc/rsyslog.conf
[root@SyslogClient-IP07 ]#cat /etc/rsyslog.conf
.............
#### RULES ####
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# 保留上面行，日志可儲存本地；增加下面行，日志通過TCP514端口傳送到 192.168.250.18 伺服器記錄
*.info;mail.none;authpriv.none;cron.none @@192.168.250.18:514
.................
[root@SyslogClient-IP07 ]#systemctl restart rsyslog 2.2. 啟用日志伺服器網絡記錄功能
基本任務：設定這個以目錄下日志檔案記錄來自網絡的其他主機日志的伺服器意圖是能更好了解網絡中日志傳輸流程和原理。本次計劃将IP192.168.250.7和IP192.168.250.8兩台主機日志，集中記錄到IP192.168.250.18 的 /var/log/messages下，雖然集中記錄在同一個檔案内，日志檔案中對不同主機的日志都有明确辨別，能完美分辨出來自哪台主機的日志。
## 日志伺服器 (檔案記錄方式) 主機名：Rsyslog-server-IP18 IP: 192.168.250.18的配置
[root@CentOS84 ]#hostnamectl set-hostname Rsyslog-server-IP18
[root@CentOS84 ]#exit
[root@Rsyslog-server-IP18 ]#systemctl enable --now chronyd.service
[root@Rsyslog-server-IP18 ]#
[root@Rsyslog-server-IP18 ]#cat /etc/rsyslog.conf
................
#### MODULES ####
................
# Provides UDP syslog reception 取消下面行行首的#注釋符，讓其生效，接受來自網絡的UPD514日志
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
# Provides TCP syslog reception 取消下面行行首的#注釋符，讓其生效，接受來自網絡的TCP514日志
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")
.................
[root@Rsyslog-server-IP18 ]#
[root@Rsyslog-server-IP18 ]#systemctl restart rsyslog 2.3. 驗證日志傳輸和記錄
# 在192.168.250.7上用logger發送一個測試日志資訊
[root@SyslogClient-IP07 <sub>]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07 </sub>]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07 <sub>]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07 </sub>]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07 <sub>]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07 </sub>]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07 <sub>]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07 </sub>]# logger "hello,I am log test! IP IS 192.168.250.7 TCP"
[root@SyslogClient-IP07 ~]#
# 在192.168.250.8上用logger發送一個測試日志資訊
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP"
[root@SyslogClient-IP08 ]#
# 在192.168.250.18上用檢視日志資訊記錄
[root@Rsyslog-server-IP18 ]#tail -f /var/log/messages
...................
Mar 10 19:03:05 SyslogClient-IP08 root[21952]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:09 SyslogClient-IP08 root[21953]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:09 SyslogClient-IP08 root[21954]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:10 SyslogClient-IP08 root[21955]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:10 SyslogClient-IP08 root[21956]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:11 SyslogClient-IP08 root[21957]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:11 SyslogClient-IP08 root[21958]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:11 SyslogClient-IP08 root[21959]: hello,I am log test! IP IS 192.168.250.8 UDP
Mar 10 19:03:17 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP
Mar 10 19:03:17 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP
Mar 10 19:03:17 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP
Mar 10 19:03:17 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP
Mar 10 19:03:18 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP
Mar 10 19:03:18 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP
Mar 10 19:03:19 syslogclient-ip07 root: hello,I am log test! IP IS 192.168.250.7 TCP 3. 利用 MySQL 存儲日志資訊
基本任務：利用Rsyslog-server-IP18伺服器的rsyslog日志服務， 将日志伺服器 (檔案記錄方式) syslog-server-IP18收集記錄的日志，轉儲到日志伺服器(MySQ庫)Syslog-MySQL-IP28上利用 MySQL 存儲日志資訊。
3.1. 在Rsyslog-server-IP18伺服器上安裝連接配接mysql子產品相關的程式包
# 安裝資料庫連接配接包
[root@Rsyslog-server-IP18 ]#yum -y install rsyslog-mysql
# 驗證并檢視包内容
[root@Rsyslog-server-IP18 ]#rpm -ql rsyslog-mysql
/usr/lib/.build-id
/usr/lib/.build-id/e6
/usr/lib/.build-id/e6/aa0e40c19a2e0524d72780eee3b1698684cbe7
/usr/lib64/rsyslog/ommysql.so
/usr/share/doc/rsyslog/mysql-createDB.sql #這個是寫好的後端資料庫檔案
# 檢視SQL腳本檔案内容，是後端伺服器生成資料庫和表用
[root@Rsyslog-server-IP18 ]#cat /usr/share/doc/rsyslog/mysql-createDB.sql
CREATE DATABASE Syslog;
USE Syslog;
CREATE TABLE SystemEvents
(
ID int unsigned not null auto_increment primary key,
CustomerID bigint,
ReceivedAt datetime NULL,
DeviceReportedTime datetime NULL,
Facility smallint NULL,
Priority smallint NULL,
FromHost varchar(60) NULL,
Message text,
NTSeverity int NULL,
Importance int NULL,
EventSource varchar(60),
EventUser varchar(60) NULL,
EventCategory int NULL,
EventID int NULL,
EventBinaryData text NULL,
MaxAvailable int NULL,
CurrUsage int NULL,
MinUsage int NULL,
MaxUsage int NULL,
InfoUnitID int NULL ,
SysLogTag varchar(60),
EventLogType varchar(60),
GenericFileName VarChar(60),
SystemID int NULL
);
CREATE TABLE SystemEventsProperties
(
ID int unsigned not null auto_increment primary key,
SystemEventID int NULL ,
ParamName varchar(255) NULL ,
ParamValue text NULL
);
#将sql腳本複制到資料庫伺服器192.168.250.28上
[root@Rsyslog-server-IP18 ]#scp /usr/share/doc/rsyslog/mysql-createDB.sql 192.168.250.28:/data/ 3.2. 準備日志存放的MySQL資料庫伺服器
# 修改主機名和同步時間
[root@CentOS84 ]#hostnamectl set-hostname Syslog-MySQL-IP28
[root@CentOS84 ]#exit
[root@Syslog-MySQL-IP28 ]#systemctl enable --now chronyd.service
[root@Syslog-MySQL-IP28 ]#date
# 安裝資料庫并啟動
[root@Syslog-MySQL-IP28 ]#yum -y install mysql-server
[root@Syslog-MySQL-IP28 ]#systemctl start mysqld
# 利用前面複制過來的sql腳本生成日志資料庫和表
[root@Syslog-MySQL-IP28 ]#mysql < mysql-createDB.sql
[root@Syslog-MySQL-IP28 ]#mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 8.0.26 Source distribution
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| Syslog |
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.01 sec)
mysql> use Syslog
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+------------------------+
| Tables_in_Syslog |
+------------------------+
| SystemEvents |
| SystemEventsProperties |
+------------------------+
2 rows in set (0.00 sec)
# 在MySQL資料庫伺服器上建立好相關資料庫和表，授權rsyslog IP18 能連接配接至目前伺服器
mysql> create user rsyslog@'192.168.250.%' identified by 'shone123456';
Query OK, 0 rows affected (0.01 sec)
mysql> grant all on Syslog.* to rsyslog@'192.168.250.%';
Query OK, 0 rows affected (0.01 sec)
# 用下面指令記錄下資料庫内的記錄
mysql> select * from SystemEvents\G 3.3. 配置日志伺服器将日志發送到後端資料庫伺服器
基本内容：配置日志伺服器IP192.168.250.18 将日志發送到後端資料庫伺服器IP192.168.250.28
# 配置rsyslog将日志儲存到mysql中
# 對/etc/rsyslog.conf檔案進行再次修改，增加 module(load="ommysql") 和 *.info;mail.none;authpriv.none;cron.none :ommysql:192.168.250.28,Syslog,rsyslog,shone123456 這兩行。
[root@Rsyslog-server-IP18 ]#vim /etc/rsyslog.conf
[root@Rsyslog-server-IP18 ]#cat /etc/rsyslog.conf
................
#### MODULES ####
................
# 下面行為本次增加的内容，啟動sql連接配接
module(load="ommysql")
# Provides UDP syslog reception
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
# Provides TCP syslog reception
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")
#### RULES ####
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# 下面行為本次增加的内容，将日志轉存到後端的MySQL資料庫，定義和後端資料庫連接配接的使用者名和密&碼
*.info;mail.none;authpriv.none;cron.none :ommysql:192.168.250.28,Syslog,rsyslog,shone123456
[root@Rsyslog-server-IP18 ]#systemctl restart rsyslog.service 3.4. 驗證日志傳輸和記錄
# 在兩台日志client主機上發送兩台測試日志
[root@syslogclient-ip07 ~]# logger "hello,I am log test! IP IS 192.168.250.7 TCP MYSQL LOG"
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP MYSQL LOG"
# 在資料庫内查詢，并确認日志是否被正确記錄
[root@Syslog-MySQL-IP28 ]#mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 54
Server version: 8.0.26 Source distribution
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| Syslog |
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.00 sec)
mysql> use Syslog
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from SystemEvents\G
....................
*************************<strong> 37725. row </strong>***********************<strong>
ID: 37725
CustomerID: NULL
ReceivedAt: 2022-03-10 19:52:49
DeviceReportedTime: 2022-03-10 19:52:49
Facility: 1
Priority: 5
FromHost: SyslogClient-IP08
Message: hello,I am log test! IP IS 192.168.250.8 UDP MYSQL LOG
NTSeverity: NULL
Importance: NULL
EventSource: NULL
EventUser: NULL
EventCategory: NULL
EventID: NULL
EventBinaryData: NULL
MaxAvailable: NULL
CurrUsage: NULL
MinUsage: NULL
MaxUsage: NULL
InfoUnitID: 1
SysLogTag: root[22397]:
EventLogType: NULL
GenericFileName: NULL
SystemID: NULL
processid:
</strong>***********************<strong> 37729. row </strong>*************************
ID: 37729
CustomerID: NULL
ReceivedAt: 2022-03-10 19:52:53
DeviceReportedTime: 2022-03-10 19:52:53
Facility: 1
Priority: 5
FromHost: syslogclient-ip07
Message: hello,I am log test! IP IS 192.168.250.7 TCP MYSQL LOG
NTSeverity: NULL
Importance: NULL
EventSource: NULL
EventUser: NULL
EventCategory: NULL
EventID: NULL
EventBinaryData: NULL
MaxAvailable: NULL
CurrUsage: NULL
MinUsage: NULL
MaxUsage: NULL
InfoUnitID: 1
SysLogTag: root:
EventLogType: NULL
GenericFileName: NULL
SystemID: NULL
processid:
.............. 4. 通過 loganalyzer 展示資料庫中記錄的日志
基本任務：前面完成了日志兩台clients主機（192.168.250.7和192.168.250.8）、一台日志伺服器（檔案記錄）192.168.250.18 、一台日志伺服器（MySQL記錄）192.168.250.28 的配置，在這個小章節内将完成在一台192.168.250.38伺服器上安裝httpd+php +loganalyzer以圖形展示日志資訊。
4.1. 安裝 php和相關軟體包
# 同步時間，修改主機名
[root@CentOS84 ]#hostnamectl set-hostname LogAnalyzer-IP38
[root@CentOS84 ]#exit
[root@LogAnalyzer-IP38 ]#systemctl enable --now chronyd.service
# 安裝httpd，php，其他依賴包，其中php-gd是PHP中生成圖形的支撐包，不安裝無法彙出圖
[root@LogAnalyzer-IP38 ]#yum -y install httpd php-fpm php-mysqlnd php-gd
# 啟動服務
[root@LogAnalyzer-IP38 ]#systemctl enable --now httpd php-fpm
# 預設安裝後的PHP是以UDS的方式運作
[root@LogAnalyzer-IP38 ]#ll /run/php-fpm/
total 4
-rw-r--r-- 1 root root 6 Mar 9 23:11 php-fpm.pid
srw-rw----+ 1 root root 0 Mar 9 23:11 www.sock
[root@LogAnalyzer-IP38 ]#ll /run/php-fpm/www.sock
srw-rw----+ 1 root root 0 Mar 9 23:11 /run/php-fpm/www.sock
[root@LogAnalyzer-IP38 ]#grep www.sock /etc/httpd/conf.d/php.conf
SetHandler "proxy:unix:/run/php-fpm/www.sock|fcgi://localhost"
[root@LogAnalyzer-IP38 ]#rpm -qf /etc/httpd/conf.d/php.conf
php-fpm-7.2.24-1.module_el8.2.0+313+b04d0a66.x86_64
# 準備一個PHP測試頁面
[root@LogAnalyzer-IP38 ]#vim /var/www/html/phpinfo.php
[root@LogAnalyzer-IP38 ]#cat /var/www/html/phpinfo.php
<?php phpinfo() ?> 在浏覽器内打開PHP測試頁面，驗證PHP環境正确配置了 http://192.168.250.38/phpinfo.php
4.2. 下載下傳LogAnalyzer
登入官網 https://loganalyzer.adiscon.com/ ，下載下傳 loganalyzer-4.1.12.tar.gz安裝包
4.3. 安裝 LogAnalyzer
# 解壓安裝包
[root@LogAnalyzer-IP38 ]#ll
-rw-r--r-- 1 root root 5028816 Mar 9 23:12 loganalyzer-4.1.12.tar.gz
[root@LogAnalyzer-IP38 ]#tar xvf loganalyzer-4.1.12.tar.gz
[root@LogAnalyzer-IP38 ]#ll
drwxrwxr-x 5 root root 90 Apr 29 2021 loganalyzer-4.1.12
-rw-r--r-- 1 root root 5028816 Mar 9 23:12 loganalyzer-4.1.12.tar.gz
[root@LogAnalyzer-IP38 ]#ll loganalyzer-4.1.12
total 104
-rw-rw-r-- 1 root root 50019 Apr 29 2021 ChangeLog
drwxrwxr-x 2 root root 43 Apr 29 2021 contrib
-rw-rw-r-- 1 root root 35497 Apr 29 2021 COPYING
drwxrwxr-x 3 root root 258 Apr 29 2021 doc
-rw-rw-r-- 1 root root 8449 Apr 29 2021 INSTALL
drwxrwxr-x 13 root root 4096 Apr 29 2021 src
# 将loganalyzer源檔案遷移到網站預設目錄/var/www/html/下
[root@LogAnalyzer-IP38 ]#mv loganalyzer-4.1.12/src/ /var/www/html/log
# 建立好後面PHP初始化需要重寫的配置檔案，并授權。如果不建立這個檔案，初始化會提示錯誤。
[root@LogAnalyzer-IP38 ]#touch /var/www/html/log/config.php
[root@LogAnalyzer-IP38 ]#chmod 666 /var/www/html/log/config.php 4.4. 基于 web 頁面初始化
4.4.1 浏覽器内通路 http://192.168.250.38/log 實作初始化
選擇：MySQL Native, Syslog Fields, Monitorware
4.4.2 配置完成後可以以圖形方式顯示存放在MySQL中的日志
4.5. 安全加強
縮小配置檔案權限，提高安全性，也避免這個檔案被重新。
[root@LogAnalyzer-IP38 ]#chmod 644 /var/www/html/log/config.php 4.6. 驗證圖形下日志資訊
# 在兩台日志clients主機上發送幾條日志測試資訊
[root@SyslogClient-IP08 ]#logger "hello,I am log test! IP IS 192.168.250.8 UDP MYSQL LOG"
[root@syslogclient-ip07 ~]#logger "hello,I am log test! IP IS 192.168.250.7 TCP MYSQL LOG"