Linux-ELK日志收集

1.ELK簡介

ELK是三個開源軟體的縮寫，分别表示：Elasticsearch , Logstash, Kibana , 它們都是開源軟體。新增了一個FileBeat，它是一個輕量級的日志收集處理工具(Agent)，Filebeat占用資源少，适合于在各個伺服器上搜集日志後傳輸給Logstash，官方也推薦此工具。

Elasticsearch是個開源分布式搜尋引擎，提供搜集、分析、存儲資料三大功能。它的特點有：分布式，零配置，自動發現，索引自動分片，索引副本機制，restful風格接口，多資料源，自動搜尋負載等。

Logstash 主要是用來日志的搜集、分析、過濾日志的工具，支援大量的資料擷取方式。一般工作方式為c/s架構，client端安裝在需要收集日志的主機上，server端負責将收到的各節點日志進行過濾、修改等操作在一并發往elasticsearch上去。

Kibana 也是一個開源和免費的工具，Kibana可以為 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以幫助彙總、分析和搜尋重要資料日志。

Filebeat隸屬于Beats。目前Beats包含四種工具：

                        1.Packetbeat（搜集網絡流量資料）

                        2.Topbeat（搜集系統、程序和檔案系統級别的 CPU 和記憶體使用情況等資料）

                        3.Filebeat（搜集檔案資料）

                        4.Winlogbeat（搜集 Windows 事件日志資料）

2.elk架構

3.環境準備

名稱	IP
es01	10.0.0.267
es02	10.0.0.268
web01	10.0.0.269

4.搭建elk

4.1關閉防火牆和selinux

systemctl stop firewalld 
systemctl disable firewalld
sed -i \'s/=enforcing/=disabled/g\' /etc/selinux/config
setenforce 0      

4.2配置yum源

[[email protected] ~]# cd /etc/yum.repos.d/
[[email protected] ~]# wget http://mirrors.aliyun.com/repo/Centos-7.repo      

4.3在es01和es02上，安裝jdk

[[email protected] /elk_soft]# yum install java-1.8.0-openjdk -y      

4.4在es01和es02上，上傳軟體到指定目錄，進入目錄安裝elasticsearch

[[email protected] ~]# cd /elk_soft/
[[email protected] /elk_soft]# ls
elasticsearch-6.4.1.rpm  kibana-6.4.1-x86_64.rpm  logstash-6.4.1.rpm
[[email protected] /elk_soft]# yum localinstall elasticsearch-6.4.1.rpm -y      

4.5在es01和es02上，修改elasticsearch配置檔案

[[email protected] /elk_soft]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: my-cluster        #叢集名稱
node.name: es02                 #節點名稱
path.data: /var/lib/elasticsearch     #存放資料的目錄（如果修改将目錄的屬組屬主改為elasticearch）
path.logs: /var/log/elasticsearch     #日志目錄（如果修改将目錄的屬組屬主改為elasticearch） 
network.host: 10.0.0.238        #本機IP位址
http.port: 9200                 #預設端口
discovery.zen.ping.unicast.hosts: ["10.0.0.237", "10.0.0.238"]       #叢集中每個node的IP位址
#在配置檔案倒數第二行加入，如果不添加是不能發現别的主機，隻能發現自己
http.cors.enabled: true
http.cors.allow-origin: "*"      

4.6啟動elasticearch服務

[[email protected] /elk_soft]# systemctl start elasticsearch.service 
[[email protected] /elk_soft]# systemctl enable elasticsearch.service 
[[email protected] /elk_soft]# netstat -lntup 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      6141/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      6222/master         
tcp6       0      0 10.0.0.237:9200         :::*                    LISTEN      8998/java           
tcp6       0      0 10.0.0.237:9300         :::*                    LISTEN      8998/java       

4.7測試

5.安裝head插件來管理叢集

5.1在es01上安裝head插件，配置epel源

wget http://mirrors.aliyun.com/repo/epel-7.repo      

5.2安裝nodejs，npm，openssl，screen

[[email protected] ~]# yum install nodejs npm openssl screen -y      

5.3檢視版本資訊

[[email protected] ~]# node -v
v6.17.1
[[email protected] ~]# npm -v
3.10.10      

5.4安裝cnpm

[[email protected] ~]# npm install -g cnpm --registry=https://registry.npm.taobao.org      

5.5安裝Git工具,把elasticearch-head下面的檔案克隆到本地

[[email protected] ~]# yum install git -y
[[email protected] ~]# cd /opt/
[[email protected] /opt]# git clone git://github.com/mobz/elasticsearch-head.git
[[email protected] /opt]# cd elasticsearch-head/
[[email protected] /opt/elasticsearch-head]# ls
crx                elasticsearch-head.sublime-project  index.html    plugin-descriptor.properties  _site
Dockerfile         Gruntfile.js                        LICENCE       proxy                         src
Dockerfile-alpine  grunt_fileSets.js                   package.json  README.textile                test
[[email protected] /opt/elasticsearch-head]# cnmp install 
[[email protected] /opt/elasticsearch-head]# screen -S es-head   #切屏指令
[[email protected] /opt/elasticsearch-head]# cnpm run start

> [email protected] start /opt/elasticsearch-head
> grunt server

Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100
###Ctrl+A+D  #切換快捷鍵  背景運作程式      

5.6如果正常啟動，開啟的端口是9100，可以在浏覽器中輸入http：//10.0.0.237:9100通路，如下圖所示：

5.7叢集的三種健康狀态

• 綠色：表示叢集健康
• 黃色：表示亞健康狀态，勉強可以運作
• 紅色：表示已經很危險，有可能資料已經丢失

6.elasticearch資料庫操作

6.1查入三條資料

curl -XPUT \'10.0.0.237:9200/megacorp/employee/1?pretty\' -H \'Content-Type: application/json\' -d\'
    {
        "first_name" : "Li",
        "last_name": "si",
        "age" : 28,
        "about" : "I love cat", "interests": [ "swimming" ]
    }
\'
curl -XPUT \'10.0.0.237:9200/megacorp/employee/2?pretty\' -H \'Content-Type: application/json\' -d\'
    {
        "first_name" : "Zhang",
        "last_name": "san",
        "age" : 22,
        "about" : "I love dog", "interests": [ "swimming" ]
    }
    \'
curl -XPUT \'10.0.0.237:9200/megacorp/employee/3?pretty\' -H \'Content-Type: application/json\' -d\'
    {
        "first_name" : "Wang",
        "last_name": "wu",
        "age" : 30,
        "about" : "I love beautiful girl", "interests": [ "swimming" ]
    }
    \'      

6.2資料插入成功後可以通過資料浏覽來檢視

6.3索引介紹

es資料庫	mysql資料庫
index（索引）	database（資料庫）
type （類型）	tables（資料庫中的表）
id （id号）	對應表中的字段

6.4如何查詢一條資料

[[email protected] /opt/elasticsearch-head]# curl -XGET \'10.0.0.237:9200/megacorp/employee/1?pretty\'
{
  "_index" : "megacorp",
  "_type" : "employee",
  "_id" : "1",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "first_name" : "Li",
    "last_name" : "si",
    "age" : 28,
    "about" : "I love cat",
    "interests" : [
      "swimming"
    ]
  }
}      

6.5如何删除一條資料

[[email protected] /opt/elasticsearch-head]#  curl -XDELETE \'10.0.0.237:9200/megacorp/employee/1?pretty\'
{
  "_index" : "megacorp",
  "_type" : "employee",
  "_id" : "1",
  "_version" : 2,
  "result" : "deleted",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 1,
  "_primary_term" : 1
}      

6.6如何建立索引

[[email protected] /opt/elasticsearch-head]# curl -XPUT \'10.0.0.237:9200/student_message?pretty\'
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "student_message"
}      

6.7建立一個索引為my_temp_index，分片為2，副本為1

curl -XPUT \'10.0.0.237:9200/my_temp_index?pretty\' -H \'Content-Type: application/json\' -d\' {
"settings": {
        "number_of_shards" : 2,
        "number_of_replicas" : 1
    }
}
\'      

7.kibana安裝配置

7.1在es01上安裝kibana

[[email protected] /elk_soft]# yum localinstall kibana-6.4.1-x86_64.rpm -y      

7.2修改配置檔案

[[email protected] /elk_soft]# vim /etc/kibana/kibana.yml 
server.port: 5601                    #服務監聽端口
server.host: "10.0.0.237"            #服務IP位址 
elasticsearch.url: "http://10.0.0.237:9200"         #elasticsearch的位址
elasticsearch.pingTimeout: 30000000
logging.dest: stdout      #如果指定日志存儲路徑，需要修改檔案的所有者及所屬組      

7.3開啟kibana服務，并設定開機自啟

[[email protected] /elk_soft]# systemctl start kibana.service 
[[email protected] /elk_soft]# systemctl enable kibana.service
[[email protected] /elk_soft]# ss -lnt
State      Recv-Q Send-Q             Local Address:Port                            Peer Address:Port              
LISTEN     0      128                            *:9100                                       *:*                  
LISTEN     0      128                            *:22                                         *:*                  
LISTEN     0      100                    127.0.0.1:25                                         *:*                  
LISTEN     0      128                   10.0.0.237:5601                                       *:*           

7.4通路測試

8.在web01上安裝nginx+logstash

8.1配置nginx的yum源，安裝nginx

[[email protected] /etc/yum.repos.d]# vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1
[[email protected] ~]# yum install nginx -y      

8.2修改nginx配置檔案

[[email protected] ~]# vim /etc/nginx/conf.d/elk.conf
server {
      listen 80;
      server_name elk.test.com;
      location / {
          proxy_pass      http://10.0.0.237:5601;
          proxy_set_header Host   $host;
          proxy_set_header X-Real-IP      $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      }
#      access_log  /var/log/nginx/elk_access.log main;
}      

8.3修改nginx.conf配置檔案，增加nginx手機json格式的日志檔案

[[email protected] ~]#  vim /etc/nginx/nginx.conf     
log_format access_json \'{"@timestamp":"$time_iso8601",\'
        \'"host":"$server_addr",\'
        \'"clientip":"$remote_addr",\'
        \'"size":$body_bytes_sent,\'
        \'"responsetime":$request_time,\'
        \'"upstreamtime":"$upstream_response_time",\'
        \'"upstreamhost":"$upstream_addr",\'
        \'"http_host":"$host",\'
        \'"url":"$uri",\'
        \'"domain":"$host",\'
        \'"xff":"$http_x_forwarded_for",\'
        \'"referer":"$http_referer",\'
        \'"status":"$status"}\';
    access_log  /var/log/nginx/   access_json;      

8.4啟動nginx

[[email protected] ~]# systemctl start nginx
[[email protected] ~]# systemctl enable nginx      

8.5修改本地解析檔案進行測試

10.0.0.239  elk.tast.com      

8.6安裝jdk

[[email protected] /elk_soft]# yum install java-1.8.0-openjdk -y      

8.7安裝logstash

[[email protected] /elk_soft]# ls
filebeat-6.4.1-x86_64.rpm  logstash-6.4.1.rpm      

8.8修改logstash配置檔案

[[email protected] /elk_soft]# vim /etc/logstash/logstash.yml 
path.data: /var/lib/logstash  #資料存放目錄（如需修改記得修改屬主屬組）
http.host: "10.0.0.239"      #http的IP位址
http.port: 9600              #監聽端口
path.logs: /var/log/logstash   #日志存放目錄（如需修改記得修改屬主屬組）      

8.9在/etc/logstash/conf.d/下新增nginx_access.conf檔案

[[email protected] ~]# vim /etc/logstash/conf.d/nginx_access.conf
input {
  file {
    path => "/var/log/nginx/access_json.log"
    start_position => "end"
    type => "nginx_access"
    codec => json
  }
}

output {
    elasticsearch {
      hosts => ["10.0.0.237:9200"]
      index => "nginx_access-%{+YYYY.MM.dd}"
   }
}      

9.0啟動logstash，并設定開機自啟

[[email protected] /etc/logstash]# sudo /usr/share/logstash/bin/system-install /etc/logstash/startup.options systemd
Using provided startup.options file: /etc/logstash/startup.options
Manually creating startup for specified platform: systemd
Successfully created system startup script for Logstash
[[email protected] /etc/logstash]# systemctl start logstash
[[email protected] /etc/logstash]# systemctl enable logstash      

9.1如果elasticearch沒收集到nginx日志，重新開機一下elasticearch服務

[[email protected] /elk_soft]# systemctl restart elasticsearch.service      

9.2.kibana平台展示所收到的日志資訊

9.2.1首先添加一個索引模式

9.2.2添加成功

9.2.3添加一個餅狀圖

9.2.4建立排班表

9.2.5通過dashboard進行詳細展示

9.通過rdis安裝部署新的架構

9.1停掉web01上的logstash服務

[[email protected] ~]# systemctl stop logstash.service       

9.2在web01上安裝filebeat

[[email protected] /elk_soft]# yum localinstall filebeat-6.4.1-x86_64.rpm  -y      

9.3修改filebeat配置檔案

24   enabled: true
 27   paths:
 28     - /var/log/nginx/access_json.log
##配置檔案最後添加  
  output.redis:
   hosts: ["10.0.0.237"]    #資料要發給的主機
   port: 6379               #Redis的端口
   key: "nginx-log"         #日志類型
   db: 0                    #兩端要一樣
   timeout: 5               #逾時時間5秒         

9.4啟動filebeat服務

[[email protected] ~]# systemctl start filebeat.service 
[[email protected] ~]# systemctl enable filebeat.service       

9.5在es01上安裝Redis

[[email protected] ~]# yum install redis -y      

9.6修改Redis配置檔案

[[email protected] ~]# vim /etc/redis.conf 
  61 bind 10.0.0.237       #redis本機ip位址
  port 6379                #redis預設端口
  128 daemonize yes        #開啟守護程序      

9.7啟動Redis

[[email protected] ~]# systemctl start redis 
[[email protected] ~]# systemctl enable redis
[[email protected] ~]# redis-cli -h 10.0.0.237
10.0.0.237:6379>       

9.8檢查Redis是否取到日志

10.0.0.237:6379> keys *
1) "nginx-log"
10.0.0.237:6379> info keyspace
# Keyspace
db0:keys=1,expires=0,avg_ttl=0      

9.8在es01上安裝logstash

[[email protected] /elk_soft]# yum localinstall logstash-6.4.1.rpm  -y      

10修改logstash的配置檔案

http.host: "10.0.0.237"      #http的IP位址
http.port: 9600              #監聽端口      

11在/etc/logstash/conf.d/下新增redis_input.conf檔案

[[email protected] /etc/logstash/conf.d]# vim redis_input.conf
input {

  redis {
        host => "10.0.0.237"
        port => "6379"
        db => "0"
        data_type => "list"
        key => "nginx-log"
   }


}
output {
    stdout { codec => rubydebug }
    elasticsearch {
        hosts => ["10.0.0.237:9200"]
        index => "nginx-redis-%{+YYYY.MM.dd}"
  }
}      

12啟動logstash

[[email protected] /etc/logstash/conf.d]# /usr/share/logstash/bin/system-install /etc/logstash/startup.options systemd
[[email protected] /etc/logstash/conf.d]# systemctl start logstash
[[email protected] /etc/logstash/conf.d]# systemctl enable logstash      

13kibana展示