ELK技术4--收集docker日志+kibana自定义画图

文章目录

• 一.架构图
• 二.安装docker
• 三.运行nginx镜像
• • 1.运行nginx镜像
  • 2.查看日志
• 四.配置filebeat通过标签收集多个容器日志
• • 1.安装docker-compose
  • 2.pip安装
  • 3.继续安装docker-compose
  • 4.检查
  • 5.编写docker-compose.yml
  • 6.清理镜像
  • 7.运行docker-compose.yml
  • 8.检查日志是否增加了lable标签
  • 9.配置filebeat通过标签收集多个容器日志
• 五.配置filebeat通过服务类型和日志类型多条件创建不同索引
• 六.使用fillebeat modules配置
• • 1.实验准备
  • 2.激活nginx模块
  • 3.修改nginx日志为普通格式
  • 4.验证
• 七.导入kibana视图
• • 1.实验配置
  • 2.使用kibana画图

一.架构图

二.安装docker

1.安装依赖包
yum install -y yum-utils device-mapper-persistent-data lvm2
2.yum源准备
wget -O /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
sed -i 's+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+' /etc/yum.repos.d/docker-ce.repo
yum makecache fast
3.安装docker-ce
yum install docker-ce
4.启动docker服务
systemctl start docker

使得下载镜像更加快速
阿里云Docker-hub
https://cr.console.aliyum.com/cn-hangzhou/mirrors

mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
	"registry-mirrors": ["https://uoggbpok.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload
systemctl restart docker
           

三.运行nginx镜像

1.运行nginx镜像

docker pull nginx
[[email protected] ~]#  docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS        PORTS               NAMES
[[email protected] ~]# systemctl stop nginx
[[email protected] ~]# docker run --name nginx -p 80:80 -d nginx
0e0c9217852da129259f83714a6115576d797dfbe68c58a91bd2afa982068cb6
[[email protected] ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS             PORTS                NAMES
0e0c9217852d        nginx               "nginx -g 'daemon of…"   17 seconds ago      Up 15 seconds       0.0.0.0:80->80/tcp   nginx
[[email protected] ~]# docker start 0e0c9217852d
0e0c9217852d
[[email protected] ~]# docker exec -it 0e0c9217852d /bin/bash
[email protected]:/# cat /etc/debian_version
10.4
[email protected]:/# 
           

http://10.0.0.51/

2.查看日志

[[email protected] ~]# docker logs -f nginx
10.0.0.1 - - [19/Jul/2020:00:52:27 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" "-"

日志存放的地方：
[[email protected] ~]# cd /var/lib/docker/containers
[[email protected] containers]# ll
total 0
drwx------ 4 root root 237 Jul 19 08:43 244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1
[[email protected] containers]# cd  244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1
[[email protected] 244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1]# ll
total 28
-rw-r----- 1 root root 1952 Jul 19 08:52 244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1-json.log
           

四.配置filebeat通过标签收集多个容器日志

假如我们有多个docker镜像或者重新提交了新镜像，那么直接指定ID的就不是太方便了。
我们从当前的容器提交一个新的镜像并且运行起来
docker commit nginx nginx:v2
docker images
docker run --name nginx-v2 -p 8080:80 -d nginx:v2
此时我们的容器目录下就有了两个不同的容器目录
[[email protected] ~]# ls /var/lib/docker/containers/
244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1
f1ed338bba38a7dc45524d03a8ce2b47dcd243d0b310315ad459faa3776e1064
如果直接配置filebeat存到es里本台机器所有的容器日志都会混在一起没有办法区分
多容器日志收集处理：
其实收集的日志本质来说还是文件，而这个日志是以容器-json.log命名存放在默认目录下的json格式的文件：
[[email protected] ~]# head -1 /var/lib/docker/containers/244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1/244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1-json.log 
{"log":"10.0.0.1 - - [19/Jul/2020:00:52:31 +0000] \"GET /favicon.ico HTTP/1.1\" 404 555 \"http://10.0.0.51/\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36\" \"-\"\n","stream":"stdout","time":"2020-07-19T00:52:31.714101121Z"}

但是每个容器的ID都不一样，为了区分不同服务运行的不同容器，可以使用docker-compose通过给容器添加labels标签来作为区分
然后filbeat把容器日志当作普通的json格式来解析并传输到es
           

1.安装docker-compose

yum install -y python2-pip
           

2.pip安装

默认源为国外，可以使用国内加速，相关网站
https://mirrors.tuna.tsinghua.edu.cn/help/pypi/
pip加速操作命令
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple pip -U
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
           

3.继续安装docker-compose

[[email protected] ~]# yum install -y gcc
[[email protected] ~]# yum install -y libffi-devel python-devel openssl-devel
[[email protected] ~]#pip install docker-compose
           

4.检查

docker-compose version
           

5.编写docker-compose.yml

[[email protected] ~]# vi docker-compose.yml （注意要用vi而不是vim，vim复制的代码会乱）
version: '3'
services:
  nginx:
    image: nginx:v2
    # 设置labels
    labels:
      service: nginx
    # logging设置增加labels.service
    logging:
      options:
        labels: "service"
    ports:
      - "8080:80"
  db:
    image: nginx:latest
    # 设置labels
    labels:
      service: db 
    # logging设置增加labels.service
    logging:
      options:
        labels: "service"
    ports:
      - "80:80"
           

6.清理镜像

[[email protected] ~]#  docker stop nginx
nginx
[[email protected] ~]#  docker stop nginx-v2
nginx-v2
[[email protected] ~]# docker rm nginx
nginx
[[email protected] ~]# docker rm nginx-v2
nginx-v2 
[[email protected] ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS        PORTS               NAMES
           

7.运行docker-compose.yml

docker-compose up 
会卡住
打开另一个窗口：
[[email protected] ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS             PORTS                  NAMES
17ff5d3bc2d2        nginx:latest        "nginx -g 'daemon of…"   7 minutes ago       Up 2 seconds        0.0.0.0:80->80/tcp     root_db_1
0624dc20ff14        nginx:v2            "nginx -g 'daemon of…"   7 minutes ago       Up 2 seconds        0.0.0.0:8080->80/tcp   root_nginx_1
[[email protected] ~]# systemctl restart docker 
           

访问：

http://10.0.0.51/nginx

http://10.0.0.51:8080/db

8.检查日志是否增加了lable标签

[[email protected] ~]# cd /var/lib/docker/containers/
[[email protected] containers]# ll
total 0
drwx------ 4 root root 237 Jul 19 10:58 47174401e9bfe79d53accb4199233a65c1d6806f328224a4bc32d69739011009
drwx------ 4 root root 237 Jul 19 10:58 9838b5324afb43b6544afcef841429db9176eeb89025d7586231d2513d19302d
[[email protected] containers]# cd 47174401e9bfe79d53accb4199233a65c1d6806f328224a4bc32d69739011009
[[email protected] 47174401e9bfe79d53accb4199233a65c1d6806f328224a4bc32d69739011009]# ll
total 44
-rw-r----- 1 root root 18732 Jul 19 10:58 47174401e9bfe79d53accb4199233a65c1d6806f328224a4bc32d69739011009-json.log
[[email protected] 47174401e9bfe79d53accb4199233a65c1d6806f328224a4bc32d69739011009]# cat 47174401e9bfe79d53accb4199233a65c1d6806f328224a4bc32d69739011009-json.log#查看得到日志
           

9.配置filebeat通过标签收集多个容器日志

[[email protected] ~]# cat /etc/filebeat/filebeat.yml    
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "docker-nginx-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        attrs.service: "nginx"
    - index: "docker-db-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        attrs.service: "db"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
           

五.配置filebeat通过服务类型和日志类型多条件创建不同索引

目前为止，已经可以按服务来收集日志了，但是错误日志和正确日志混在了一起，不好区分，所以可以进一步进行条件判断，根据服务和日志类型创建不同的索引
filebeat配置文件开始：docker容器拆解日志
[[email protected] ~]# cat /etc/filebeat/filebeat.yml        
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
          attrs.service: "nginx"
          stream: "stdout"
    - index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
          attrs.service: "nginx"
          stream: "stderr"
    - index: "docker-db-access-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
          attrs.service: "db"
          stream: "stdout"
    - index: "docker-db-error-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
          attrs.service: "db"
          stream: "stderr"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
[[email protected] ~]# systemctl restart filebeat
           

查看：

http://10.0.0.51:5601

六.使用fillebeat modules配置

1.实验准备

[[email protected] ~]# >/etc/filebeat/filebeat.yml
配置文件
[[email protected] ~]# vi /etc/filebeat/filebeat.yml
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "docker-access-%{[beat.version]}-%{+yyyy.MM}"
[[email protected] ~]# filebeat modules list
           

2.激活nginx模块

（2种方法）
查看状态：
[[email protected] ~]# cd /etc/filebeat/modules.d/
[[email protected] modules.d]# ll
第一种：
[[email protected] ~]# filebeat modules enable nginx
Enabled nginx
[[email protected] ~]# filebeat modules list
Enabled:
nginx
第二种：
[[email protected] modules.d]# mv nginx.yml.disabled nginx.yml
[[email protected] modules.d]# filebeat modules list
Enabled:
nginx
           

3.修改nginx日志为普通格式

[[email protected] modules.d]# vim /etc/nginx/nginx.conf
access_log  /var/log/nginx/access.log  main;
出现问题：
[[email protected] modules.d]# systemctl start nginx
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
解决方法：
[[email protected] modules.d]# systemctl stop docker
[[email protected] modules.d]# systemctl disable docker
[[email protected] modules.d]# iptables -F
[[email protected] modules.d]# iptables -X
[[email protected] modules.d]# iptables -Z
[[email protected] modules.d]# iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
[[email protected] modules.d]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[[email protected] modules.d]# systemctl start nginx
           

4.验证

[[email protected] modules.d]# curl 10.0.0.51
[[email protected] modules.d]# tail -f /var/log/nginx/access.log
10.0.0.51 - - [19/Jul/2020:11:26:53 +0800] "GET / HTTP/1.1" 200 4833 "-" "curl/7.29.0" "-"
[[email protected] modules.d]# ab -n 20 -c 20 http://10.0.0.51/test
配置文件
[[email protected] modules.d]# vi nginx.yml
- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/access.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/error.log"]
"nginx.yml" 16L, 426C written
[[email protected] modules.d]# vim /etc/filebeat/filebeat.yml
#index: "docker-access-%{[beat.version]}-%{+yyyy.MM}"注释掉这一行
[[email protected] modules.d]# systemctl restart filebeat
[[email protected] modules.d]# find / -name "elasticsearch-plugin"
/usr/share/elasticsearch/bin/elasticsearch-plugin
[[email protected] modules.d]# /usr/share/elasticsearch/bin/elasticsearch-plugin  install ingest-user-agent
[[email protected] modules.d]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
[[email protected] modules.d]# systemctl restart filebeat

[[email protected] modules.d]# vi /etc/filebeat/filebeat.yml
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

  setup.kibana:
    host: "10.0.0.51:5601"
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
  - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM.dd}"
    when.contains:
        fileset.name: "access"
  - index: "nginx_error-%{[beat.version]}-%{+yyyy.MM.dd}"
    when.contains:
        fileset.name: "error"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true

"/etc/filebeat/filebeat.yml" 23L, 558C written
[[email protected] modules.d]# systemctl restart nginx
[[email protected] modules.d]# systemctl restart elasticsearch
[[email protected] modules.d]# systemctl restart filebeat
           

查看：

七.导入kibana视图

1.实验配置

默认如果使用filbeat模版导入视图会把所有的服务都导入进去,而我们实际上并不需要这么多视图，
而且默认的视图模版只能匹配filebeat-*开头的索引，所以这里我们有2个需要解决：
1.通过一定处理只导入我们需要的模版
2.导入的视图模版索引名称可以自定义
解决方法：
1.备份一份filebeat的kibana视图，删除不需要的视图模版文件
2.修改视图文件里默认的索引名称为我们需要的索引名称
[[email protected] ~]# cp -a /usr/share/filebeat/kibana /root
[[email protected] ~]# cd kibana/
[[email protected] kibana]# rm -rf 5
[[email protected] kibana]# cd 6/dashboard/
[[email protected] dashboard]# find . -type f ! -name "*nginx*"|xargs rm -rf
[[email protected] dashboard]# rm -fr ml-nginx-*
[[email protected] dashboard]# sed -i 's#filebeat\-\*#nginx\-\*#g' Filebeat-nginx-overview.json
[[email protected] dashboard]# sed -i 's#filebeat\-\*#nginx\-\*#g' Filebeat-nginx-logs.json
[[email protected] dashboard]# cd ..
[[email protected] 6]# ls
dashboard  index-pattern
[[email protected] 6]# cd index-pattern/
[[email protected] index-pattern]# sed -i 's#filebeat\-\*#nginx\-\*#g' filebeat.json
[[email protected] index-pattern]# cd ..
[[email protected] 6]# ls
dashboard  index-pattern
[[email protected] 6]# cd
[[email protected] ~]# filebeat setup --dashboards -E setup.dashboards.directory=/root/kibana/
           

2.使用kibana画图

画图实验准备：
[[email protected] ~]# >/var/log/nginx/access.log
上传nginx_access.tar.gz（准备好的日志文件）
[[email protected] ~]# tar -zxvf nginx_access.tar.gz
var/log/nginx/access.log
[[email protected] ~]# mv var/log/nginx/access.log /var/log/nginx/access.log
[[email protected]01 ~]#vi /etc/filebeat/filebeat.yml
[[email protected] ~]# cat  /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

setup.kibana:
  host: "10.0.0.51:5601"

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  #index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "access"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true

[[email protected] ~]# systemctl restart filebeat
[[email protected] ~]# systemctl restart nginx
           

实验操作：

删除之前的项目

新建一个项目

把之前的模板删除，自己画图

调整ip地址方向

保存画图