天天看點
傳回

部署 docker 元件F. 部署 docker 元件

tags: worker, docker

F. 部署 docker 元件

<!-- TOC -->

    • 安裝依賴包
    • 下載下傳和分發 docker 二進制檔案
    • 建立和分發 systemd unit 檔案
    • 配置和分發 docker 配置檔案
    • 啟動 docker 服務
    • 檢查服務運作狀态
    • 檢查 docker0 網橋
    • 檢視 docker 的狀态資訊
    • 更新 kubelet 配置并重新開機服務（每個節點上都操作）

<!-- /TOC -->

docker 運作和管理容器，kubelet 通過 Container Runtime Interface (CRI) 與它進行互動。

注意：

  1. 如果沒有特殊指明，本文檔的所有操作均在 zhangjun-k8s01 節點上執行，然後遠端分發檔案和執行指令；
  2. 需要先安裝 flannel，請參考附件 E.部署flannel網絡.md；

參考 06-1.部署worker節點.md。

到 docker 下載下傳頁面 下載下傳最新釋出包：

cd /opt/k8s/work
wget https://download.docker.com/linux/static/stable/x86_64/docker-18.09.6.tgz
tar -xvf docker-18.09.6.tgz

分發二進制檔案到所有 worker 節點：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp docker/*  root@${node_ip}:/opt/k8s/bin/
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
  done           

cd /opt/k8s/work
cat > docker.service <<"EOF"
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io

[Service]
WorkingDirectory=##DOCKER_DIR##
Environment="PATH=/opt/k8s/bin:/bin:/sbin:/usr/bin:/usr/sbin"
EnvironmentFile=-/run/flannel/docker
ExecStart=/opt/k8s/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target
EOF
  • EOF 前後有雙引号，這樣 bash 不會替換文檔中的變量，如

    $DOCKER_NETWORK_OPTIONS

     (這些環境變量是 systemd 負責替換的。)；
  • dockerd 運作時會調用其它 docker 指令，如 docker-proxy，是以需要将 docker 指令所在的目錄加到 PATH 環境變量中；
  • flanneld 啟動時将網絡配置寫入

    /run/flannel/docker

     檔案中，dockerd 啟動前讀取該檔案中的環境變量

    DOCKER_NETWORK_OPTIONS

     ，然後設定 docker0 網橋網段；
  • 如果指定了多個

    EnvironmentFile

     選項，則必須将

    /run/flannel/docker

     放在最後(確定 docker0 使用 flanneld 生成的 bip 參數)；
  • docker 需要以 root 用于運作；
  • docker 從 1.13 版本開始，可能将 iptables FORWARD chain的預設政策設定為DROP，進而導緻 ping 其它 Node 上的 Pod IP 失敗，遇到這種情況時，需要手動設定政策為

    ACCEPT

     ： 
    $ sudo iptables -P FORWARD ACCEPT
    并且把以下指令寫入

    /etc/rc.local

     檔案中，防止節點重新開機iptables FORWARD chain的預設政策又還原為DROP 
    /sbin/iptables -P FORWARD ACCEPT

分發 systemd unit 檔案到所有 worker 機器:

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
sed -i -e "s|##DOCKER_DIR##|${DOCKER_DIR}|" docker.service
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp docker.service root@${node_ip}:/etc/systemd/system/
  done

使用國内的倉庫鏡像伺服器以加快 pull image 的速度，同時增加下載下傳的并發數 (需要重新開機 dockerd 生效)：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > docker-daemon.json <<EOF
{
    "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn","https://hub-mirror.c.163.com"],
    "insecure-registries": ["docker02:35000"],
    "max-concurrent-downloads": 20,
    "live-restore": true,
    "max-concurrent-uploads": 10,
    "debug": true,
    "data-root": "${DOCKER_DIR}/data",
    "exec-root": "${DOCKER_DIR}/exec",
    "log-opts": {
      "max-size": "100m",
      "max-file": "5"
    }
}
EOF

分發 docker 配置檔案到所有 worker 節點：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p  /etc/docker/ ${DOCKER_DIR}/{data,exec}"
    scp docker-daemon.json root@${node_ip}:/etc/docker/daemon.json
  done           

source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable docker && systemctl restart docker"
  done           

source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status docker|grep Active"
  done

確定狀态為

active (running)

，否則檢視日志，确認原因：

journalctl -u docker           

source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "/usr/sbin/ip addr show flannel.1 && /usr/sbin/ip addr show docker0"
  done

确認各 worker 節點的 docker0 網橋和 flannel.1 接口的 IP 處于同一個網段中(如下 172.30.80.0/32 位于 172.30.80.1/21 中)：

>>> 172.27.137.240
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether ce:9c:a9:08:50:03 brd ff:ff:ff:ff:ff:ff
    inet 172.30.80.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:5c:c1:77:03 brd ff:ff:ff:ff:ff:ff
    inet 172.30.80.1/21 brd 172.30.87.255 scope global docker0
       valid_lft forever preferred_lft forever           
systemctl stop docker
ip link delete docker0
systemctl start docker           

$ ps -elfH|grep docker
4 S root     116590      1  0  80   0 - 131420 futex_ 11:22 ?       00:00:01   /opt/k8s/bin/dockerd --bip=172.30.80.1/21 --ip-masq=false --mtu=1450
4 S root     116668 116590  1  80   0 - 161643 futex_ 11:22 ?       00:00:03     containerd --config /data/k8s/docker/exec/containerd/containerd.toml --log-level debug           
$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.09.6
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.14.110-0.el7.4pd.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.64GiB
Name: zhangjun-k8s01
ID: VJYK:3T6T:EPHU:65SM:3OZD:DMNE:MT5J:O22I:TCG2:F3JR:MZ76:B3EF
Docker Root Dir: /data/k8s/docker/data
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 22
 Goroutines: 43
 System Time: 2019-05-26T11:26:21.2494815+08:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 docker02:35000
 127.0.0.0/8
Registry Mirrors:
 https://docker.mirrors.ustc.edu.cn/
 https://hub-mirror.c.163.com/
Live Restore Enabled: true
Product License: Community Engine

WARNING: No swap limit support           

--network-plugin=cni \\
  --cni-conf-dir=/etc/cni/net.d \\
  --container-runtime=remote \\
  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \\           
systemctl restart kubelet
Docker
上一篇: 10.算法分析
下一篇: E. 部署 flannel 網絡E. 部署 flannel 網絡

繼續閱讀