天天看點
傳回

kubeadm部署K8S，使用containerd做運作時因為有節點是多網卡，是以需要在資源清單檔案中指定内網網卡

微信公衆号：運維開發故事，作者：喬克

前言

去年12月份，當Kubernetes社群宣布1.20版本之後會逐漸棄用

dockershim

，當時也有很多自媒體在宣傳Kubernetes棄用Docker。其實，我覺得這是一種誤導，也許僅僅是為了蹭熱度。

dockershim

是Kubernetes的一個元件，其作用是為了操作Docker。Docker是在2013年面世的，而Kubernetes是在2016年，是以Docker剛開始并沒有想到編排，也不會知道會出現Kubernetes這個龐然大物（它要是知道，也不會敗的那麼快…）。但是Kubernetes在建立的時候就是以Docker作為容器運作時，很多操作邏輯都是針對的Docker，随着社群越來越健壯，為了相容更多的容器運作時，才将Docker的相關邏輯獨立出來組成了

dockershim

。

正因為這樣，隻要Kubernetes的任何變動或者Docker的任何變動，都必須維護

dockershim

，這樣才能保證足夠的支援，但是通過

dockershim

操作Docker，其本質還是操作Docker的底層運作時

Containerd

，而且

Containerd

自身也是支援

CRI

（Container Runtime Interface），那為什麼還要繞一層Docker呢？是不是可以直接通過

CRI

和

Containerd

進行互動？這也是社群希望啟動

dockershim

的原因之一吧。

再來看看啟動

dockershim

究竟對使用者、對維護者有多少影響。

對上層使用者來說，其實并沒有影響，因為上層已經屏蔽調了這些細節，隻管用就可以了。更多的影響隻是針對我們這些YAML工程師，因為我們主要是考慮用哪個容器運作時，如果繼續用Docker，以後版本更新有沒有影響？如果不用Docker，維護的成本、複雜度、學習成本會不會增加？其實我們是想多了，事情也遠沒那麼複雜，喜歡用docker的依舊可以用docker，想用containerd的就用containerd，改動也不大，後面也有相關的部署文檔。而且也隻是kubernetes社群不再維護dockershim而已，Mirantis 和 Docker 已經決定之後共同合作維護 dockershim 元件，也就是說dockershim依然可以作為連接配接docker的橋梁，隻是從kubernetes内置攜帶改成獨立的而已。

那什麼是containerd呢？

Containerd是從Docker中分離的一個項目，旨在為Kubernetes提供容器運作時，負責管理鏡像和容器的生命周期。不過Containerd是可以抛開Docker獨立工作的。它的特性如下：

  • 支援OCI鏡像規範，也就是runc
  • 支援OCI運作時規範
  • 支援鏡像的pull
  • 支援容器網絡管理
  • 存儲支援多租戶
  • 支援容器運作時和容器的生命周期管理
  • 支援管理網絡名稱空間

Containerd和Docker在指令使用上的一些差別主要如下：

功能 Docker Containerd
顯示本地鏡像清單 docker images crictl images
下載下傳鏡像 docker pull crictl pull
上傳鏡像 docker push 無
删除本地鏡像 docker rmi crictl rmi
檢視鏡像詳情 docker inspect IMAGE-ID crictl inspecti IMAGE-ID
顯示容器清單 docker ps crictl ps
建立容器 docker create crictl create
啟動容器 docker start crictl start
停止容器 docker stop crictl stop
删除容器 docker rm crictl rm
檢視容器詳情 docker inspect crictl inspect
attach docker attach crictl attach
exec docker exec crictl exec
logs docker logs crictl logs
stats docker stats crictl stats

可以看到使用方式大同小異。

下面介紹一下使用kubeadm安裝K8S叢集，并使用containerd作為容器運作時的具體安裝步驟。

環境說明

主機節點

IP位址 系統 核心
192.168.0.5 CentOS7.6 3.10
192.168.0.125 CentOS7.6 3.10

軟體說明

軟體 版本
kubernetes 1.20.5
containerd 1.4.4

環境準備

（1）在每個節點上添加 hosts 資訊：

$ cat /etc/hosts

192.168.0.5 k8s-master
192.168.0.125 k8s-node01

（2）禁用防火牆：

$ systemctl stop firewalld
$ systemctl disable firewalld

（3）禁用SELINUX：

$ setenforce 0
$ cat /etc/selinux/config
SELINUX=disabled

（4）建立/etc/sysctl.d/k8s.conf檔案，添加如下内容：

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

（5）執行如下指令使修改生效：

$ modprobe br_netfilter
$ sysctl -p /etc/sysctl.d/k8s.conf

（6）安裝 ipvs

$ cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
$ chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

上面腳本建立了的/etc/sysconfig/modules/ipvs.modules檔案，保證在節點重新開機後能自動加載所需子產品。使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4指令檢視是否已經正确加載所需的核心子產品。

（7）安裝了 ipset 軟體包：

$ yum install ipset -y

為了便于檢視 ipvs 的代理規則，最好安裝一下管理工具 ipvsadm：

$ yum install ipvsadm -y

（8）同步伺服器時間

$ yum install chrony -y
$ systemctl enable chronyd
$ systemctl start chronyd
$ chronyc sources

（9）關閉 swap 分區：

$ swapoff -a

（10）修改/etc/fstab檔案，注釋掉 SWAP 的自動挂載，使用free -m确認 swap 已經關閉。swappiness 參數調整，修改/etc/sysctl.d/k8s.conf添加下面一行：

執行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。

（11）接下來可以安裝 Containerd

$ yum install -y yum-utils \
 device-mapper-persistent-data \
 lvm2
$ yum-config-manager \
 --add-repo \
 https://download.docker.com/linux/centos/docker-ce.repo
$ yum list | grep containerd

可以選擇安裝一個版本，比如我們這裡安裝最新版本：

$ yum install containerd.io-1.4.4 -y

（12）建立containerd配置檔案：

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# 替換配置檔案
sed -i "s#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g"  /etc/containerd/config.toml
sed -i '/containerd.runtimes.runc.options/a\ \ \ \ \ \ \ \ \ \ \ \ SystemdCgroup = true' /etc/containerd/config.toml
sed -i "s#https://registry-1.docker.io#https://registry.cn-hangzhou.aliyuncs.com#g"  /etc/containerd/config.toml

（13）啟動Containerd:

systemctl daemon-reload
systemctl enable containerd
systemctl restart containerd

在確定 Containerd安裝完成後，上面的相關環境配置也完成了，現在我們就可以來安裝 Kubeadm 了，我們這裡是通過指定yum 源的方式來進行安裝，使用阿裡雲的源進行安裝：

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
 http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

然後安裝 kubeadm、kubelet、kubectl（我安裝的是最新版，有版本要求自己設定版本）：

設定運作時：

$ crictl config runtime-endpoint /run/containerd/containerd.sock

可以看到我們這裡安裝的是 v1.20.5版本，然後将 kubelet 設定成開機啟動：

$ systemctl daemon-reload
$ systemctl enable kubelet && systemctl start kubelet
到這裡為止上面所有的操作都需要在所有節點執行配置。

**

初始化叢集

初始化Master

然後接下來在 master 節點配置 kubeadm 初始化檔案，可以通過如下指令導出預設的初始化配置：

$ kubeadm config print init-defaults > kubeadm.yaml

然後根據我們自己的需求修改配置，比如修改 imageRepository 的值，kube-proxy 的模式為 ipvs，需要注意的是由于我們使用的containerd作為運作時，是以在初始化節點的時候需要指定

cgroupDriver

為

systemd

 
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.0.5 
  bindPort: 6443
nodeRegistration:
  criSocket: /run/containerd/containerd.sock 
  name: k8s-master
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.20.5
networking:
  dnsDomain: cluster.local
  podSubnet: 172.16.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

然後使用上面的配置檔案進行初始化：

$ kubeadm init --config=kubeadm.yaml

[init] Using Kubernetes version: v1.20.5
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.5]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.5 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.5 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 70.001862 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.5:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:446623b965cdb0289c687e74af53f9e9c2063e854a42ee36be9aa249d3f0ccec

拷貝 kubeconfig 檔案

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

**

添加節點

記住初始化叢集上面的配置和操作要提前做好，将 master 節點上面的 $HOME/.kube/config 檔案拷貝到 node 節點對應的檔案中，安裝 kubeadm、kubelet、kubectl，然後執行上面初始化完成後提示的 join 指令即可：

# kubeadm join 192.168.0.5:6443 --token abcdef.0123456789abcdef \
>     --discovery-token-ca-cert-hash sha256:446623b965cdb0289c687e74af53f9e9c2063e854a42ee36be9aa249d3f0ccec 
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
如果忘記了上面的 join 指令可以使用指令kubeadm token create --print-join-command重新擷取。

執行成功後運作 get nodes 指令：

$ kubectl get no
NAME         STATUS     ROLES                  AGE   VERSION
k8s-master   NotReady   control-plane,master   29m   v1.20.5
k8s-node01   NotReady   <none>                 28m   v1.20.5

可以看到是 NotReady 狀态，這是因為還沒有安裝網絡插件，接下來安裝網絡插件，可以在文檔 https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ 中選擇我們自己的網絡插件，這裡我們安裝 calio:

$ wget https://docs.projectcalico.org/v3.8/manifests/calico.yaml

因為有節點是多網卡，是以需要在資源清單檔案中指定内網網卡

$ vi calico.yaml

......
spec:
 containers:
 - env:
 - name: DATASTORE_TYPE
   value: kubernetes
 - name: IP_AUTODETECTION_METHOD # DaemonSet中添加該環境變量
   value: interface=eth0 # 指定内網網卡
 - name: WAIT_FOR_DATASTORE
   value: "true"
- name: CALICO_IPV4POOL_CIDR # 由于在init的時候配置的172網段，是以這裡需要修改
  value: "172.16.0.0/16"

......

安裝calico網絡插件

$ kubectl apply -f calico.yaml

隔一會兒檢視 Pod 運作狀态：

# kubectl get pod -n kube-system 
NAME                                      READY   STATUS              RESTARTS   AGE
calico-kube-controllers-bcc6f659f-zmw8n   0/1     ContainerCreating   0          7m58s
calico-node-c4vv7                         1/1     Running             0          7m58s
calico-node-dtw7g                         0/1     PodInitializing     0          7m58s
coredns-54d67798b7-mrj2b                  1/1     Running             0          46m
coredns-54d67798b7-p667d                  1/1     Running             0          46m
etcd-k8s-master                           1/1     Running             0          46m
kube-apiserver-k8s-master                 1/1     Running             0          46m
kube-controller-manager-k8s-master        1/1     Running             0          46m
kube-proxy-clf4s                          1/1     Running             0          45m
kube-proxy-mt7tt                          1/1     Running             0          46m
kube-scheduler-k8s-master                 1/1     Running             0          46m

網絡插件運作成功了，node 狀态也正常了：

# kubectl get nodes 
NAME         STATUS   ROLES                  AGE   VERSION
k8s-master   Ready    control-plane,master   47m   v1.20.5
k8s-node01   Ready    <none>                 46m   v1.20.5

用同樣的方法添加另外一個節點即可。**

配置指令自動補全

yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

踩坑

在1.20版本以上，當使用nfs做存儲的時候，在建立PVC時，會報以下錯誤：

I0323 08:41:25.264754       1 controller.go:987] provision "default/test-nfs-pvc2" class "nfs-client-storageclass": started
E0323 08:41:25.267631       1 controller.go:1004] provision "default/test-nfs-pvc2" class "nfs-client-storageclass": unexpected error getting claim reference: selfLink was empty, can't make reference

這是因為kubernetes1.20.0廢棄了

selfLink

，解決辦法是重新加回來，如下在kube-apiserver.yaml中添加如下參數：

$ vim /etc/kubernetes/manifests/kube-apiserver.yaml
# 增加一行
- --feature-gates=RemoveSelfLink=false

然後重新apply以下使之生效：

kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml

參考文檔

【1】：https://github.com/containerd/containerd/issues/4857

【2】：https://github.com/containerd/containerd

【3】：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md

kubernetes 容器 Container 運維 Linux
上一篇: kubeadm部署K8S并使用containerd做運作時kubeadm部署K8S并使用containerd做運作時因為有節點是多網卡，是以需要在資源清單檔案中指定内網網卡
下一篇: containerd容器介紹部署與使用（版本V1.4.3）教你如何像使用docker一樣絲滑containerd容器介紹部署與使用（版本V1.4.3）教你如何像使用docker一樣絲滑

繼續閱讀