搭建Kubernetes 1.20.5叢集（二進制方式）

搭建Kubernetes叢集

此文以Kubernetes 1.20.5版本為例！

如未指定，下述指令在所有節點執行！

一、系統資源及元件規劃

節點名稱	系統名稱	CPU/記憶體	網卡	磁盤	IP位址	OS
Master	master	2C/4G	ens33	128G	192.168.0.10	CentOS7
Worker1	worker1	2C/4G	ens33	128G	192.168.0.11	CentOS7
Worker2	worker2	2C/4G	ens33	128G	192.168.0.12	CentOS7

二、系統軟體安裝與設定

1、安裝基本軟體

yum -y install vim git lrzsz wget net-tools bash-completion
           

2、設定名稱解析

echo 192.168.0.10 master >> /etc/hosts
echo 192.168.0.11 worker1 >> /etc/hosts
echo 192.168.0.12 worker2 >> /etc/hosts
           

3、設定NTP

yum -y install chrony
           

systemctl start chronyd
systemctl enable chronyd
systemctl status chronyd
           

chronyc sources

4、設定SELinux、防火牆

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
           

5、設定網橋

配置L2網橋在轉發包時會被iptables的FORWARD規則所過濾，CNI插件需要該配置

建立/etc/sysctl.d/k8s.conf檔案，添加如下内容：

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
           

執行指令，使修改生效：

modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf
           

6、設定swap

關閉系統swap分區：

swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak | grep -v swap > /etc/fstab
rm -rf /etc/fstab_bak
           

echo vm.swappiness = 0 >> /etc/sysctl.d/k8s.conf
sysctl -p /etc/sysctl.d/k8s.conf
           

7、設定ipvs

安裝ipvsadm ipset：

yum -y install ipvsadm ipset
           

建立ipvs設定腳本：

cat > /etc/sysconfig/modules/ipvs.modules << EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
           

執行腳本，驗證修改結果：

chmod 755 /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules
lsmod | grep -e ip_vs -e nf_conntrack_ipv4
           

三、Kubernetes叢集配置

1、部署證書生成工具

在Master節點上下載下傳證書生成工具：

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
           

2、部署ETCD叢集

在Master節點上建立配置目錄和證書目錄：

mkdir -p /etc/etcd/ssl/

在Master節點上建立CA CSR請求檔案：

cat > /etc/etcd/ssl/ca-csr.json << EOF
{
  "CN": "kubernetes",
  "key": {
      "algo": "rsa",
      "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "system"
    }
  ],
  "ca": {
          "expiry": "87600h"
  }
}
EOF
           

在Master節點上生成CA憑證：

cd /etc/etcd/ssl/
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ll ca*.pem
           

在Master節點上生成證書政策：

cat > /etc/etcd/ssl/ca-config.json << EOF
{
  "signing": {
      "default": {
          "expiry": "87600h"
        },
      "profiles": {
          "kubernetes": {
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "87600h"
          }
      }
  }
}
EOF
           

在Master節點上建立ETCD CSR請求檔案：

cat > /etc/etcd/ssl/etcd-csr.json << EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.0.10"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "BeiJing",
    "L": "BeiJing",
    "O": "k8s",
    "OU": "system"
  }]
}
EOF
           

在Master節點上生成ETCD證書：

cd /etc/etcd/ssl/
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
ll etcd*.pem
           

下載下傳ETCD二進制檔案：

參考位址：https://github.com/etcd-io/etcd/releases

下載下傳位址：https://github.com/etcd-io/etcd/releases/download/v3.4.15/etcd-v3.4.15-linux-amd64.tar.gz

在Master節點上解壓ETCD二進制檔案至系統目錄：

tar -xf /root/etcd-v3.4.15-linux-amd64.tar.gz -C /root/
mv /root/etcd-v3.4.15-linux-amd64/{etcd,etcdctl} /usr/local/bin/
           

在Master節點上建立ETCD配置檔案：

cat > /etc/etcd/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.0.10:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.10:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.10:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.10:2379"
ETCD_INITIAL_CLUSTER="etcd=https://192.168.0.10:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
           

ETCD配置檔案參數說明：

ETCD_NAME：節點名稱，叢集中唯一

ETCD_DATA_DIR：資料目錄

ETCD_LISTEN_PEER_URLS：叢集通信監聽位址

ETCD_LISTEN_CLIENT_URLS：用戶端通路監聽位址

ETCD_INITIAL_ADVERTISE_PEER_URLS：叢集通告位址

ETCD_ADVERTISE_CLIENT_URLS：用戶端通告位址

ETCD_INITIAL_CLUSTER：叢集節點位址

ETCD_INITIAL_CLUSTER_TOKEN：叢集Token

ETCD_INITIAL_CLUSTER_STATE：加入叢集的目前狀态，new是新叢集，existing表示加入已有叢集

在Master節點上配置systemd管理ETCD：

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd \\
--cert-file=/etc/etcd/ssl/etcd.pem \\
--key-file=/etc/etcd/ssl/etcd-key.pem \\
--trusted-ca-file=/etc/etcd/ssl/ca.pem \\
--peer-cert-file=/etc/etcd/ssl/etcd.pem \\
--peer-key-file=/etc/etcd/ssl/etcd-key.pem \\
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \\
--peer-client-cert-auth \\
--client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
           

在Master節點上啟動ETCD，并設定自啟動：

systemctl start etcd
systemctl enable etcd
systemctl status etcd
           

在Master節點上檢視ETCD叢集狀态：

ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.0.10:2379 endpoint health
           

3、部署Docker

下載下傳Docker二進制檔案：

參考位址：https://download.docker.com/linux/static/stable/x86_64/

下載下傳位址：https://download.docker.com/linux/static/stable/x86_64/docker-20.10.5.tgz

解壓Docker二進制檔案至系統目錄：

tar -xf /root/docker-20.10.5.tgz -C /root/
mv /root/docker/* /usr/local/bin
           

配置systemd管理Docker：

cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/local/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target
EOF
           

啟動Docker，并設定自啟動：

systemctl start docker
systemctl enable docker
systemctl status docker
           

配置Docker鏡像源和Cgroup驅動：

cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://7y88q662.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
systemctl restart docker
docker info | grep "Cgroup Driver"
           

4、部署Master節點

在Master節點上下載下傳Kubernetes二進制檔案：

參考位址：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md

下載下傳位址：https://dl.k8s.io/v1.20.5/kubernetes-server-linux-amd64.tar.gz

解壓Kubernetes二進制檔案至系統目錄：

tar -xf /root/kubernetes-server-linux-amd64.tar.gz -C /root/
cp
/root/kubernetes/server/bin/{kubectl,kube-apiserver,kube-scheduler,kube-controller-manager}
/usr/local/bin/
           

在所有節點上建立配置目錄和證書目錄：

mkdir -p /etc/kubernetes/ssl/
           

在Master節點上建立日志目錄：

mkdir /var/log/kubernetes/
           

在Master節點上建立CA CSR請求檔案：

cat > /etc/kubernetes/ssl/ca-csr.json << EOF
{
  "CN": "kubernetes",
  "key": {
      "algo": "rsa",
      "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "system"
    }
  ],
  "ca": {
          "expiry": "87600h"
  }
}
EOF
           

在Master節點上生成CA憑證：

cd /etc/kubernetes/ssl/
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ll ca*.pem
           

将Master上CA憑證拷貝至Worker節點：

scp /etc/kubernetes/ssl/ca.pem [email protected]:/etc/kubernetes/ssl/
scp /etc/kubernetes/ssl/ca.pem [email protected]:/etc/kubernetes/ssl/
           

在Master節點上部署Kubernetes：

在Master節點上生成證書政策：

cat > /etc/kubernetes/ssl/ca-config.json << EOF
{
  "signing": {
      "default": {
          "expiry": "87600h"
        },
      "profiles": {
          "kubernetes": {
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "87600h"
          }
      }
  }
}
EOF
           

在Master節點上建立kube-apiserver CSR請求檔案：

cat > /etc/kubernetes/ssl/kube-apiserver-csr.json << EOF
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.0.10",
    "192.168.0.11",
    "192.168.0.12",
    "10.96.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
EOF
           

在Master節點上生成kube-apiserver證書：

cd /etc/kubernetes/ssl/
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
ll kube-apiserver*.pem
           

在Master節點上生成token：

cat > /etc/kubernetes/token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
           

在Master節點上建立kube-apiserver配置檔案：

cat > /etc/kubernetes/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
--anonymous-auth=false \\
--bind-address=192.168.0.10 \\
--secure-port=6443 \\
--advertise-address=192.168.0.10 \\
--insecure-port=0 \\
--authorization-mode=Node,RBAC \\
--runtime-config=api/all=true \\
--enable-bootstrap-token-auth \\
--service-cluster-ip-range=10.96.0.0/24 \\
--token-auth-file=/etc/kubernetes/token.csv \\
--service-node-port-range=30000-50000 \\
--tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \\
--tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \\
--client-ca-file=/etc/kubernetes/ssl/ca.pem \\
--kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \\
--kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \\
--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \\
--service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \\
--service-account-issuer=https://kubernetes.default.svc.cluster.local \\
--etcd-cafile=/etc/etcd/ssl/ca.pem \\
--etcd-certfile=/etc/etcd/ssl/etcd.pem \\
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \\
--etcd-servers=https://192.168.0.10:2379 \\
--enable-swagger-ui=true \\
--allow-privileged=true \\
--apiserver-count=1 \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/var/log/kube-apiserver-audit.log \\
--event-ttl=1h \\
--alsologtostderr=true \\
--logtostderr=false \\
--log-dir=/var/log/kubernetes \\
--v=2"
EOF
           

在Master節點上配置systemd管理kube-apiserver：

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service

[Service]
EnvironmentFile=/etc/kubernetes/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
           

在Master節點上啟動kube-apiserver，并設定自啟動：

systemctl start kube-apiserver
systemctl enable kube-apiserver
systemctl status kube-apiserver
           

在Master節點上部署kubectl元件：

在Master節點上建立kubectl CSR請求檔案：

cat > /etc/kubernetes/ssl/admin-csr.json << EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "system:masters",
      "OU": "system"
    }
  ]
}
EOF
           

在Master節點上生成kubectl證書：

cd /etc/kubernetes/ssl/
cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubernetes/ssl/ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
ll admin*.pem
           

在Master節點上生成kube.config：

設定叢集參數：

kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=https://192.168.0.10:6443 --kubeconfig=/etc/kubernetes/kube.config
           

設定用戶端認證參數：

kubectl config set-credentials admin --client-certificate=/etc/kubernetes/ssl/admin.pem --client-key=/etc/kubernetes/ssl/admin-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/kube.config
           

設定上下文參數：

kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=/etc/kubernetes/kube.config
           

設定預設上下文：

kubectl config use-context kubernetes --kubeconfig=/etc/kubernetes/kube.config
           

将kube.config拷貝至預設目錄：

mkdir ~/.kube
cp /etc/kubernetes/kube.config ~/.kube/config
           

授權kubernetes證書通路kubelet api權限：

kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes
           

此時可以通過kubectl管理叢集。

在Master節點上部署kube-controller-manager：

在Master節點上建立kube-controller-manager CSR請求檔案：

cat > /etc/kubernetes/ssl/kube-controller-manager-csr.json << EOF
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.0.10"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "BeiJing",
        "L": "BeiJing",
        "O": "system:kube-controller-manager",
        "OU": "system"
      }
    ]
}
EOF
           

在Master節點上生成kube-controller-manager證書：

cd /etc/kubernetes/ssl/
cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubernetes/ssl/ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
ll kube-controller-manage*.pem
           

在Master節點上生成kube-controller-manager.kubeconfig：

設定叢集參數：

kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=https://192.168.0.10:6443 --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig
           

設定用戶端認證參數：

kubectl config set-credentials system:kube-controller-manager --client-certificate=/etc/kubernetes/ssl/kube-controller-manager.pem --client-key=/etc/kubernetes/ssl/kube-controller-manager-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig
           

設定上下文參數：

kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig
           

設定預設上下文：

kubectl config use-context system:kube-controller-manager --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig
           

在Master節點上建立kube-controller-manager配置檔案：

cat > /etc/kubernetes/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
--service-cluster-ip-range=10.96.0.0/24 \\
--cluster-name=kubernetes \\
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--experimental-cluster-signing-duration=87600h \\
--root-ca-file=/etc/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \\
--leader-elect=true \\
--feature-gates=RotateKubeletServerCertificate=true \\
--controllers=*,bootstrapsigner,tokencleaner \\
--horizontal-pod-autoscaler-use-rest-clients=true \\
--horizontal-pod-autoscaler-sync-period=10s \\
--tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \\
--tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \\
--use-service-account-credentials=true \\
--alsologtostderr=true \\
--logtostderr=false \\
--log-dir=/var/log/kubernetes \\
--v=2"
EOF
           

在Master節點上配置systemd管理kube-controller-manager：

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/etc/kubernetes/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
           

在Master節點上啟動kube-controller-manager，并設定自啟動：

systemctl start kube-controller-manager
systemctl enable kube-controller-manager
systemctl status kube-controller-manager
           

在Master節點上部署kube-scheduler元件：

在Master節點上建立kube-scheduler CSR請求檔案：

cat > /etc/kubernetes/ssl/kube-scheduler-csr.json << EOF
{
    "CN": "system:kube-scheduler",
    "hosts": [
      "127.0.0.1",
      "192.168.0.10"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "BeiJing",
        "L": "BeiJing",
        "O": "system:kube-scheduler",
        "OU": "system"
      }
    ]
}
EOF
           

在Master節點上生成kube-scheduler證書：

cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubernetes/ssl/ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
ll kube-scheduler*.pem
           

在Master節點上生成kube-scheduler.kubeconfig：

設定叢集參數：

kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=https://192.168.0.10:6443 --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig
           

設定用戶端認證參數：

kubectl config set-credentials system:kube-scheduler --client-certificate=/etc/kubernetes/ssl/kube-scheduler.pem --client-key=/etc/kubernetes/ssl/kube-scheduler-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig
           

設定上下文參數：

kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig
           

設定預設上下文：

kubectl config use-context system:kube-scheduler --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig
           

在Master節點上建立kube-scheduler配置檔案：

cat > /etc/kubernetes/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--address=127.0.0.1 \\
--kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\
--leader-elect=true \\
--alsologtostderr=true \\
--logtostderr=false \\
--log-dir=/var/log/kubernetes \\
--v=2"
EOF
           

在Master節點上配置systemd管理kube-scheduler：

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/etc/kubernetes/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
           

在Master節點上啟動kube-scheduler，并設定自啟動：

systemctl start kube-scheduler
systemctl enable kube-scheduler
systemctl status kube-scheduler
           

5、部署Worker節點

Master節點同時作為Worker節點，需配置kubelet和kube-proxy。

在所有節點上部署kubelet元件：

在Master節點上将Kubernetes二進制檔案至系統目錄：

cp /root/kubernetes/server/bin/kubelet /usr/local/bin
           

在Master節點上Kubernetes二進制檔案拷貝至Worker節點：

scp /root/kubernetes/server/bin/kubelet [email protected]:/usr/local/bin/
scp /root/kubernetes/server/bin/kubelet [email protected]:/usr/local/bin/
           

在Master節點上生成kubelet-bootstrapr.kubeconfig：

設定叢集參數：

kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=https://192.168.0.10:6443 --kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig
           

設定用戶端認證參數：

BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv)
kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig
           

設定上下文參數：

kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig
           

設定預設上下文：

kubectl config use-context default --kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig
           

建立角色綁定：

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
           

在Master節點上kubelet-bootstrap.kubeconfig檔案拷貝至Worker節點：

scp /etc/kubernetes/kubelet-bootstrap.kubeconfig [email protected]:/etc/kubernetes/
scp /etc/kubernetes/kubelet-bootstrap.kubeconfig [email protected]:/etc/kubernetes/
           

在Worker節點上建立日志目錄：

mkdir /var/log/kubernetes/
           

在所有節點上建立kubelet配置檔案：

cat > /etc/kubernetes/kubelet.yaml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  x509:
    clientCAFile: /etc/kubernetes/ssl/ca.pem
  webhook:
    enabled: true
    cacheTTL: 2m0s
  anonymous:
    enabled: false
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
address: 192.168.0.10
port: 10250
readOnlyPort: 10255
cgroupDriver: systemd
hairpinMode: promiscuous-bridge
serializeImagePulls: false
featureGates:
  RotateKubeletClientCertificate: true
  RotateKubeletServerCertificate: true
clusterDomain: cluster.local.
clusterDNS:
  - 10.96.0.2
EOF
           

标紅部分修改為節點實際IP位址。

在所有節點上配置systemd管理kubelet：

cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \\
--cert-dir=/etc/kubernetes/ssl \\
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
--config=/etc/kubernetes/kubelet.yaml \\
--network-plugin=cni \\
--pod-infra-container-image=k8s.gcr.io/pause:3.2 \\
--alsologtostderr=true \\
--logtostderr=false \\
--log-dir=/var/log/kubernetes \\
--v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
           

k8s.gcr.io/pause:3.2無法直接下載下傳，需通過阿裡雲鏡像倉庫下載下傳：

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 k8s.gcr.io/pause:3.2
docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
           

在所有節點上啟動kubelet，并設定自啟動：

systemctl start kubelet
systemctl enable kubelet
systemctl status kubelet
           

在Master節點上檢視kubelet證書申請：

kubectl get csr
           

在Master節點上準許kubelet證書申請：

kubectl certificate approve node-csr-xxx
           

在Master節點上檢視Node狀态：

kubectl get node
           

在所有節點上部署kube-proxy元件：

在Master節點上将Kubernetes二進制檔案至系統目錄：

cp /root/kubernetes/server/bin/kube-proxy /usr/local/bin
           

在Master節點上Kubernetes二進制檔案拷貝至Worker節點：

scp /root/kubernetes/server/bin/kube-proxy [email protected]:/usr/local/bin/
scp /root/kubernetes/server/bin/kube-proxy [email protected]:/usr/local/bin/
           

在Master節點上建立kube-proxy CSR請求檔案：

cat > /etc/kubernetes/ssl/kube-proxy-csr.json << EOF
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
EOF
           

在Master節點上生成kube-proxy證書：

cd /etc/kubernetes/ssl/
cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubernetes/ssl/ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
ll kube-proxy*.pem
           

在Master節點上生成kube-proxy.kubeconfig：

設定叢集參數：

kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=https://192.168.0.10:6443 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
           

設定用戶端認證參數：

kubectl config set-credentials kube-proxy --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
           

設定上下文參數：

kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
           

設定預設上下文：

kubectl config use-context default --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
           

在Master節點上kube-proxy.kubeconfig檔案拷貝至Worker節點：

scp /etc/kubernetes/kube-proxy.kubeconfig [email protected]:/etc/kubernetes/
scp /etc/kubernetes/kube-proxy.kubeconfig [email protected]:/etc/kubernetes/
           

在所有節點上kube-proxy配置檔案：

cat > /etc/kubernetes/kube-proxy.yaml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.0.10
clientConnection:
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 10.244.0.0/16
healthzBindAddress: 192.168.0.10:10256
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.0.10:10249
mode: "ipvs"
EOF
           

标紅部分修改為節點實際IP位址。

在所有節點上配置systemd管理kube-proxy：

cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \\
--config=/etc/kubernetes/kube-proxy.yaml \\
--alsologtostderr=true \\
--logtostderr=false \\
--log-dir=/var/log/kubernetes \\
--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
           

在所有節點上啟動kube-apiserver，并設定自啟動：

systemctl start kube-proxy
systemctl enable kube-proxy
systemctl status kube-proxy
           

6、部署CNI網絡

在Master節點上部署CNI網絡：

下載下傳calico部署檔案：

下載下傳位址：https://docs.projectcalico.org/manifests/calico.yaml

在Master節點上修改calico.yaml：

增加

- name: CALICO_IPV4POOL_CIDR
  value: "10.244.0.0/16"
- name: IP_AUTODETECTION_METHOD
  value: "interen.*"
           

calico.yaml中的CIDR需與初始化叢集中的參數一緻。

在Master節點上部署CNI網絡：

kubectl apply -f calico.yaml
           

在Master節點上檢視Pod狀态：

kubectl get pod -o wide -n kube-system
           

在Master節點上檢視Node狀态：

kubectl get node
           

7、部署CoreDNS

在Master節點上解壓kubernetes-src.tar.gz檔案：

tar -xf /root/kubernetes/kubernetes-src.tar.gz -C /root/kubernetes/
           

在Master節點上修改/root/kubernetes/cluster/addons/dns/coredns/transforms2sed.sed檔案中$DNS_SERVER_IP、$DNS_DOMAIN、$DNS_MEMORY_LIMIT參數，如下：

s/__DNS__SERVER__/10.96.0.2/g
s/__DNS__DOMAIN__/cluster.local/g
s/__CLUSTER_CIDR__/$SERVICE_CLUSTER_IP_RANGE/g
s/__DNS__MEMORY__LIMIT__/200Mi/g
s/__MACHINE_GENERATED_WARNING__/Warning: This is a file generated from the base underscore template file: __SOURCE_FILENAME__/g
           

在Master節點上使用模闆檔案生成CoreDNS配置檔案coredns.yaml：

cd /root/kubernetes/cluster/addons/dns/coredns/
           

sed -f transforms2sed.sed coredns.yaml.base > coredns.yaml
           

在Master節點上修改CoreDNS配置檔案coredns.yaml

修改image部分參數

image: coredns/coredns:1.7.0
           

删除capabilities部分：

capabilities:
  add:
  - NET_BIND_SERVICE
  drop:
  - all
           

在Master節點上部署CoreDNS：

kubectl apply -f coredns.yaml
           

在Master節點上檢視Pod狀态：

kubectl get pod -o wide -n kube-system