之前接觸的 Kubernetes 叢集部署工具大多數都是依賴于 ssh 連接配接到待部署的節點上進行部署操作，這樣就要求部署前需要提前準備好叢集節點，且要保證這些節點的網絡互通以及時鐘同步等問題。類似于 kubespray 或者 kubekey 這些部署工具是不會去管這些底層的 IaaS 資源的建立，是要自己提前準備好。但是在一些企業私有雲環境中，使用了如 VMware vShpere[1] 或 OpenStack[2] 這些虛拟化平台，是可以将 K8s 叢集部署與 IaaS 資源建立這兩步統一起來的，這樣就可以避免手動建立和配置虛拟機這些繁瑣的步驟。
目前将 IaaS 資源建立與 K8s 叢集部署結合起來也有比較成熟的方案，比如基于 cluster-api[3] 項目的 tanzu[4] 。本文就以 VMware Tanzu 社群版[5] 為例在一台實體伺服器上，從安裝 ESXi OS 到部署完成 Tanzu Workload 叢集，來體驗一下這種部署方案的與衆不同之處。
部署流程
- 下載下傳依賴檔案
- 安裝 govc 依賴
- 安裝 ESXi OS
- 安裝 vCenter
- 配置 vCenter
- 建立 bootstrap 虛拟機
- 初始化 bootstrap 節點
- 部署 Tanzu Manager 叢集
- 部署 Tanzu Workload 叢集
勸退三連 😂
- 需要有一個 VMware 的賬戶[6] 用于下載下傳一些 ISO 鏡像和虛拟機模版;
- 需要有一台實體伺服器，推薦最低配置 8C 32G，至少 256GB 存儲；
- 需要一台 DHCP 伺服器，由于預設是使用 DHCP 擷取 IP 來配置設定給虛拟機的，是以 ESXi 所在的 VM Network 網絡中必須有一台 DHCP 伺服器用于給虛拟機配置設定 IP；
下載下傳依賴檔案
整個部署流程所需要的依檔案賴如下，可以先将這些依賴下載下傳到本地的機器上，友善後續使用。
[email protected]:/root/tanzu # tree -sh
.
├── [ 12M] govc_Linux_x86_64.tar.gz
├── [ 895M] photon-3-kube-v1.21.2+vmware.1-tkg.2-12816990095845873721.ova
├── [ 225M] photon-ova-4.0-c001795b80.ova
├── [ 170M] tce-linux-amd64-v0.9.1.tar.gz
├── [ 9.0G] VMware-VCSA-all-7.0.3-18778458.iso
└── [ 390M] VMware-VMvisor-Installer-7.0U2a-17867351.x86_64.iso
| 檔案 | 用途 | 下載下傳方式 |
|---|---|---|
| VMware-VMvisor-Installer-7.0U2a-17867351.x86_64.iso[7] | 安裝 ESXi OS | VMware 需賬戶 |
| VMware-VCSA-all-7.0.3-19234570.iso[8] | 安裝 vCenter | VMware 需賬戶 |
| photon-ova-4.0-c001795b80.ova[9] | bootstrap 節點 | VMware |
| photon-3-kube-v1.21.2+vmware.1-tkg.2-12816990095845873721.ova[10] | tanzu 叢集節點 | VMware 需賬戶 |
| tce-linux-amd64-v0.9.1.tar.gz[11] | tanzu 社群版 | GitHub release |
| govc_Linux_x86_64.tar.gz[12] | 安裝/配置 vCenter | GitHub release |
注意 ESXi 和 vCenter 的版本最好是 7.0 及以上，我隻在 ESXi 7.0.2 和 vCenter 7.0.3 上測試過，其他版本可能會有些差異；另外 ESXi 的版本不建議使用最新的 7.0.3，因為有比較嚴重的 bug，官方也建議使用者生産環境不要使用該版本了 vSphere 7.0 Update 3 Critical Known Issues - Workarounds & Fix (86287)[13] 。
安裝 govc 及依賴
在本地機器上安裝好 govc 和 jq，這兩個工具後面在配置 vCenter 的時候會用到。
- macOS
$ brew install govc jq
- Debian/Ubuntu
$ tar -xf govc_Linux_x86_64.tar.gz -C /usr/local/bin
$ apt install jq -y
- 其他 Linux 可以在 govc 和 jq 的 GitHub 上下載下傳對應的安裝檔案進行安裝。
安裝 ESXi OS
ESXi OS 的安裝網上有很多教程，沒有太多值得講解的地方，是以就參照一下其他大佬寫的部落格或者官方的安裝文檔 VMware ESXi 安裝和設定[14] 來就行；需要注意一點，ESXi OS 安裝時 VMFSL 分區将會占用大量的存儲空間，這将會使得 ESXi OS 安裝所在的磁盤最終建立出來的 datastore 比預期小很多，而且這個 VMFSL 分區在安裝好之後就很難再做調整了。是以如果磁盤存儲空間比較緊張，在安裝 ESXi OS 之前可以考慮下如何去掉這個分區；或者和我一樣将 ESXI OS 安裝在了一個 16G 的 USB Dom 盤上，不過生産環境不建議采用這種方案 😂（其實個人覺着安裝在 U 盤上問題不大，ESXi OS 啟動之後是加載到記憶體中運作的，不會對 U 盤有大量的讀寫操作，隻不過在機房中 U 盤被人不小心拔走就涼了。
- 設定 govc 環境變量
# ESXi 節點的 IP
export ESXI_IP="192.168.18.47"
# ESXi 登入的使用者名，初次安裝後預設為 root
export GOVC_USERNAME="root"
# 在 ESXi 安裝時設定的 root 密碼
export GOVC_PASSWORD="[email protected]"
# 允許不安全的 SSL 連接配接
export GOVC_INSECURE=true
export GOVC_URL="https://${ESXI_IP}"
export GOVC_DATASTORE=datastore1
- 測試 govc 是否能正常連接配接 ESXi 主機
Name: localhost.local
Path: /ha-datacenter/host/localhost/localhost
Manufacturer: Dell
Logical CPUs: 20 CPUs @ 2394MHz
Processor type: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz
CPU usage: 579 MHz (1.2%)
Memory: 261765MB
Memory usage: 16457 MB (6.3%)
Boot time: 2022-02-02 11:53:59.630124 +0000 UTC
State: connected
安裝 vCenter
按照 VMware 官方的 vCenter 安裝文檔 關于 vCenter Server 安裝和設定[15] 來安裝實在是過于繁瑣，其實官方的 ISO 安裝方式無非是運作一個 installer web 服務，然後在浏覽器上配置好 vCenter 虛拟機的參數，再将填寫的配置資訊在部署 vcsa 虛拟機的時候注入到 ova 的配置參數中。
知道這個安裝過程的原理之後我們也可以自己配置 vCenter 的參數資訊，然後通過 govc 來部署 ova；這比使用 UI 的方式簡單友善很多，最終隻需要填寫一個配置檔案，一條指令就可以部署完成啦。
- 首先是挂載 vCenter 的 ISO，找到 vcsa ova 檔案，它是 vCenter 虛拟機的模版
$ mount -o loop VMware-VCSA-all-7.0.3-18778458.iso /mnt
$ ls /mnt/vcsa/VMware-vCenter-Server-Appliance-7.0.3.00100-18778458_OVF10.ova
/mnt/vcsa/VMware-vCenter-Server-Appliance-7.0.3.00100-18778458_OVF10.ova
- 根據自己的環境資訊修改下面安裝腳本中的相關配置：
#!/usr/bin/env bash
VCSA_OVA_FILE=$1
set -o errexit
set -o nounset
set -o pipefail
# ESXi 的 IP 位址
export ESXI_IP="192.168.18.47"
# ESXi 的使用者名
export GOVC_USERNAME="root"
# ESXI 的密碼
export GOVC_PASSWORD="[email protected]"
# 安裝 vCenter 虛拟機使用的 datastore 名稱
export GOVC_DATASTORE=datastore1
export GOVC_INSECURE=true
export GOVC_URL="https://${ESXI_IP}"
# vCenter 的登入密碼
VM_PASSWORD="[email protected]"
# vCenter 的 IP 位址
VM_IP=192.168.20.92
# vCenter 虛拟機的名稱
VM_NAME=vCenter-Server-Appliance
# vCenter 虛拟機使用的網絡
VM_NETWORK="VM Network"
# DNS 伺服器
VM_DNS="223.6.6.6"
# NTP 伺服器
VM_NTP="0.pool.ntp.org"
deploy_vcsa_vm(){
config=$(govc host.info -k -json | jq -r '.HostSystems[].Config')
gateway=$(jq -r '.Network.IpRouteConfig.DefaultGateway' <<<"$config")
route=$(jq -r '.Network.RouteTableInfo.IpRoute[] | select(.DeviceName == "vmk0") | select(.Gateway == "0.0.0.0")' <<<"$config")
prefix=$(jq -r '.PrefixLength' <<<"$route")
opts=(
cis.vmdir.password=${VM_PASSWORD}
cis.appliance.root.passwd=${VM_PASSWORD}
cis.appliance.root.shell=/bin/bash
cis.deployment.node.type=embedded
cis.vmdir.domain-name=vsphere.local
cis.vmdir.site-name=VCSA
cis.appliance.net.addr.family=ipv4
cis.appliance.ssh.enabled=True
cis.ceip_enabled=False
cis.deployment.autoconfig=True
cis.appliance.net.addr=${VM_IP}
cis.appliance.net.prefix=${prefix}
cis.appliance.net.dns.servers=${VM_DNS}
cis.appliance.net.gateway=$gateway
cis.appliance.ntp.servers="${VM_NTP}"
cis.appliance.net.mode=static
)
props=$(printf -- "guestinfo.%s\n" "${opts[@]}" | jq --slurp -R 'split("\n") | map(select(. != "")) | map(split("=")) | map({"Key": .[0], "Value": .[1]})')
cat <<EOF | govc import.${VCSA_OVA_FILE##*.} -options - "${VCSA_OVA_FILE}"
{
"Name": "${VM_NAME}",
"Deployment": "tiny",
"DiskProvisioning": "thin",
"IPProtocol": "IPv4",
"Annotation": "VMware vCenter Server Appliance",
"PowerOn": false,
"WaitForIP": false,
"InjectOvfEnv": true,
"NetworkMapping": [
{
"Name": "Network 1",
"Network": "${VM_NETWORK}"
}
],
"PropertyMapping": ${props}
}
EOF
}
deploy_vcsa_vm
govc vm.change -vm "${VM_NAME}" -g vmwarePhoton64Guest
govc vm.power -on "${VM_NAME}"
govc vm.ip -a "${VM_NAME}"
- 通過腳本安裝 vCenter，指定第一參數為 OVA 的絕對路徑。運作完後将會自動将 ova 導入到 vCenter，并啟動虛拟機；
# 執行該腳本，第一個參數傳入 vCenter ISO 中 vcsa ova 檔案的絕對路徑
$ bash install-vcsa.sh /mnt/vcsa/VMware-vCenter-Server-Appliance-7.0.3.00100-18778458_OVF10.ova
[03-02-22 18:40:19] Uploading VMware-vCenter-Server-Appliance-7.0.3.00100-18778458_OVF10-disk1.vmdk... OK
[03-02-22 18:41:09] Uploading VMware-vCenter-Server-Appliance-7.0.3.00100-18778458_OVF10-disk2.vmdk... (29%, 52.5MiB/s)
[03-02-22 18:43:08] Uploading VMware-vCenter-Server-Appliance-7.0.3.00100-18778458_OVF10-disk2.vmdk... OK
[03-02-22 18:43:08] Injecting OVF environment...
Powering on VirtualMachine:3... OK
fe80::20c:29ff:fe03:2f80
- 設定 vCenter 登入的環境變量，我們使用 govc 來配置 vCenter，通過浏覽器 Web UI 的方式配置起來效率有點低，不如 govc 指令一把梭友善 😂
export GOVC_URL="https://192.168.20.92"
export GOVC_USERNAME="[email protected]"
export GOVC_PASSWORD="[email protected]"
export GOVC_INSECURE=true
export GOVC_DATASTORE=datastore1
- 虛拟機啟動後将自動進行 vCenter 的安裝配置，等待一段時間 vCenter 安裝好之後，使用 govc about 檢視 vCenter 的資訊，如果能正确或管道說明 vCenter 就安裝好了；
$ govc about
FullName: VMware vCenter Server 7.0.3 build-18778458
Name: VMware vCenter Server
Vendor: VMware, Inc.
Version: 7.0.3
Build: 18778458
OS type: linux-x64
API type: VirtualCenter
API version: 7.0.3.0
Product ID: vpx
UUID: 0b49e119-e38f-4fbc-84a8-d7a0e548027d
配置 vCenter
這一步驟主要是配置 vCenter：建立 Datacenter、cluster、folder 等資源，并将 ESXi 主機添加到 cluster 中；
- 配置 vCenter
# 建立 Datacenter 資料中心
$ govc datacenter.create SH-IDC
# 建立 Cluster 叢集
$ govc cluster.create -dc=SH-IDC Tanzu-Cluster
# 将 ESXi 主機添加到 Cluster 當中
$ govc cluster.add -dc=SH-IDC -cluster=Tanzu-Cluster -hostname=192.168.18.47 --username=root -password='[email protected]' -noverify
# 建立 folder，用于将 Tanzu 的節點虛拟機存放到該檔案夾下
$ govc folder.create /SH-IDC/vm/Tanzu-node
# 導入 tanzu 汲取節點的虛拟機 ova 模版
$ govc import.ova -dc='SH-IDC' -ds='datastore1' photon-3-kube-v1.21.2+vmware.1-tkg.2-12816990095845873721.ova
# 将虛拟機轉換為模版，後續 tanzu 叢集将以該模版建立虛拟機
$ govc vm.markastemplate photon-3-kube-v1.21.2
初始化 bootstrap 節點
bootstrap 節點節點是用于運作 tanzu 部署工具的節點，官方是支援 Linux/macOS/Windows 三種作業系統的，但有一些比較嚴格的要求：
| Arch: x86; ARM is currently unsupported |
|---|
| RAM: 6 GB |
| CPU: 2 |
| Docker[16] Add your non-root user account to the docker user group. Create the group if it does not already exist. This lets the Tanzu CLI access the Docker socket, which is owned by the root user. For more information, see steps 1 to 4 in the Manage Docker as a non-root user[17] procedure in the Docker documentation. |
| Kubectl[18] |
| Latest version of Chrome, Firefox, Safari, Internet Explorer, or Edge |
| System time is synchronized with a Network Time Protocol (NTP) server. |
| Ensure your bootstrap machine is using cgroup v1[19]. For more information, see Check and set the cgroup[20]. |
在這裡為了避免這些麻煩的配置，我就直接使用的 VMware 官方的 Photon OS 4.0 Rev2[21] ，下載下傳 OVA 格式的鏡像直接導入到 ESXi 主機啟動一台虛拟機即可，能節省不少麻煩的配置；還有一個好處就是在一台單獨的虛拟機上運作 tanzu 部署工具不會污染本地的開發環境。
$ wget https://packages.vmware.com/photon/4.0/Rev2/ova/photon-ova-4.0-c001795b80.ova
# 導入 OVA 虛拟機模版
$ govc import.ova -ds='datastore1' -name bootstrap-node photon-ova-4.0-c001795b80.ova
# 修改一下虛拟機的配置，調整為 4C8G
$ govc vm.change -c 4 -m 8192 -vm bootstrap-node
# 開啟虛拟機
$ govc vm.power -on bootstrap-node
# 檢視虛拟機擷取到的 IPv4 位址
$ govc vm.ip -a -wait 1m bootstrap-node
$ ssh [email protected]
# 密碼預設為 changeme，輸入完密碼之後提示在輸入一遍 changeme，然後再修改新的密碼
[email protected] [ ~ ]# cat /etc/os-release
NAME="VMware Photon OS"
VERSION="4.0"
ID=photon
VERSION_ID=4.0
PRETTY_NAME="VMware Photon OS/Linux"
ANSI_COLOR="1;34"
HOME_URL="https://vmware.github.io/photon/"
BUG_REPORT_URL="https://github.com/vmware/photon/issues"
- 安裝部署時需要的一些工具（切，Photon OS 裡竟然連個 tar 指令都沒有 😠
[email protected] [ ~ ]# tdnf install sudo tar -y
[email protected] [ ~ ]# curl -LO https://dl.k8s.io/release/v1.21.2/bin/linux/amd64/kubectl
[email protected] [ ~ ]# sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
- 啟動 docker，bootstrap 節點會以 kind 的方式運作一個 K8s 叢集，需要用到 docker。雖然可以使用外部的 k8s 叢集，但不是很推薦，因為 cluster-api 依賴 k8s 的版本，不能太高也不能太低；
[email protected] [ ~ ]# systemctl enable docker --now
- 從 vmware-tanzu/community-edition[22] 下載下傳 tanzu 社群版的安裝包，然後解壓後安裝；
[email protected] [ ~ ]# curl -LO https://github.com/vmware-tanzu/community-edition/releases/download/v0.9.1/tce-linux-amd64-v0.9.1.tar.gz
[email protected] [ ~ ]# tar -xf tce-linux-amd64-v0.9.1.tar.gz
[email protected] [ ~ ]# cd tce-linux-amd64-v0.9.1/
[email protected] [ ~ ]# bash install.sh
然而不幸地翻車了， install.sh 腳本中禁止 root 使用者運作
+ ALLOW_INSTALL_AS_ROOT=
+ [[ 0 -eq 0 ]]
+ [[ '' != \t\r\u\e ]]
+ echo 'Do not run this script as root'
Do not run this script as root
+ exit 1
我就偏偏要以 root 使用者來運作怎麼惹 😡
# sed 去掉第一個 exit 1 就可以了
[email protected] [ ~ ]# sed -i.bak "s/exit 1//" install.sh
[email protected] [ ~ ]# bash install.sh
安裝好之後會輸出
Installation complete!
（講真官方的 install.sh 腳本輸出很不友好，污染我的 terminal
+ tanzu init
| initializing ✔ successfully initialized CLI
++ tanzu plugin repo list
++ grep tce
+ TCE_REPO=
+ [[ -z '' ]]
+ tanzu plugin repo add --name tce --gcp-bucket-name tce-tanzu-cli-plugins --gcp-root-path artifacts
++ tanzu plugin repo list
++ grep core-admin
+ TCE_REPO=
+ [[ -z '' ]]
+ tanzu plugin repo add --name core-admin --gcp-bucket-name tce-tanzu-cli-framework-admin --gcp-root-path artifacts-admin
+ echo 'Installation complete!'
Installation complete!
部署管理叢集
先是部署一個 tanzu 的管理叢集，有兩種方式，一種是通過 官方文檔[23] 提到的通過 Web UI 的方式。目前這個 UI 界面比較拉垮，它主要是用來讓使用者填寫一些配置參數，然後調用背景的 tanzu 指令來部署叢集。并把叢集部署的日志和進度展示出來；部署完成之後，這個 UI 又不能管理這些叢集，又不支援部署 workload 叢集（
另一種就是通過 tanzu 指令指定配置檔案來部署，這種方式不需要通過浏覽器在 web 頁面上傻乎乎地點來點去填一些參數，隻需要提前填寫好一個 yaml 格式的配置檔案即可。下面我們就采用 tanzu 指令來部署叢集，管理叢集的配置檔案模版如下：
- tanzu-mgt-cluster.yaml
# Cluster Pod IP 的 CIDR
CLUSTER_CIDR: 100.96.0.0/11
# Service 的 CIDR
SERVICE_CIDR: 100.64.0.0/13
# 叢集的名稱
CLUSTER_NAME: tanzu-control-plan
# 叢集的類型
CLUSTER_PLAN: dev
# 叢集節點的 arch
OS_ARCH: amd64
# 叢集節點的 OS 名稱
OS_NAME: photon
# 叢集節點 OS 版本
OS_VERSION: "3"
# 基礎設施資源的提供方
INFRASTRUCTURE_PROVIDER: vsphere
# 叢集的 VIP
VSPHERE_CONTROL_PLANE_ENDPOINT: 192.168.75.194
# control-plan 節點的磁盤大小
VSPHERE_CONTROL_PLANE_DISK_GIB: "20"
# control-plan 節點的記憶體大小
VSPHERE_CONTROL_PLANE_MEM_MIB: "8192"
# control-plan 節點的 CPU 核心數量
VSPHERE_CONTROL_PLANE_NUM_CPUS: "4"
# work 節點的磁盤大小
VSPHERE_WORKER_DISK_GIB: "20"
# work 節點的記憶體大小
VSPHERE_WORKER_MEM_MIB: "4096"
# work 節點的 CPU 核心數量
VSPHERE_WORKER_NUM_CPUS: "2"
# vCenter 的 Datacenter 路徑
VSPHERE_DATACENTER: /SH-IDC
# 虛拟機建立的 Datastore 路徑
VSPHERE_DATASTORE: /SH-IDC/datastore/datastore1
# 虛拟機建立的檔案夾
VSPHERE_FOLDER: /SH-IDC/vm/Tanzu-node
# 虛拟機使用的網絡
VSPHERE_NETWORK: /SH-IDC/network/VM Network
# 虛拟機關聯的資源池
VSPHERE_RESOURCE_POOL: /SH-IDC/host/Tanzu-Cluster/Resources
# vCenter 的 IP
VSPHERE_SERVER: 192.168.75.110
# vCenter 的使用者名
VSPHERE_USERNAME: [email protected]
# vCenter 的密碼，以 base64 編碼
VSPHERE_PASSWORD: <encoded:base64password>
# vCenter 的證書指紋，可以通過 govc about.cert -json | jq -r '.ThumbprintSHA1' 擷取
VSPHERE_TLS_THUMBPRINT: EB:F3:D8:7A:E8:3D:1A:59:B0:DE:73:96:DC:B9:5F:13:86:EF:B6:27
# 虛拟機注入的 ssh 公鑰，需要用它來 ssh 登入叢集節點
VSPHERE_SSH_AUTHORIZED_KEY: ssh-rsa
# 一些預設參數
AVI_ENABLE: "false"
IDENTITY_MANAGEMENT_TYPE: none
ENABLE_AUDIT_LOGGING: "false"
ENABLE_CEIP_PARTICIPATION: "false"
TKG_HTTP_PROXY_ENABLED: "false"
DEPLOY_TKG_ON_VSPHERE7: "true"
- 通過 tanzu CLI 部署管理叢集
$ tanzu management-cluster create --file tanzu-mgt-cluster.yaml -v6
# 如果沒有配置 VSPHERE_TLS_THUMBPRINT 會有一個确認 vSphere thumbprint 的互動，輸入 Y 就可以
Validating the pre-requisites...
Do you want to continue with the vSphere thumbprint EB:F3:D8:7A:E8:3D:1A:59:B0:DE:73:96:DC:B9:5F:13:86:EF:B6:27 [y/N]: y
部署日志
[email protected] [ ~ ]# tanzu management-cluster create --file tanzu-mgt-cluster.yaml -v 6
compatibility file (/root/.config/tanzu/tkg/compatibility/tkg-compatibility.yaml) already exists, skipping download
BOM files inside /root/.config/tanzu/tkg/bom already exists, skipping download
CEIP Opt-in status: false
Validating the pre-requisites...
vSphere 7.0 Environment Detected.
You have connected to a vSphere 7.0 environment which does not have vSphere with Tanzu enabled. vSphere with Tanzu includes
an integrated Tanzu Kubernetes Grid Service which turns a vSphere cluster into a platform for running Kubernetes workloads in dedicated
resource pools. Configuring Tanzu Kubernetes Grid Service is done through vSphere HTML5 client.
Tanzu Kubernetes Grid Service is the preferred way to consume Tanzu Kubernetes Grid in vSphere 7.0 environments. Alternatively you may
deploy a non-integrated Tanzu Kubernetes Grid instance on vSphere 7.0.
Deploying TKG management cluster on vSphere 7.0 ...
Identity Provider not configured. Some authentication features won't work.
Checking if VSPHERE_CONTROL_PLANE_ENDPOINT 192.168.20.94 is already in use
Setting up management cluster...
Validating configuration...
Using infrastructure provider vsphere:v0.7.10
Generating cluster configuration...
Setting up bootstrapper...
Fetching configuration for kind node image...
kindConfig:
&{{Cluster kind.x-k8s.io/v1alpha4} [{ map[] [{/var/run/docker.sock /var/run/docker.sock false false }] [] [] []}] { 0 100.96.0.0/11 100.64.0.0/13 false } map[] map[] [apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
imageRepository: projects.registry.vmware.com/tkg
etcd:
local:
imageRepository: projects.registry.vmware.com/tkg
imageTag: v3.4.13_vmware.15
dns:
type: CoreDNS
imageRepository: projects.registry.vmware.com/tkg
imageTag: v1.8.0_vmware.5] [] [] []}
Creating kind cluster: tkg-kind-c7vj6kds0a6sf43e6210
Creating cluster "tkg-kind-c7vj6kds0a6sf43e6210" ...
Ensuring node image (projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1) ...
Pulling image: projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1 ...
Preparing nodes ...
Writing configuration ...
Starting control-plane ...
Installing CNI ...
Installing StorageClass ...
Waiting 2m0s for control-plane = Ready ...
Ready after 19s
Bootstrapper created. Kubeconfig: /root/.kube-tkg/tmp/config_3fkzTCOL
Installing providers on bootstrapper...
Fetching providers
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
installed Component=="cluster-api" Type=="CoreProvider" Version=="v0.3.23"
installed Component=="kubeadm" Type=="BootstrapProvider" Version=="v0.3.23"
installed Component=="kubeadm" Type=="ControlPlaneProvider" Version=="v0.3.23"
installed Component=="vsphere" Type=="InfrastructureProvider" Version=="v0.7.10"
Waiting for provider infrastructure-vsphere
Waiting for provider control-plane-kubeadm
Waiting for provider cluster-api
Waiting for provider bootstrap-kubeadm
Waiting for resource capi-kubeadm-control-plane-controller-manager of type *v1.Deployment to be up and running
pods are not yet running for deployment 'capi-kubeadm-control-plane-controller-manager' in namespace 'capi-kubeadm-control-plane-system', retrying
Passed waiting on provider bootstrap-kubeadm after 25.205820854s
pods are not yet running for deployment 'capi-controller-manager' in namespace 'capi-webhook-system', retrying
Passed waiting on provider infrastructure-vsphere after 30.185406332s
Passed waiting on provider cluster-api after 30.213216243s
Success waiting on all providers.
Start creating management cluster...
patch cluster object with operation status:
{
"metadata": {
"annotations": {
"TKGOperationInfo" : "{\"Operation\":\"Create\",\"OperationStartTimestamp\":\"2022-02-06 02:35:34.30219421 +0000 UTC\",\"OperationTimeout\":1800}",
"TKGOperationLastObservedTimestamp" : "2022-02-06 02:35:34.30219421 +0000 UTC"
}
}
}
cluster control plane is still being initialized, retrying
Getting secret for cluster
Waiting for resource tanzu-control-plan-kubeconfig of type *v1.Secret to be up and running
Saving management cluster kubeconfig into /root/.kube/config
Installing providers on management cluster...
Fetching providers
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
installed Component=="cluster-api" Type=="CoreProvider" Version=="v0.3.23"
installed Component=="kubeadm" Type=="BootstrapProvider" Version=="v0.3.23"
installed Component=="kubeadm" Type=="ControlPlaneProvider" Version=="v0.3.23"
installed Component=="vsphere" Type=="InfrastructureProvider" Version=="v0.7.10"
Waiting for provider control-plane-kubeadm
Waiting for provider bootstrap-kubeadm
Waiting for provider infrastructure-vsphere
Waiting for provider cluster-api
Waiting for resource capi-kubeadm-control-plane-controller-manager of type *v1.Deployment to be up and running
Passed waiting on provider control-plane-kubeadm after 10.046865402s
Waiting for resource antrea-controller of type *v1.Deployment to be up and running
Moving all Cluster API objects from bootstrap cluster to management cluster...
Performing move...
Discovering Cluster API objects
Moving Cluster API objects Clusters=1
Creating objects in the target cluster
Deleting objects from the source cluster
Waiting for additional components to be up and running...
Waiting for packages to be up and running...
Waiting for package: antrea
Waiting for package: metrics-server
Waiting for package: tanzu-addons-manager
Waiting for package: vsphere-cpi
Waiting for package: vsphere-csi
Waiting for resource antrea of type *v1alpha1.PackageInstall to be up and running
Waiting for resource vsphere-cpi of type *v1alpha1.PackageInstall to be up and running
Waiting for resource vsphere-csi of type *v1alpha1.PackageInstall to be up and running
Waiting for resource metrics-server of type *v1alpha1.PackageInstall to be up and running
Waiting for resource tanzu-addons-manager of type *v1alpha1.PackageInstall to be up and running
Successfully reconciled package: antrea
Successfully reconciled package: vsphere-csi
Successfully reconciled package: metrics-server
Context set for management cluster tanzu-control-plan as '[email protected]'.
Deleting kind cluster: tkg-kind-c7vj6kds0a6sf43e6210
Management cluster created!
You can now create your first workload cluster by running the following:
tanzu cluster create [name] -f [file]
Some addons might be getting installed! Check their status by running the following:
kubectl get apps -A
- 部署完成之後，将管理叢集的 kubeconfig 檔案複制到 kubectl 預設的目錄下
[email protected] [ ~ ]# cp ${HOME}/.kube-tkg/config ${HOME}/.kube/config
- 檢視叢集狀态資訊
# 管理叢集的 cluster 資源資訊，管理叢集的 CR 預設儲存在了 tkg-system namespace 下
[email protected] [ ~ ]# kubectl get cluster -A
NAMESPACE NAME PHASE
tkg-system tanzu-control-plan Provisioned
# 管理叢集的 machine 資源資訊
[email protected] [ ~ ]# kubectl get machine -A
NAMESPACE NAME PROVIDERID PHASE VERSION
tkg-system tanzu-control-plan-control-plane-gs4bl vsphere://4239c450-f621-d78e-3c44-4ac8890c0cd3 Running v1.21.2+vmware.1
tkg-system tanzu-control-plan-md-0-7cdc97c7c6-kxcnx vsphere://4239d776-c04c-aacc-db12-3380542a6d03 Provisioned v1.21.2+vmware.1
# 運作的元件狀态
[email protected] [ ~ ]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-6494884869-wlzhx 2/2 Running 0 8m37s
capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-857d687b9d-tpznv 2/2 Running 0 8m35s
capi-system capi-controller-manager-778bd4dfb9-tkvwg 2/2 Running 0 8m41s
capi-webhook-system capi-controller-manager-9995bdc94-svjm2 2/2 Running 0 8m41s
capi-webhook-system capi-kubeadm-bootstrap-controller-manager-68845b65f8-sllgv 2/2 Running 0 8m38s
capi-webhook-system capi-kubeadm-control-plane-controller-manager-9847c6747-vvz6g 2/2 Running 0 8m35s
capi-webhook-system capv-controller-manager-55bf67fbd5-4t46v 2/2 Running 0 8m31s
capv-system capv-controller-manager-587fbf697f-bbzs9 2/2 Running 0 8m31s
cert-manager cert-manager-77f6fb8fd5-8tq6n 1/1 Running 0 11m
cert-manager cert-manager-cainjector-6bd4cff7bb-6vlzx 1/1 Running 0 11m
cert-manager cert-manager-webhook-fbfcb9d6c-qpkbc 1/1 Running 0 11m
kube-system antrea-agent-5m9d4 2/2 Running 0 6m
kube-system antrea-agent-8mpr7 2/2 Running 0 5m40s
kube-system antrea-controller-5bbcb98667-hklss 1/1 Running 0 5m50s
kube-system coredns-8dcb5c56b-ckvb7 1/1 Running 0 12m
kube-system coredns-8dcb5c56b-d98hf 1/1 Running 0 12m
kube-system etcd-tanzu-control-plan-control-plane-gs4bl 1/1 Running 0 12m
kube-system kube-apiserver-tanzu-control-plan-control-plane-gs4bl 1/1 Running 0 12m
kube-system kube-controller-manager-tanzu-control-plan-control-plane-gs4bl 1/1 Running 0 12m
kube-system kube-proxy-d4wq4 1/1 Running 0 12m
kube-system kube-proxy-nhkgg 1/1 Running 0 11m
kube-system kube-scheduler-tanzu-control-plan-control-plane-gs4bl 1/1 Running 0 12m
kube-system kube-vip-tanzu-control-plan-control-plane-gs4bl 1/1 Running 0 12m
kube-system metrics-server-59fcb9fcf-xjznj 1/1 Running 0 6m29s
kube-system vsphere-cloud-controller-manager-kzffm 1/1 Running 0 5m50s
kube-system vsphere-csi-controller-74675c9488-q9h5c 6/6 Running 0 6m31s
kube-system vsphere-csi-node-dmvvr 3/3 Running 0 6m31s
kube-system vsphere-csi-node-k6x98 3/3 Running 0 6m31s
tkg-system kapp-controller-6499b8866-xnql7 1/1 Running 0 10m
tkg-system tanzu-addons-controller-manager-657c587556-rpbjm 1/1 Running 0 7m58s
tkg-system tanzu-capabilities-controller-manager-6ff97656b8-cq7m7 1/1 Running 0 11m
tkr-system tkr-controller-manager-6bc455b5d4-wm98s 1/1 Running 0 10m
部署流程
結合 tanzu 的源碼[24] 和部署輸出的日志我們大體可以得知，tanzu 管理叢集部署大緻分為如下幾步：
// https://github.com/vmware-tanzu/tanzu-framework/blob/main/pkg/v1/tkg/client/init.go
// management cluster init step constants
const (
StepConfigPrerequisite = "Configure prerequisite"
StepValidateConfiguration = "Validate configuration"
StepGenerateClusterConfiguration = "Generate cluster configuration"
StepSetupBootstrapCluster = "Setup bootstrap cluster"
StepInstallProvidersOnBootstrapCluster = "Install providers on bootstrap cluster"
StepCreateManagementCluster = "Create management cluster"
StepInstallProvidersOnRegionalCluster = "Install providers on management cluster"
StepMoveClusterAPIObjects = "Move cluster-api objects from bootstrap cluster to management cluster"
)
// InitRegionSteps management cluster init step sequence
var InitRegionSteps = []string{
StepConfigPrerequisite,
StepValidateConfiguration,
StepGenerateClusterConfiguration,
StepSetupBootstrapCluster,
StepInstallProvidersOnBootstrapCluster,
StepCreateManagementCluster,
StepInstallProvidersOnRegionalCluster,
StepMoveClusterAPIObjects,
}
- ConfigPrerequisite 準備階段，會下載下傳
tkg-compatibility
tkg-bom
Downloading TKG compatibility file from 'projects.registry.vmware.com/tkg/framework-zshippable/tkg-compatibility'
Downloading the TKG Bill of Materials (BOM) file from 'projects.registry.vmware.com/tkg/tkg-bom:v1.4.0'
Downloading the TKr Bill of Materials (BOM) file from 'projects.registry.vmware.com/tkg/tkr-bom:v1.21.2_vmware.1-tkg.1'
ERROR 2022/02/06 02:24:46 svType != tvType; key=release, st=map[string]interface {}, tt=<nil>, sv=map[version:], tv=<nil>
CEIP Opt-in status: false
- ValidateConfiguration 配置檔案校驗，根據填寫的參數校驗配置是否正确，以及檢查 vCenter 當中有無比對的虛拟機模版；
Validating the pre-requisites...
vSphere 7.0 Environment Detected.
You have connected to a vSphere 7.0 environment which does not have vSphere with Tanzu enabled. vSphere with Tanzu includes
an integrated Tanzu Kubernetes Grid Service which turns a vSphere cluster into a platform for running Kubernetes workloads in dedicated
resource pools. Configuring Tanzu Kubernetes Grid Service is done through vSphere HTML5 client.
Tanzu Kubernetes Grid Service is the preferred way to consume Tanzu Kubernetes Grid in vSphere 7.0 environments. Alternatively you may
deploy a non-integrated Tanzu Kubernetes Grid instance on vSphere 7.0.
Deploying TKG management cluster on vSphere 7.0 ...
Identity Provider not configured. Some authentication features won't work.
Checking if VSPHERE_CONTROL_PLANE_ENDPOINT 192.168.20.94 is already in use
Setting up management cluster...
Validating configuration...
Using infrastructure provider vsphere:v0.7.10
- GenerateClusterConfiguration 生成叢集配置檔案資訊；
Generating cluster configuration...
- SetupBootstrapCluster 設定 bootstrap 叢集，目前預設為 kind。會運作一個 docker 容器，裡面套娃運作着一個 k8s 叢集；這個 bootstrap k8s 叢集隻是臨時運作 cluster-api 來部署管理叢集用的，部署完成之後 bootstrap 叢集也就沒用了，會自動删掉；
Setting up bootstrapper...
Fetching configuration for kind node image...
kindConfig:
&{{Cluster kind.x-k8s.io/v1alpha4} [{ map[] [{/var/run/docker.sock /var/run/docker.sock false false }] [] [] []}] { 0 100.96.0.0/11 100.64.0.0/13 false } map[] map[] [apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
imageRepository: projects.registry.vmware.com/tkg
etcd:
local:
imageRepository: projects.registry.vmware.com/tkg
imageTag: v3.4.13_vmware.15
dns:
type: CoreDNS
imageRepository: projects.registry.vmware.com/tkg
imageTag: v1.8.0_vmware.5] [] [] []}
Creating kind cluster: tkg-kind-c7vj6kds0a6sf43e6210
Creating cluster "tkg-kind-c7vj6kds0a6sf43e6210" ...
Ensuring node image (projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1) ...
Pulling image: projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1 ...
Preparing nodes ...
Writing configuration ...
Starting control-plane ...
Installing CNI ...
Installing StorageClass ...
Waiting 2m0s for control-plane = Ready ...
Ready after 19s
Bootstrapper created. Kubeconfig: /root/.kube-tkg/tmp/config_3fkzTCOL
- InstallProvidersOnBootstrapCluster 在 bootstrap 叢集上安裝 cluste-api 相關元件；
Installing providers on bootstrapper...
Fetching providers
# 安裝 cert-manager 主要是為了生成 k8s 叢集部署所依賴的那一堆證書
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
installed Component=="cluster-api" Type=="CoreProvider" Version=="v0.3.23"
installed Component=="kubeadm" Type=="BootstrapProvider" Version=="v0.3.23"
installed Component=="kubeadm" Type=="ControlPlaneProvider" Version=="v0.3.23"
installed Component=="vsphere" Type=="InfrastructureProvider" Version=="v0.7.10"
Waiting for provider infrastructure-vsphere
Waiting for provider control-plane-kubeadm
Waiting for provider cluster-api
Waiting for provider bootstrap-kubeadm
Passed waiting on provider infrastructure-vsphere after 30.185406332s
Passed waiting on provider cluster-api after 30.213216243s
Success waiting on all providers.
- CreateManagementCluster 建立管理叢集，這一步主要是建立虛拟機、初始化節點、運作 kubeadm 部署 k8s 叢集；
Start creating management cluster...
patch cluster object with operation status:
{
"metadata": {
"annotations": {
"TKGOperationInfo" : "{\"Operation\":\"Create\",\"OperationStartTimestamp\":\"2022-02-06 02:35:34.30219421 +0000 UTC\",\"OperationTimeout\":1800}",
"TKGOperationLastObservedTimestamp" : "2022-02-06 02:35:34.30219421 +0000 UTC"
}
}
}
cluster control plane is still being initialized, retrying
Getting secret for cluster
Waiting for resource tanzu-control-plan-kubeconfig of type *v1.Secret to be up and running
Saving management cluster kubeconfig into /root/.kube/config
- InstallProvidersOnRegionalCluster 在管理叢集上安裝 cluster-api 相關元件；
Installing providers on management cluster...
Fetching providers
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
installed Component=="cluster-api" Type=="CoreProvider" Version=="v0.3.23"
installed Component=="kubeadm" Type=="BootstrapProvider" Version=="v0.3.23"
installed Component=="kubeadm" Type=="ControlPlaneProvider" Version=="v0.3.23"
installed Component=="vsphere" Type=="InfrastructureProvider" Version=="v0.7.10"
Waiting for provider control-plane-kubeadm
Waiting for provider bootstrap-kubeadm
Waiting for provider infrastructure-vsphere
Waiting for provider cluster-api
Waiting for resource capv-controller-manager of type *v1.Deployment to be up and running
Passed waiting on provider infrastructure-vsphere after 20.091935635s
Passed waiting on provider cluster-api after 20.109419304s
Success waiting on all providers.
Waiting for the management cluster to get ready for move...
Waiting for resource tanzu-control-plan of type *v1alpha3.Cluster to be up and running
Waiting for resources type *v1alpha3.MachineDeploymentList to be up and running
Waiting for resources type *v1alpha3.MachineList to be up and running
Waiting for addons installation...
Waiting for resources type *v1alpha3.ClusterResourceSetList to be up and running
Waiting for resource antrea-controller of type *v1.Deployment to be up and running
- MoveClusterAPIObjects 将 bootstrap 叢集上 cluster-api 相關的資源轉移到管理叢集上。這一步的目的是為了達到 self-hosted 自托管的功能：即管理叢集自身的擴縮容也是通過 cluster-api 來完成，這樣就不用再依賴先前的那個 bootstrap 叢集了；
Moving all Cluster API objects from bootstrap cluster to management cluster...
Performing move...
Discovering Cluster API objects
Moving Cluster API objects Clusters=1
Creating objects in the target cluster
Deleting objects from the source cluster
Context set for management cluster tanzu-control-plan as '[email protected]'.
Deleting kind cluster: tkg-kind-c7vj6kds0a6sf43e6210
Management cluster created!
You can now create your first workload cluster by running the following:
tanzu cluster create [name] -f [file]
Some addons might be getting installed! Check their status by running the following:
kubectl get apps -A
部署完成後會删除 bootstrap 叢集，因為 bootstrap 叢集中的資源已經轉移到了管理叢集中，它繼續存在的意義不大。
部署 workload 叢集
上面我們隻是部署好了一個 tanzu 管理叢集，我們真正的工作負載并不适合運作在這個叢集上，是以我們還需要再部署一個 workload 叢集，類似于 k8s 叢集中的 worker 節點。部署 workload 叢集的時候不再依賴 bootstrap 叢集，而是使用管理叢集。
根據官方文檔 vSphere Workload Cluster Template[25] 中給出的模版建立一個配置檔案，然後再通過 tanzu 指令來部署即可。配置檔案内容如下：
# Cluster Pod IP 的 CIDR
CLUSTER_CIDR: 100.96.0.0/11
# Service 的 CIDR
SERVICE_CIDR: 100.64.0.0/13
# 叢集的名稱
CLUSTER_NAME: tanzu-workload-cluster
# 叢集的類型
CLUSTER_PLAN: dev
# 叢集節點的 arch
OS_ARCH: amd64
# 叢集節點的 OS 名稱
OS_NAME: photon
# 叢集節點 OS 版本
OS_VERSION: "3"
# 基礎設施資源的提供方
INFRASTRUCTURE_PROVIDER: vsphere
# cluster, machine 等自定義資源建立的 namespace
NAMESPACE: default
# CNI 選用類型，目前應該隻支援 VMware 自家的 antrea
CNI: antrea
# 叢集的 VIP
VSPHERE_CONTROL_PLANE_ENDPOINT: 192.168.20.95
# control-plan 節點的磁盤大小
VSPHERE_CONTROL_PLANE_DISK_GIB: "20"
# control-plan 節點的記憶體大小
VSPHERE_CONTROL_PLANE_MEM_MIB: "8192"
# control-plan 節點的 CPU 核心數量
VSPHERE_CONTROL_PLANE_NUM_CPUS: "4"
# work 節點的磁盤大小
VSPHERE_WORKER_DISK_GIB: "20"
# work 節點的記憶體大小
VSPHERE_WORKER_MEM_MIB: "4096"
# work 節點的 CPU 核心數量
VSPHERE_WORKER_NUM_CPUS: "2"
# vCenter 的 Datacenter 路徑
VSPHERE_DATACENTER: /SH-IDC
# 虛拟機建立的 Datastore 路徑
VSPHERE_DATASTORE: /SH-IDC/datastore/datastore1
# 虛拟機建立的檔案夾
VSPHERE_FOLDER: /SH-IDC/vm/Tanzu-node
# 虛拟機使用的網絡
VSPHERE_NETWORK: /SH-IDC/network/VM Network
# 虛拟機關聯的資源池
VSPHERE_RESOURCE_POOL: /SH-IDC/host/Tanzu-Cluster/Resources
# vCenter 的 IP
VSPHERE_SERVER: 192.168.20.92
# vCenter 的使用者名
VSPHERE_USERNAME: [email protected]
# vCenter 的密碼，以 base64 編碼
VSPHERE_PASSWORD: <encoded:YWRtaW5AMjAyMA==>
# vCenter 的證書指紋，可以通過 govc about.cert -json | jq -r '.ThumbprintSHA1' 擷取
VSPHERE_TLS_THUMBPRINT: CB:23:48:E8:93:34:AD:27:D8:FD:88:1C:D7:08:4B:47:9B:12:F4:E0
# 虛拟機注入的 ssh 公鑰，需要用它來 ssh 登入叢集節點
VSPHERE_SSH_AUTHORIZED_KEY: ssh-rsa
# 一些預設參數
AVI_ENABLE: "false"
IDENTITY_MANAGEMENT_TYPE: none
ENABLE_AUDIT_LOGGING: "false"
ENABLE_CEIP_PARTICIPATION: "false"
TKG_HTTP_PROXY_ENABLED: "false"
DEPLOY_TKG_ON_VSPHERE7: "true"
# 是否開啟虛拟機健康檢查
ENABLE_MHC: true
MHC_UNKNOWN_STATUS_TIMEOUT: 5m
MHC_FALSE_STATUS_TIMEOUT: 12m
# 是否部署 vsphere cis 元件
ENABLE_DEFAULT_STORAGE_CLASS: true
# 是否開啟叢集自動擴縮容
ENABLE_AUTOSCALER: false
- 通過 tanzu 指令來部署 workload 叢集
[email protected] [ ~ ]# tanzu cluster create tanzu-workload-cluster --file tanzu-workload-cluster.yaml
Validating configuration...
Warning: Pinniped configuration not found. Skipping pinniped configuration in workload cluster. Please refer to the documentation to check if you can configure pinniped on workload cluster manually
Creating workload cluster 'tanzu-workload-cluster'...
Waiting for cluster to be initialized...
Waiting for cluster nodes to be available...
Waiting for cluster autoscaler to be available...
Unable to wait for autoscaler deployment to be ready. reason: deployments.apps "tanzu-workload-cluster-cluster-autoscaler" not found
Waiting for addons installation...
Waiting for packages to be up and running...
Workload cluster 'tanzu-workload-cluster' created
- 部署完成之後檢視一下叢集的 CR 資訊
[email protected] [ ~ ]# kubectl get cluster
NAME PHASE
tanzu-workload-cluster Provisioned
# machine 狀态處于 Running 說明節點已經正常運作了
[email protected] [ ~ ]# kubectl get machine
NAME PROVIDERID PHASE VERSION
tanzu-workload-cluster-control-plane-4tdwq vsphere://423950ac-1c6d-e5ef-3132-77b6a53cf626 Running v1.21.2+vmware.1
tanzu-workload-cluster-md-0-8555bbbfc-74vdg vsphere://4239b83b-6003-d990-4555-a72ac4dec484 Running v1.21.2+vmware.1
擴容叢集
叢集部署好之後，如果想對叢集節點進行擴縮容，我們可以像 deployment 的一樣，隻需要修改一些 CR 的資訊即可。cluster-api 相關元件會 watch 到這些 CR 的變化，并根據它的 spec 資訊進行一系列調諧操作。如果目前叢集節點數量低于所定義的節點副本數量，則會自動調用對應的 Provider 建立虛拟機，并對虛拟機進行初始化操作，将它轉換為 k8s 裡的一個 node 資源；
擴容 control-plan 節點
即擴容 master 節點，通過修改
KubeadmControlPlane
這個 CR 中的
replicas
副本數即可：
[email protected] [ ~ ]# kubectl scale kcp tanzu-workload-cluster-control-plane --replicas=3
# 可以看到 machine 已經處于 Provisioning 狀态，說明叢集節點對應的虛拟機正在建立中
[email protected] [ ~ ]# kubectl get machine
NAME PROVIDERID PHASE VERSION
tanzu-workload-cluster-control-plane-4tdwq vsphere://423950ac-1c6d-e5ef-3132-77b6a53cf626 Running v1.21.2+vmware.1
tanzu-workload-cluster-control-plane-mkmd2 Provisioning v1.21.2+vmware.1
tanzu-workload-cluster-md-0-8555bbbfc-74vdg vsphere://4239b83b-6003-d990-4555-a72ac4dec484 Running v1.21.2+vmware.1
擴容 work 節點
擴容 worker 節點，通過修改
MachineDeployment
這個 CR 中的
replicas
副本數即可：
[email protected] [ ~ ]# kubectl scale md tanzu-workload-cluster-md-0 --replicas=3
[email protected] [ ~ ]# kubectl get machine
NAME PROVIDERID PHASE VERSION
tanzu-workload-cluster-control-plane-4tdwq vsphere://423950ac-1c6d-e5ef-3132-77b6a53cf626 Running v1.21.2+vmware.1
tanzu-workload-cluster-control-plane-mkmd2 vsphere://4239278c-0503-f03a-08b8-df92286bcdd7 Running v1.21.2+vmware.1
tanzu-workload-cluster-control-plane-rt5mb vsphere://4239c882-2fe5-a394-60c0-616941a6363e Running v1.21.2+vmware.1
tanzu-workload-cluster-md-0-8555bbbfc-4hlqk vsphere://42395deb-e706-8b4b-a44f-c755c222575c Running v1.21.2+vmware.1
tanzu-workload-cluster-md-0-8555bbbfc-74vdg vsphere://4239b83b-6003-d990-4555-a72ac4dec484 Running v1.21.2+vmware.1
tanzu-workload-cluster-md-0-8555bbbfc-ftmlp vsphere://42399640-8e94-85e5-c4bd-8436d84966e0 Running v1.21.2+vmware.1
後續
本文隻是介紹了 tanzu 叢集部署的大體流程，裡面包含了 cluster-api 相關的概念在本文并沒有做深入的分析，因為實在是太複雜了 😂，到現在我還是沒太了解其中的一些原理，是以後續我再單獨寫一篇部落格來講解一些 cluster-api 相關的内容，到那時候在結合本文來看就容易了解很多。
參考
- community-edition[26]
- vmware/photon[27]
- tanzu-framework[28]
- cluster-api-provider-vsphere[29]
- Deploying a workload cluster[30]
- Examine the Management Cluster Deployment[31]
- Prepare to Deploy a Management or Standalone Clusters to vSphere[32]
引用連結
[1]
VMware vShpere: https://docs.vmware.com/cn/VMware-vSphere/index.html
[2]
OpenStack: https://www.openstack.org/
[3]
cluster-api: https://github.com/kubernetes-sigs/cluster-api
[4]
tanzu: https://github.com/vmware-tanzu
[5]
VMware Tanzu 社群版: https://github.com/vmware-tanzu/community-edition
[6]
VMware 的賬戶: https://customerconnect.vmware.com/login
[7]
VMware-VMvisor-Installer-7.0U2a-17867351.x86_64.iso: https://customerconnect.vmware.com/downloads/details?downloadGroup=ESXI70U2A&productId=974&rPId=46384
[8]
VMware-VCSA-all-7.0.3-19234570.iso: https://customerconnect.vmware.com/downloads/details?downloadGroup=VC70U3C&productId=974&rPId=83853
[9]
photon-ova-4.0-c001795b80.ova: https://packages.vmware.com/photon/4.0/Rev2/ova/photon-ova-4.0-c001795b80.ova
[10]
photon-3-kube-v1.21.2+vmware.1-tkg.2-12816990095845873721.ova: https://customerconnect.vmware.com/downloads/get-download?downloadGroup=TCE-090
[11]
tce-linux-amd64-v0.9.1.tar.gz: https://github.com/vmware-tanzu/community-edition/releases/download/v0.9.1/tce-linux-amd64-v0.9.1.tar.gz
[12]
govc_Linux_x86_64.tar.gz: https://github.com/vmware/govmomi/releases/download/v0.27.3/govc_Linux_x86_64.tar.gz
[13]
vSphere 7.0 Update 3 Critical Known Issues - Workarounds & Fix (86287): https://kb.vmware.com/s/article/86287
[14]
VMware ESXi 安裝和設定: https://docs.vmware.com/cn/VMware-vSphere/7.0/vsphere-esxi-701-installation-setup-guide.pdf
[15]
關于 vCenter Server 安裝和設定: https://docs.vmware.com/cn/VMware-vSphere/7.0/com.vmware.vcenter.install.doc/GUID-8DC3866D-5087-40A2-8067-1361A2AF95BD.html
[16]
Docker: https://docs.docker.com/engine/install/
[17]
Manage Docker as a non-root user: https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user
[18]
Kubectl: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/
[19]
cgroup v1: https://man7.org/linux/man-pages/man7/cgroups.7.html
[20]
Check and set the cgroup: https://tanzucommunityedition.io/docs/latest/support-matrix/#check-and-set-the-cgroup
[21]
Photon OS 4.0 Rev2: https://github.com/vmware/photon/wiki/Downloading-Photon-OS#photon-os-40-rev2-binaries
[22]
vmware-tanzu/community-edition: https://github.com/vmware-tanzu/community-edition/releases/tag/v0.9.1
[23]
官方文檔: https://tanzucommunityedition.io/docs/latest/getting-started/
[24]
tanzu 的源碼: https://github.com/vmware-tanzu/tanzu-framework/blob/main/pkg/v1/tkg/client/init.go
[25]
vSphere Workload Cluster Template: https://tanzucommunityedition.io/docs/latest/vsphere-wl-template/
[26]
community-edition: https://github.com/vmware-tanzu/community-edition
[27]
vmware/photon: https://github.com/vmware/photon
[28]
tanzu-framework: https://github.com/vmware-tanzu/tanzu-framework/blob/main/pkg/v1/tkg/client/init.go
[29]
cluster-api-provider-vsphere: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere
[30]
Deploying a workload cluster: https://tanzucommunityedition.io/docs/latest/workload-clusters/
[31]
Examine the Management Cluster Deployment: https://tanzucommunityedition.io/docs/latest/verify-deployment/
[32]
Prepare to Deploy a Management or Standalone Clusters to vSphere: https://tanzucommunityedition.io/docs/latest/vsphere/
原文連結：https://blog.k8s.li/deploy-tanzu-k8s-cluster.html
你可能還喜歡
點選下方圖檔即可閱讀
Pod 的狀态出現 UnexpectedAdmissionError 是什麼鬼?
雲原生是一種信仰 🤘
關注公衆号
背景回複◉k8s◉擷取史上最友善快捷的 Kubernetes 高可用部署工具，隻需一條指令，連 ssh 都不需要！
點選 "閱讀原文" 擷取更好的閱讀體驗！
發現朋友圈變“安靜”了嗎？