1.12 配置Linux核心（cilium必選）
cilium和其他的cni元件最大的不同在于其底層使用了ebpf技術，而該技術對于Linux的系統核心版本有較高的要求，完成的要求可以檢視官網的詳細連結，這裡我們着重看核心版本、核心參數這兩個部分。
Linux核心版本
預設情況下我們可以參考cilium官方給出的一個系統要求總結。因為我們是在k8s叢集中部署（使用容器），是以隻需要關注Linux核心版本和etcd版本即可。根據前面部署的經驗我們可以知道1.23.6版本的k8s預設使用的etcd版本是3.5.+，是以重點就來到了Linux核心版本這裡。
This requirement is only needed if you run cilium-agent natively. If you are using the Cilium container image cilium/cilium, clang+LLVM is included in the container image.
iproute2 is only needed if you run cilium-agent directly on the host machine. iproute2 is included in 毫無疑問CentOS7内置的預設核心版本3.10.x版本的核心是無法滿足需求的，但是在更新核心之前，我們再看看其他的一些要求。
cilium官方還給出了一份清單描述了各項進階功能對核心版本的要求：
cilium官方對各項功能所需要開啟的核心參數列舉如下：
In order for the eBPF feature to be enabled properly, the following kernel configuration options must be enabled.
This is typically the case with distribution kernels. When an option can be built as a module or statically linked, either choice is valid.
為了正确啟用 eBPF 功能，必須啟用以下核心配置選項。這通常因核心版本情況而異。任何一個選項都可以建構為子產品或靜态連結，兩個選擇都是有效的。 我們暫時隻看最基本的
Base Requirements
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NET_CLS_BPF=y
CONFIG_BPF_JIT=y
CONFIG_NET_CLS_ACT=y
CONFIG_NET_SCH_INGRESS=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_BPF=y 對比我們使用的
5.17.6-1.el7.elrepo.x86_64
核心可以發現有兩個子產品是為m
$ egrep "^CONFIG_BPF=|^CONFIG_BPF_SYSCALL=|^CONFIG_NET_CLS_BPF=|^CONFIG_BPF_JIT=|^CONFIG_NET_CLS_ACT=|^CONFIG_NET_SCH_INGRESS=|^CONFIG_CRYPTO_SHA1=|^CONFIG_CRYPTO_USER_API_HASH=|^CONFIG_CGROUPS=|^CONFIG_CGROUP_BPF=" /boot/config-5.17.6-1.el7.elrepo.x86_64
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_BPF=y
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_CLS_ACT=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_USER_API_HASH=y 缺少的這兩個子產品我們可以在/usr/lib/modules/$(uname -r)目錄下面找到它們： $ realpath ./kernel/net/sched/sch_ingress.ko
/usr/lib/modules/5.17.6-1.el7.elrepo.x86_64/kernel/net/sched/sch_ingress.ko
$ realpath ./kernel/net/sched/cls_bpf.ko
/usr/lib/modules/5.17.6-1.el7.elrepo.x86_64/kernel/net/sched/cls_bpf.ko 确認相關核心子產品存在我們直接加載核心即可：
# 直接使用modprobe指令加載
$ modprobe cls_bpf
$ modprobe sch_ingress
$ lsmod | egrep "cls_bpf|sch_ingress"
sch_ingress 16384 0
cls_bpf 24576 0
# 配置開機自動加載cilium所需相關子產品
cat <<EOF | tee /etc/modules-load.d/cilium-base-requirements.conf
cls_bpf
sch_ingress
EOF 安裝CNI
安裝cilium
快速安裝的教程可以參考官網文檔，基本的安裝思路就是先下載下傳cilium官方的cli工具，然後使用cli工具進行安裝。
這種安裝方式的優勢就是簡單快捷，缺點就是缺少自定義參數配置的功能，隻能使用官方原先設定的預設參數，比較适合快速初始化搭建可用環境用來學習和測試。
# cilium的cli工具是一個二進制的可執行檔案
$ curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
$ sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
cilium-linux-amd64.tar.gz: OK
$ sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
cilium
# 使用該指令即可完成cilium的安裝
$ cilium install
ℹ️ using Cilium version "v1.11.3"
🔮 Auto-detected cluster name: kubernetes
🔮 Auto-detected IPAM mode: cluster-pool
ℹ️ helm template --namespace kube-system cilium cilium/cilium --version 1.11.3 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,ipam.mode=cluster-pool,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator
ℹ️ Storing helm values file in kube-system/cilium-cli-helm-values Secret
🔑 Created CA in secret cilium-ca
🔑 Generating certificates for Hubble...
🚀 Creating Service accounts...
🚀 Creating Cluster roles...
🚀 Creating ConfigMap for Cilium version 1.11.3...
🚀 Creating Agent DaemonSet...
level=warning msg="spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[1].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use \"kubernetes.io/os\" instead" subsys=klog
level=warning msg="spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the \"priorityClassName\" field instead" subsys=klog
🚀 Creating Operator Deployment...
⌛ Waiting for Cilium to be installed and ready...
✅ Cilium was successfully installed! Run 'cilium status' to view installation health
# 檢視cilium的狀态
$ cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Hubble: disabled
\__/¯¯\__/ ClusterMesh: disabled
\__/
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium-operator Running: 1
cilium Running: 3
Cluster Pods: 2/2 managed by Cilium
Image versions cilium quay.io/cilium/cilium:v1.11.3@sha256:cb6aac121e348abd61a692c435a90a6e2ad3f25baa9915346be7b333de8a767f: 3
cilium-operator quay.io/cilium/operator-generic:v1.11.3@sha256:5b81db7a32cb7e2d00bb3cf332277ec2b3be239d9e94a8d979915f4e6648c787: 1 配置hubble
# 我們先使用cilium-cli工具在k8s叢集中部署hubble，隻需要下面一條指令即可
$ cilium hubble enable
🔑 Found CA in secret cilium-ca
ℹ️ helm template --namespace kube-system cilium cilium/cilium --version 1.11.3 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,hubble.enabled=true,hubble.relay.enabled=true,hubble.tls.ca.cert=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNGRENDQWJxZ0F3SUJBZ0lVSDRQcit1UU0xSXZtdWQvVlV3YWlycGllSEZBd0NnWUlLb1pJemowRUF3SXcKYURFTE1Ba0dBMVVFQmhNQ1ZWTXhGakFVQmdOVkJBZ1REVk5oYmlCR2NtRnVZMmx6WTI4eEN6QUpCZ05WQkFjVApBa05CTVE4d0RRWURWUVFLRXdaRGFXeHBkVzB4RHpBTkJnTlZCQXNUQmtOcGJHbDFiVEVTTUJBR0ExVUVBeE1KClEybHNhWFZ0SUVOQk1CNFhEVEl5TURVd09UQTVNREF3TUZvWERUSTNNRFV3T0RBNU1EQXdNRm93YURFTE1Ba0cKQTFVRUJoTUNWVk14RmpBVUJnTlZCQWdURFZOaGJpQkdjbUZ1WTJselkyOHhDekFKQmdOVkJBY1RBa05CTVE4dwpEUVlEVlFRS0V3WkRhV3hwZFcweER6QU5CZ05WQkFzVEJrTnBiR2wxYlRFU01CQUdBMVVFQXhNSlEybHNhWFZ0CklFTkJNRmt3RXdZSEtvWkl6ajBDQVFZSUtvWkl6ajBEQVFjRFFnQUU3Z21EQ05WOERseEIxS3VYYzhEdndCeUoKWUxuSENZNjVDWUhBb3ZBY3FUM3drcitLVVNwelcyVjN0QW9IaFdZV0UyQ2lUNjNIOXZLV1ZRY3pHeXp1T0tOQwpNRUF3RGdZRFZSMFBBUUgvQkFRREFnRUdNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdIUVlEVlIwT0JCWUVGQmMrClNDb3F1Y0JBc09sdDBWaEVCbkwyYjEyNE1Bb0dDQ3FHU000OUJBTUNBMGdBTUVVQ0lRRDJsNWVqaDVLVTkySysKSHJJUXIweUwrL05pZ3NSUHRBblA5T3lDcHExbFJBSWdYeGY5a2t5N2xYU0pOYmpkREFjbnBrNlJFTFp2eEkzbQpKaG9JRkRlbER0dz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=,hubble.tls.ca.key=[--- REDACTED WHEN PRINTING TO TERMINAL (USE --redact-helm-certificate-keys=false TO PRINT) ---],ipam.mode=cluster-pool,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator
✨ Patching ConfigMap cilium-config to enable Hubble...
🚀 Creating ConfigMap for Cilium version 1.11.3...
♻️ Restarted Cilium pods
⌛ Waiting for Cilium to become ready before deploying other Hubble component(s)...
✨ Generating certificates...
🔑 Generating certificates for Relay...
✨ Deploying Relay...
⌛ Waiting for Hubble to be installed...
ℹ️ Storing helm values file in kube-system/cilium-cli-helm-values Secret
✅ Hubble was successfully enabled!
# 安裝hubble-cli工具，安裝邏輯和cilium-cli的邏輯相似
$ export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
$ curl -L --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz{,.sha256sum}
$ sha256sum --check hubble-linux-amd64.tar.gz.sha256sum
hubble-linux-amd64.tar.gz: OK
$ sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin
hubble
# 首先我們要開啟hubble的api，使用cilium-cli開啟轉發
$ cilium hubble port-forward&
[1] 15512
$ kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hubble-relay ClusterIP 10.188.55.197 <none> 80/TCP 16h
hubble-ui ClusterIP 10.188.17.78 <none> 80/TCP 16h
kube-dns ClusterIP 10.188.0.10 <none> 53/UDP,53/TCP,9153/TCP 17h
$ netstat -ntulp | grep 4245
tcp 0 0 0.0.0.0:4245 0.0.0.0:* LISTEN 15527/kubectl
tcp6 0 0 :::4245 :::* LISTEN 15527/kubectl
# 實際上執行的操作等同于下面這個指令
# kubectl port-forward -n kube-system svc/hubble-relay --address 0.0.0.0 --address :: 4245:80
# 測試和hubble-api的連通性
$ hubble status
Healthcheck (via localhost:4245): Ok
Current/Max Flows: 12,285/12,285 (100.00%)
Flows/s: 28.58
Connected Nodes: 3/3
# 使用hubble指令檢視資料的轉發情況
$ hubble observe
Handling connection for 4245
May 9 09:33:25.861: 10.0.1.47:44484 -> cilium-test/echo-same-node-5767b7b99d-xhzpb:8080 to-endpoint FORWARDED (TCP Flags: ACK, PSH)
May 9 09:33:25.863: 10.0.1.47:44484 <- cilium-test/echo-same-node-5767b7b99d-xhzpb:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
May 9 09:33:25.864: 10.0.1.47:44484 -> cilium-test/echo-same-node-5767b7b99d-xhzpb:8080 to-endpoint FORWARDED (TCP Flags: ACK, FIN)
...此處略去一堆輸出...
# 開啟hubble ui元件
$ cilium hubble enable --ui
🔑 Found CA in secret cilium-ca
ℹ️ helm template --namespace kube-system cilium cilium/cilium --version 1.11.3 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,hubble.enabled=true,hubble.relay.enabled=true,hubble.tls.ca.cert=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNGRENDQWJxZ0F3SUJBZ0lVSDRQcit1UU0xSXZtdWQvVlV3YWlycGllSEZBd0NnWUlLb1pJemowRUF3SXcKYURFTE1Ba0dBMVVFQmhNQ1ZWTXhGakFVQmdOVkJBZ1REVk5oYmlCR2NtRnVZMmx6WTI4eEN6QUpCZ05WQkFjVApBa05CTVE4d0RRWURWUVFLRXdaRGFXeHBkVzB4RHpBTkJnTlZCQXNUQmtOcGJHbDFiVEVTTUJBR0ExVUVBeE1KClEybHNhWFZ0SUVOQk1CNFhEVEl5TURVd09UQTVNREF3TUZvWERUSTNNRFV3T0RBNU1EQXdNRm93YURFTE1Ba0cKQTFVRUJoTUNWVk14RmpBVUJnTlZCQWdURFZOaGJpQkdjbUZ1WTJselkyOHhDekFKQmdOVkJBY1RBa05CTVE4dwpEUVlEVlFRS0V3WkRhV3hwZFcweER6QU5CZ05WQkFzVEJrTnBiR2wxYlRFU01CQUdBMVVFQXhNSlEybHNhWFZ0CklFTkJNRmt3RXdZSEtvWkl6ajBDQVFZSUtvWkl6ajBEQVFjRFFnQUU3Z21EQ05WOERseEIxS3VYYzhEdndCeUoKWUxuSENZNjVDWUhBb3ZBY3FUM3drcitLVVNwelcyVjN0QW9IaFdZV0UyQ2lUNjNIOXZLV1ZRY3pHeXp1T0tOQwpNRUF3RGdZRFZSMFBBUUgvQkFRREFnRUdNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdIUVlEVlIwT0JCWUVGQmMrClNDb3F1Y0JBc09sdDBWaEVCbkwyYjEyNE1Bb0dDQ3FHU000OUJBTUNBMGdBTUVVQ0lRRDJsNWVqaDVLVTkySysKSHJJUXIweUwrL05pZ3NSUHRBblA5T3lDcHExbFJBSWdYeGY5a2t5N2xYU0pOYmpkREFjbnBrNlJFTFp2eEkzbQpKaG9JRkRlbER0dz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=,hubble.tls.ca.key=[--- REDACTED WHEN PRINTING TO TERMINAL (USE --redact-helm-certificate-keys=false TO PRINT) ---],hubble.ui.enabled=true,hubble.ui.securityContext.enabled=false,ipam.mode=cluster-pool,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator
✨ Patching ConfigMap cilium-config to enable Hubble...
🚀 Creating ConfigMap for Cilium version 1.11.3...
♻️ Restarted Cilium pods
⌛ Waiting for Cilium to become ready before deploying other Hubble component(s)...
✅ Relay is already deployed
✨ Deploying Hubble UI and Hubble UI Backend...
⌛ Waiting for Hubble to be installed...
ℹ️ Storing helm values file in kube-system/cilium-cli-helm-values Secret
✅ Hubble was successfully enabled!
# 實際上這時候我們再檢視k8s叢集的狀态可以看到部署了一個名為hubble-ui的deployment
$ kubectl get deployment -n kube-system | grep hubble
hubble-relay 1/1 1 1 17h
hubble-ui 1/1 1 1 17h
$ kubectl get svc -n kube-system | grep hubble
hubble-relay ClusterIP 10.188.55.197 <none> 80/TCP 17h
hubble-ui ClusterIP 10.188.17.78 <none> 80/TCP 17h
# 将hubble-ui這個服務的80端口暴露到主控端上面的12000端口上面
$ cilium hubble ui&
[2] 5809
ℹ️ Opening "http://localhost:12000" in your browser...
# 實際上執行的操作等同于下面這個指令
# kubectl port-forward -n kube-system svc/hubble-ui --address 0.0.0.0 --address :: 12000:80 通路k8s主控端節點的IP+端口就可以看到hubble-ui的界面了
最後所有的相關服務都部署完成之後，我們再檢查一下整個cilium的狀态
$ cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Hubble: OK
\__/¯¯\__/ ClusterMesh: disabled
\__/
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
Containers: cilium Running: 3
cilium-operator Running: 1
hubble-relay Running: 1
hubble-ui Running: 1
Cluster Pods: 8/8 managed by Cilium
Image versions cilium quay.io/cilium/cilium:v1.11.3@sha256:cb6aac121e348abd61a692c435a90a6e2ad3f25baa9915346be7b333de8a767f: 3
cilium-operator quay.io/cilium/operator-generic:v1.11.3@sha256:5b81db7a32cb7e2d00bb3cf332277ec2b3be239d9e94a8d979915f4e6648c787: 1
hubble-relay quay.io/cilium/hubble-relay:v1.11.3@sha256:7256ec111259a79b4f0e0f80ba4256ea23bd472e1fc3f0865975c2ed113ccb97: 1
hubble-ui quay.io/cilium/hubble-ui:v0.8.5@sha256:4eaca1ec1741043cfba6066a165b3bf251590cf4ac66371c4f63fbed2224ebb4: 1
hubble-ui quay.io/cilium/hubble-ui-backend:v0.8.5@sha256:2bce50cf6c32719d072706f7ceccad654bfa907b2745a496da99610776fe31ed: 1
hubble-ui docker.io/envoyproxy/envoy:v1.18.4@sha256:e5c2bb2870d0e59ce917a5100311813b4ede96ce4eb0c6bfa879e3fbe3e83935: 1 添加核心yum源
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm 檢視支援的核心版本
[root@master1 yum.repos.d]# yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* elrepo-kernel: mirrors.tuna.tsinghua.edu.cn
elrepo-kernel | 3.0 kB 00:00:00
elrepo-kernel/primary_db | 2.1 MB 00:00:02
Available Packages
elrepo-release.noarch 7.0-6.el7.elrepo elrepo-kernel
kernel-lt.x86_64 5.4.210-1.el7.elrepo elrepo-kernel
kernel-lt-devel.x86_64 5.4.210-1.el7.elrepo elrepo-kernel
kernel-lt-doc.noarch 5.4.210-1.el7.elrepo elrepo-kernel
kernel-lt-headers.x86_64 5.4.210-1.el7.elrepo elrepo-kernel
kernel-lt-tools.x86_64 5.4.210-1.el7.elrepo elrepo-kernel
kernel-lt-tools-libs.x86_64 5.4.210-1.el7.elrepo elrepo-kernel
kernel-lt-tools-libs-devel.x86_64 5.4.210-1.el7.elrepo elrepo-kernel
kernel-ml.x86_64 5.19.2-1.el7.elrepo elrepo-kernel
kernel-ml-devel.x86_64 5.19.2-1.el7.elrepo elrepo-kernel
kernel-ml-doc.noarch 5.19.2-1.el7.elrepo elrepo-kernel
kernel-ml-headers.x86_64 5.19.2-1.el7.elrepo elrepo-kernel
kernel-ml-tools.x86_64 5.19.2-1.el7.elrepo elrepo-kernel
kernel-ml-tools-libs.x86_64 5.19.2-1.el7.elrepo elrepo-kernel
kernel-ml-tools-libs-devel.x86_64 5.19.2-1.el7.elrepo elrepo-kernel
perf.x86_64 5.19.2-1.el7.elrepo elrepo-kernel
python-perf.x86_64 5.19.2-1.el7.elrepo elrepo-kernel kernel-ml
kernel-ml 中的ml是英文【 mainline stable 】的縮寫，elrepo-kernel中羅列出來的是最新的穩定主線版本。
kernel-lt
kernel-lt 中的lt是英文【 long term support 】的縮寫，elrepo-kernel中羅列出來的長期支援版本。
yum --enablerepo=elrepo-kernel install kernel-ml-devel kernel-ml -y 檢視核心
[root@master1 yum.repos.d]# awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
CentOS Linux (5.19.2-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (5.9.8-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-73ed3b6945714f20bc893d361d1fcaf5) 7 #預設啟動的順序是從0開始，新核心是從頭插入（目前位置在0，而4.4.4的是在1），是以需要選擇0。
grub2-set-default 0
#重新開機
reboot