目錄
文章目錄
- Kubernetes Metrics Server
- 啟用 API Aggregator
- 安裝 Metrics Server
Kubernetes Metrics Server 是 Cluster 的核心監控資料的聚合器,kubeadm 預設是不部署的。
Metrics Server 供 Dashboard 等其他元件使用,是一個擴充的 APIServer,依賴于 API Aggregator。是以,在安裝 Metrics Server 之前需要先在 kube-apiserver 中開啟 API Aggregator。
- Metrics API 隻可以查詢目前的度量資料,并不儲存曆史資料。
- Metrics API URI 為 /apis/metrics.k8s.io/,在 k8s.io/metrics 下維護。
- 必須部署 metrics-server 才能使用該 API,metrics-server 通過調用 kubelet Summary API 擷取資料。
使用 Metrics Server 有必備兩個條件:
- API Server 啟用 Aggregator Routing 支援。否則 API Server 不識别請求:
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)
- API Server 能通路 Metrics Server Pod IP。否則 API Server 無法通路 Metrics Server:
E1223 07:23:04.330206 1 available_controller.go:420] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.171.248.214:4443/apis/metrics.k8s.io/v1beta1: Get https://10.171.248.214:4443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
API Aggregation 允許在不修改 Kubernetes 核心代碼的同時擴充 Kubernetes API,即:将第三方服務注冊到 Kubernetes API 中,這樣就可以通過 Kubernetes API 來通路第三方服務了,例如:Metrics Server API。
注:另外一種擴充 Kubernetes API 的方法是使用 CRD(Custom Resource Definition,自定義資源定義)。
- 檢查 API Server 是否開啟了 Aggregator Routing:檢視 API Server 是否具有
選項。--enable-aggregator-routing=true
$ ps -ef | grep apiserver
root 23896 29500 0 12:40 pts/0 00:00:00 grep --color=auto apiserver
root 28613 28551 1 12月21 ? 01:05:29 kube-apiserver --advertise-address=192.168.0.112 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=172.16.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
- 修改每個 API Server 的 kube-apiserver.yaml 配置開啟 Aggregator Routing:修改 manifests 配置後會 API Server 會自動重新開機生效。
$ vi /etc/kubernetes/manifests/kube-apiserver.yaml
...
spec:
containers:
- command:
...
- --enable-aggregator-routing=true
- 檢查 Cluster 是否安裝了 Metrics Server:
$ kubectl top pods
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
- 部署 Metrics Server:
# 下載下傳 YAML 檔案
wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml
# 編輯修改 metrics-server 的啟動參數:
# --kubelet-insecure-tls 跳過 TLS 認證,否則會出現 x509 的認證問題,用于測試環境。
# --kubelet-preferred-address-types=InternalIP 使用 Node IP 進行通信。
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP
- --kubelet-use-node-status-port
- --kubelet-insecure-tls
# 部署
$ kubectl apply -f components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
注意:如果出現了 ErrImagePull 的問題,那麼意味着 k8s.gcr.io/metrics-server/metrics-server:v0.4.1 鏡像下載下傳失敗了:
$ docker pull k8s.gcr.io/metrics-server/metrics-server:v0.4.1
Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
$ docker pull bitnami/metrics-server:0.4.1
$ docker tag bitnami/metrics-server:0.4.1 k8s.gcr.io/metrics-server/metrics-server:v0.4.1
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
bitnami/metrics-server 0.4.1 4fb6df85a88d 6 hours ago 171MB
k8s.gcr.io/metrics-server/metrics-server v0.4.1 4fb6df85a88d 6 hours ago 171MB
- 檢查 Metrics Server Service:
$ kubectl get svc --all-namespaces | grep metrics-server
kube-system metrics-server ClusterIP 172.16.128.176 <none> 443/TCP 5h55m
- 檢查 API Server 是否可以連通 Metrics Server:
$ kubectl describe svc metrics-server -n kube-system
Name: metrics-server
Namespace: kube-system
Labels: k8s-app=metrics-server
Annotations: Selector: k8s-app=metrics-server
Type: ClusterIP
IP: 172.16.128.176
Port: https 443/TCP
TargetPort: https/TCP
Endpoints: 10.171.248.214:4443
Session Affinity: None
Events: <none>
# 在 Master Node 上 Ping。
$ ping 10.171.248.214
64 bytes from 10.171.248.214: icmp_seq=1 ttl=63 time=0.282 ms
- 檢查 Metrics Server:
$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master-1 174m 8% 1156Mi 66%
k8s-master-2 123m 6% 1134Mi 65%
k8s-master-3 104m 5% 1075Mi 61%
k8s-node-1 78m 3% 853Mi 49%
k8s-node-2 78m 3% 824Mi 47%