天天看点

k8s 搭建HPA和metric-server 问题排查

背景

在k8s里面搭建HPA时。HPA需要metric-server进行采集数据,在过程中猜到两个坑,经过最后的折腾总算是解决了问题。所以记录一下方便其他人在遇到问题时可以快速解决。

metric-server 修改日志级别查看问题

在metric-server搭建完成后,可以通过 kubectl logs -f podName 查看metric-server输出的日志。但是有的时候查看日志看不出来问题的话。可以调整一下日志级别。这样可以输出更多有信息的日志。如下图

–v 表示日志级别。数字越大,输出的日志越多。

k8s 搭建HPA和metric-server 问题排查

HPA not able to find CPU 问题排查

当HPA的event里面有如下输出日志的时候,这种问题是HPA的target的Deployment 管理的Pod没有设置CPU的request。设置就好了。

kubectl describe hpa hpa-demo
Name:                                                  hpa-demo
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Tue, 19 Nov 2019 17:14:56 +0800
Reference:                                             Deployment/hpa-demo
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 10%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: missing request for cpu
Events:
  Type     Reason                        Age                From                       Message
  ----     ------                        ----               ----                       -------
  Warning  FailedGetResourceMetric       14s (x4 over 60s)  horizontal-pod-autoscaler  missing request for cpu
  Warning  FailedComputeMetricsReplicas  14s (x4 over 60s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: missing request for cpu
           
k8s 搭建HPA和metric-server 问题排查