天天看点

使用自定义指标进行Pod弹性伸缩

Horizontal Pod Autoscaling,简称HPA,是Kubernetes中实现Pod水平自动伸缩功能的资源对象。Kubernetes 1.9 或更高版本支持HPA V2版本,允许我们是用自定义监控指标来控制Pod数目。

本文用阿里云容器服务来介绍HPA使用自定义指标进行弹性伸缩的功能。

安装配置Prometheus和Prometheus adaptor

我们使用Prometheus收集指标,通过Prometheus adaptor来扩展Kubernetes 自定义监控指标。

使用阿里云容器服务控制台创建好集群,并创建monitoring命名空间。然后进行如下步骤:

安装Prometheus

在应用目录-App Hub中搜索prometheus可以看到如下组件

使用自定义指标进行Pod弹性伸缩

点击prometheus进入安装页面,点击参数进行查看并配置(本文persistentVolume enabled设置为false,即使用emptyDir),选择集群命名空间后点击安装。

使用自定义指标进行Pod弹性伸缩
安装Prometheus adaptor

安装过程同上,注意配置参数中prometheus url要使用之前部署好的prometheus地址

使用自定义指标进行Pod弹性伸缩
验证安装是否成功

使用控制台的cloudshell或kubectl连接集群,执行如下命令:

kubectl api-versions           

结果中可以看到如下内容,表明集群支持hpa v2并可以使用自定义监控指标

autoscaling/v2beta1
custom.metrics.k8s.io/v1beta1           

我们可以进一步查看监控项,如pod支持的监控指标

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" |jq . | grep "pods/"           
"name": "pods/start_time_seconds",
      "name": "pods/cpu_user",
      "name": "pods/fs_limit_bytes",
      "name": "pods/cpu_cfs_throttled",
      "name": "pods/memory_failcnt",
      "name": "pods/fs_writes_bytes",
      "name": "pods/kube_pod_created",
      "name": "pods/kube_pod_status_phase",
      "name": "pods/tasks_state",
      "name": "pods/last_seen",
      "name": "pods/kube_pod_container_status_running",
      "name": "pods/fs_reads_merged",
      "name": "pods/memory_swap",
      "name": "pods/spec_cpu_quota",
      "name": "pods/fs_io_current",
      "name": "pods/kube_pod_container_resource_requests_memory_bytes",
      "name": "pods/kube_pod_container_status_waiting",
      "name": "pods/fs_sector_reads",
      "name": "pods/kube_pod_status_scheduled",
      "name": "pods/kube_pod_container_status_waiting_reason",
      "name": "pods/fs_reads_bytes",
      "name": "pods/kube_pod_container_resource_requests_cpu_cores",
      "name": "pods/kube_pod_container_status_terminated_reason",
      "name": "pods/cpu_system",
      "name": "pods/fs_read",
      "name": "pods/fs_writes",
      "name": "pods/kube_pod_owner",
      "name": "pods/memory_mapped_file",
      "name": "pods/spec_cpu_shares",
      "name": "pods/kube_pod_container_resource_limits_cpu_cores",
      "name": "pods/cpu_usage",
      "name": "pods/kube_pod_container_resource_limits",
      "name": "pods/kube_pod_info",
      "name": "pods/memory_working_set_bytes",
      "name": "pods/fs_writes_merged",
      "name": "pods/memory_usage_bytes",
      "name": "pods/kube_pod_container_resource_requests",
      "name": "pods/fs_io_time_weighted",
      "name": "pods/spec_cpu_period",
      "name": "pods/spec_memory_reservation_limit_bytes",
      "name": "pods/cpu_cfs_throttled_periods",
      "name": "pods/kube_pod_container_resource_limits_memory_bytes",
      "name": "pods/cpu_cfs_periods",
      "name": "pods/kube_pod_status_ready",
      "name": "pods/kube_pod_container_status_terminated",
      "name": "pods/memory_failures",
      "name": "pods/memory_rss",
      "name": "pods/spec_memory_swap_limit_bytes",
      "name": "pods/memory_cache",
      "name": "pods/kube_pod_start_time",
      "name": "pods/fs_io_time",
      "name": "pods/fs_reads",
      "name": "pods/fs_inodes_free",
      "name": "pods/kube_pod_container_status_ready",
      "name": "pods/fs_sector_writes",
      "name": "pods/cpu_load_average_10s",
      "name": "pods/spec_memory_limit_bytes",
      "name": "pods/kube_pod_labels",
      "name": "pods/kube_pod_container_status_restarts",
      "name": "pods/kube_pod_container_info",
      "name": "pods/fs_usage_bytes",
      "name": "pods/kube_pod_completion_time",
      "name": "pods/kube_pod_container_status_last_terminated_reason",
      "name": "pods/kube_pod_status_scheduled_time",
      "name": "pods/memory_max_usage_bytes",
      "name": "pods/fs_inodes",
      "name": "pods/fs_write",           

部署Deployment

在容器服务控制台中点击无状态->使用镜像创建,创建并部署一个nginx deployment,配置如下

使用自定义指标进行Pod弹性伸缩

在访问设置中配置服务

使用自定义指标进行Pod弹性伸缩

创建成功后可以在无状态列表中找到名为nginx的deployment,点击详情可以查看到相关信息

使用自定义指标进行Pod弹性伸缩

创建HPA

在上述详情页中点击容器组水平伸缩器,点击创建,配置相关内容(指标使用pods,参考上述自定义监控指标内容填写一项,这里使用memory_usage_bytes,阈值10485760=10M)

使用自定义指标进行Pod弹性伸缩

创建成功后可以在列表中查看到nginx-hpa

使用自定义指标进行Pod弹性伸缩

测试并触发自动伸缩

使用压测工具

ab

进行压测

使用自定义指标进行Pod弹性伸缩
for a in `seq 1 50`; do ab -rSqd -c 100 -n 2000 http://ng.cfc7d1315bdd14d529a7f102c2163736d.cn-beijing.alicontainer.com/;done           

等待一段时间,可以看到pod数量的变化

使用自定义指标进行Pod弹性伸缩

总结

通过 Prometheus 来监控应用程序并暴露出用于弹性伸缩的指标,使用HPA利用这些监控指标对应用进行弹性伸缩,可以应对一些突发状况,提高应用的可用性。