prometheus专题—(十二) 采集job和instance

instance

- 用Prometheus术语来说，可以抓取的端点称为实例 instance
      

job

- 具有相同目的的实例的集合（例如，出于可伸缩性或可靠性而复制的过程）称为job

## 

举例
  - job_name: 'pushgateway'
    honor_timestamps: true
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    static_configs:
    - targets:
      - 172.20.70.205:9091
      - 172.20.70.205:9092
      - 172.20.70.215:9091

      

自动生成的标签和时间序列

当Prometheus抓取目标时，它会自动在抓取的时间序列上附加一些标签，以识别被抓取的目标：

- job：目标所属的已配置作业名称。
- instance：<host>:<port>抓取的目标网址的一部分。
- up{job="<job-name>", instance="<instance-id>"}：1实例是否正常（即可达）或0刮取失败。

- - - 设置告警查看采集失败的实例 `up==0`

- scrape_duration_seconds{job="<job-name>", instance="<instance-id>"}：刮擦的耗时

-       

举例

scrape_duration_seconds{instance="172.20.70.205", job="blackbox-ssh"} 0.001817932
scrape_duration_seconds{instance="172.20.70.205:3000", job="single-targets"} 0.005416658
scrape_duration_seconds{instance="172.20.70.205:9091", job="pushgateway"} 0.002726714
scrape_duration_seconds{instance="172.20.70.205:9092", job="pushgateway"} 0.000506256
scrape_duration_seconds{instance="172.20.70.205:9100", job="single-targets"} 0.012790691
scrape_duration_seconds{instance="172.20.70.205:9104", job="single-targets"} 0.021421043
scrape_duration_seconds{instance="172.20.70.205:9115", job="blackbox-http-targets"} 0.00427973      

用途：统计job中采集比较耗时的instance ,


- 为什么慢
  - 网络质量
  - metrics数据量太大
  - prometheus采集端有瓶颈了，需要扩容
- 上次采集最慢的五个 job+instance topk(5,scrape_duration_seconds)
- 采集时间超过3秒的 scrape_duration_seconds > 3



- scrape_samples_post_metric_relabeling{job="<job-name>", instance="<instance-id>"}：relabel之后剩余的重新标记后剩余的样本数
  - 何为样本：简单理解就是 标签组唯一 
- scrape_samples_scraped{job="<job-name>", instance="<instance-id>"}：目标暴露的样本数      

举例 topk(5,scrape_samples_scraped)

scrape_samples_scraped{instance="172.20.70.205:9256", job="single-targets"} 1691
scrape_samples_scraped{instance="172.20.70.215:9256", job="single-targets"} 1010
scrape_samples_scraped{instance="172.20.70.205:9104", job="single-targets"} 816
scrape_samples_scraped{instance="172.20.70.215:9100", job="single-targets"} 500
scrape_samples_scraped{instance="172.20.70.205:9100", job="single-targets"} 500      

• 用途： 统计样本数量按 job+instance分类

按job排序 topk(5,sum(scrape_samples_scraped) by (job))

{job="single-targets"} 4957
{job="redis_exporter_targets"} 299
{job="pushgateway"} 102
{job="blackbox-http-targets"} 72
{job="blackbox-ssh"} 6      

- scrape_series_added{job="<job-name>", instance="<instance-id>"}：此抓取中新系列的大概数量。v2.10的新功能
  - 用途 统计新增的metrics，可以用来查看写峰
  - 大部分情况应该都是旧的metrics append写入

#       

prometheus特殊tag说明

- __address__ 采集endpoint的地址
- __name__   metrics 的名称
- instance   endpoint最后的tag
- job         任务
- __metrics_path__  采集的http path 如 /metrics  /cadvisor/metrics