es搜索核心与实战Day08
一、分页于遍历
1.From/Size
- 默认情况下，查询按照相关度算分排序，返回前十条记录
- 容易理解的分页方案
-
- From：开始位置
- Size：期望获取文档的总数
2.Search After避免深度分页问题
- 避免深度分页的性能问题，可以实时获取下一页文档信息
-
- 不支持指定页数（From）
- 只能往下翻
- 第一步搜索需要指定sort，并保证值是唯一的（可以通过加入_id保证唯一性）
- 然后使用上一次，最后一个文档的sort值进行查询
POST tmdb/_search
{
"from": 0,
"size": 10,
"query": {
"match_all": {}
}
}
//Search After避免深度分页的问题
DELETE users
POST users/_doc
{"name":"user1","age":10}
POST users/_doc
{"name":"user2","age":11}
POST users/_doc
{"name":"user3","age":12}
POST users/_doc
{"name":"user4","age":13}
POST users/_count
POST users/_search
{
"size": 1,
"query": {
"match_all": {}
},
"sort": [
{"age":"desc"},
{"_id":"asc"}
]
}
POST users/_search
{
"size": 1,
"query": {
"match_all": {}
},
"search_after":
[
12,
"nVJs4HIBJjUASKqbxyMt"
]
,
"sort": [
{"age": "desc"},
{"_id":"asc"}
]
}
3.Scroll API
- 创建一个快照，有新的数据写入以后，无法被查到
- 每次查询后，输入上一次的Scroll Id
//Scroll API
DELETE users
POST users/_doc
{"name":"user1","age":10}
POST users/_doc
{"name":"user2","age":20}
POST users/_doc
{"name":"user3","age":30}
POST users/_doc
{"name":"user4","age":40}
POST users/_count
POST users/_doc
{"name":"user4","age":40}
POST /users/_search?scroll=5m
{
"size": 1,
"query": {
"match_all": {
}
}
}
POST /_search/scroll
{
"scroll":"1m",
"scroll_id":"DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAQDQWbk1XNFdQaE1TZWFneGJLQXAxUDhydw=="
}
4.不同的搜索类型和使用场景
- Regular
-
- 需要实时获取顶部的部分文档。例如查询最新的订单
- Scroll
-
- 需要全部文档，例如导出全部数据
- Pagination
-
- From和Size
- 如果需要深度分页，择选用Search After
二、处理并发读写操作
1.并发控制的必要性
- 两个Wed程序同时更新某个文档，如果缺乏有效的并发，会导致更改数据丢失
- 悲观的并发控制
-
- 假定有变更冲突的可能。会对资源加锁，防止冲突。例如数据库行锁
- 乐观并发控制
-
- 假定冲突是不会发生的，不会阻塞正在尝试的操作。如果数据在读写中被修改，更新将会失败。应用程序决定如何解决冲突，例如重试更新，使用新的数据，或者将错误报告给用户
- ES采用的是乐观并发
2.ES的乐观并发控制
- ES中的文档是不可变更的。如果你更新一个文档，会将旧文档标记为删除，同时增加一个全新的文档。同时文档的version字段加1
- 内部版本控制
-
- If_seq_no+If_primary_term
- 使用外部版本（使用其他数据库作为主要数据存储）
-
- version+version_type=external
//并发读写、并发锁
DELETE products
PUT products
PUT products/_doc/1
{
"title":"iphone",
"count":100
}
GET products/_doc/1
PUT products/_doc/1?if_seq_no=0&if_primary_term=1
{
"title":"iphone",
"count":103
}
GET products/_doc/1
PUT products/_doc/1?version=30001&version_type=external
{
"title":"iphone",
"count":100
}
三、Bucket&Metric聚合分析及嵌套聚合
1.Bucket&Metric Aggregation
- Metric - 一些系列的统计方法
- Bucket - 一组满足条件的文档
- Aggregation的语法
-
- 属于Search的一部分。一般情况下，建议将其Size指定为0
2.Metric Aggregation
- 单值分析：只输出一个分析结果
-
- min，max，，avg，sum
- Cardinality（类似distinct Count）
- 多值分析：输出多个分析结果
-
- stats，extended stats
- percentile，percentile rank
- top hite（排在前面的示例）
Demo
DELETE /employees
//定义员工表
PUT /employees/
{
"mappings" : {
"properties" : {
"age" : {
"type" : "integer"
},
"gender" : {
"type" : "keyword"
},
"job" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 50
}
}
},
"name" : {
"type" : "keyword"
},
"salary" : {
"type" : "integer"
}
}
}
}
//写入员工表
PUT /employees/_bulk
{ "index" : { "_id" : "1" } }
{ "name" : "Emma","age":32,"job":"Product Manager","gender":"female","salary":35000 }
{ "index" : { "_id" : "2" } }
{ "name" : "Underwood","age":41,"job":"Dev Manager","gender":"male","salary": 50000}
{ "index" : { "_id" : "3" } }
{ "name" : "Tran","age":25,"job":"Web Designer","gender":"male","salary":18000 }
{ "index" : { "_id" : "4" } }
{ "name" : "Rivera","age":26,"job":"Web Designer","gender":"female","salary":22000}
{ "index" : { "_id" : "5" } }
{ "name" : "Rose","age":25,"job":"QA","gender":"female","salary":18000 }
{ "index" : { "_id" : "6" } }
{ "name" : "Lucy","age":31,"job":"QA","gender":"female","salary": 25000}
{ "index" : { "_id" : "7" } }
{ "name" : "Byrd","age":27,"job":"QA","gender":"male","salary":20000 }
{ "index" : { "_id" : "8" } }
{ "name" : "Foster","age":27,"job":"Java Programmer","gender":"male","salary": 20000}
{ "index" : { "_id" : "9" } }
{ "name" : "Gregory","age":32,"job":"Java Programmer","gender":"male","salary":22000 }
{ "index" : { "_id" : "10" } }
{ "name" : "Bryant","age":20,"job":"Java Programmer","gender":"male","salary": 9000}
{ "index" : { "_id" : "11" } }
{ "name" : "Jenny","age":36,"job":"Java Programmer","gender":"female","salary":38000 }
{ "index" : { "_id" : "12" } }
{ "name" : "Mcdonald","age":31,"job":"Java Programmer","gender":"male","salary": 32000}
{ "index" : { "_id" : "13" } }
{ "name" : "Jonthna","age":30,"job":"Java Programmer","gender":"female","salary":30000}
{ "index" : { "_id" : "14" } }
{ "name" : "Marshall","age":32,"job":"Javascript Programmer","gender":"male","salary": 25000}
{ "index" : { "_id" : "15" } }
{ "name" : "King","age":33,"job":"Java Programmer","gender":"male","salary":28000 }
{ "index" : { "_id" : "16" } }
{ "name" : "Mccarthy","age":21,"job":"Javascript Programmer","gender":"male","salary": 16000}
{ "index" : { "_id" : "17" } }
{ "name" : "Goodwin","age":25,"job":"Javascript Programmer","gender":"male","salary": 16000}
{ "index" : { "_id" : "18" } }
{ "name" : "Catherine","age":29,"job":"Javascript Programmer","gender":"female","salary": 20000}
{ "index" : { "_id" : "19" } }
{ "name" : "Boone","age":30,"job":"DBA","gender":"male","salary": 30000}
{ "index" : { "_id" : "20" } }
{ "name" : "Kathy","age":29,"job":"DBA","gender":"female","salary": 20000}
# Metric 聚合，找到最低的工资
POST employees/_search
{
"size": 0,
"aggs": {
"min_salary": {
"min": {
"field":"salary"
}
}
}
}
# Metric 聚合，找到最高的工资
POST employees/_search
{
"size": 0,
"aggs": {
"max_salary": {
"max": {
"field":"salary"
}
}
}
}
# 多个 Metric 聚合，找到最低最高和平均工资
POST employees/_search
{
"size": 0,
"aggs": {
"max_salary": {
"max": {
"field": "salary"
}
},
"min_salary": {
"min": {
"field": "salary"
}
},
"avg_salary": {
"avg": {
"field": "salary"
}
}
}
}
# 一个聚合，输出多值
POST employees/_search
{
"size": 0,
"aggs": {
"stats_salary": {
"stats": {
"field":"salary"
}
}
}
}
3.Bucket Aggregation
- 按照一定的规则，将文档分配到不同的桶中，从而达到分类的目的。ES提供的一些常见的Bucket Aggregation
-
- Terms
- 数字类型
-
- Range/Data Range
- Histogram/Data Histogram
- 支持嵌套：也就在桶里再做分桶
4.Terms Aggregation
- 字段需要打开fielddata，才能进行Terms Aggregation
-
- Keyword默认支持fielddata
- Text需要在Mapping中enable。会按照分词后的结果进行分
- Demo
-
- 对job和job.keyword进行聚合
- 对性别进行Terms聚合
- 指定bucket size
# 对keword 进行聚合
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
//keyword不做分词
"field":"job.keyword"
}
}
}
}
# 对 Text 字段进行 terms 聚合查询，失败
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field":"job"
}
}
}
}
# 对 Text 字段打开 fielddata，支持terms aggregation
PUT employees/_mapping
{
"properties" : {
"job":{
"type": "text",
"fielddata": true
}
}
}
# 对job.keyword 和 job 进行 terms 聚合，分桶的总数并不一样
POST employees/_search
{
"size": 0,
"aggs": {
"cardinate": {
"cardinality": {
"field": "job"
}
}
}
}
# 对 性别的 keyword 进行聚合
POST employees/_search
{
"size": 0,
"aggs": {
"gender": {
"terms": {
"field":"gender"
}
}
}
}
#指定 bucket 的 size
POST employees/_search
{
"size": 0,
"aggs": {
"ages_5": {
"terms": {
"field":"age",
"size":3
}
}
}
}
# 指定size，不同工种中，年纪最大的3个员工的具体信息
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field":"job.keyword"
},
"aggs":{
"old_employee":{
"top_hits":{
"size":3,
"sort":[
{
"age":{
"order":"desc"
}
}
]
}
}
}
}
}
}
优化Terms聚合性能
打开配置聚合分析经常打开对性能要求索引有新的文档写入
5.Range&Histogram
- 按照数字的范围，进行分桶
- 在Range Aggregation中，可以自定义Key
- Demo
#Salary Ranges 分桶，可以自己定义 key
POST employees/_search
{
"size": 0,
"aggs": {
"salary_range": {
"range": {
"field":"salary",
"ranges":[
{
"to":10000
},
{
"from":10000,
"to":20000
},
{
"key":">20000",
"from":20000
}
]
}
}
}
}
#Salary Histogram,工资0到10万，以 5000一个区间进行分桶
POST employees/_search
{
"size": 0,
"aggs": {
"salary_histrogram": {
"histogram": {
"field":"salary",
"interval":5000,
"extended_bounds":{
"min":0,
"max":100000
}
}
}
}
}
# 嵌套聚合1，按照工作类型分桶，并统计工资信息
POST employees/_search
{
"size": 0,
"aggs": {
"Job_salary_stats": {
"terms": {
"field": "job.keyword"
},
"aggs": {
"salary": {
"stats": {
"field": "salary"
}
}
}
}
}
}
# 多次嵌套。根据工作类型分桶，然后按照性别分桶，计算工资的统计信息
POST employees/_search
{
"size": 0,
"aggs": {
"Job_gender_stats": {
"terms": {
"field": "job.keyword"
},
"aggs": {
"gender_stats": {
"terms": {
"field": "gender"
},
"aggs": {
"salary_stats": {
"stats": {
"field": "salary"
}
}
}
}
}
}
}
}
四、Pipeline聚合分析（通过bucket_path关键字指定路径）
1.Pipeline
- 管道的概念：支持对聚合分析的结果，再次进行聚合分析
- Pipeline的分析结果会输出到原有结果中，根据位置的不同，分为两类
-
- Sibling-结果和现有分析结果同级
-
- Max，min，Avg&Sum Bucket
- Stats，Extended Status Bucket
- Percentiles Bucket
- Parent-结果内嵌到现有的聚合分析结果之中
-
- Derivative（求导）
- Cumultive Sum（累计求和）
- Moving Function（滑动窗口）
#平均工资最低的工作类型
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword",
"size": 10
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
}
}
},
"min_salary_by_job":{
"min_bucket": {
"buckets_path": "jobs>avg_salary"
}
}
}
}
#平均工资最高的工作类型
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword",
"size": 10
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
}
}
},
"min_salary_by_job":{
"max_bucket": {
"buckets_path": "jobs>avg_salary"
}
}
}
}
#平均工资的平均工资
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword",
"size": 10
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
}
}
},
"avg_salary_by_job":{
"avg_bucket": {
"buckets_path": "jobs>avg_salary"
}
}
}
}
#平均工资的统计分析
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword",
"size": 10
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
}
}
},
"stats_salary_by_job":{
"stats_bucket": {
"buckets_path": "jobs>avg_salary"
}
}
}
}
#平均工资的百分位数
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword",
"size": 10
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
}
}
},
"percentiles_salary_by_job":{
"percentiles_bucket": {
"buckets_path": "jobs>avg_salary"
}
}
}
}
#按照年龄对平均工资求导
POST employees/_search
{
"size":0,
"aggs": {
"age": {
"histogram": {
"field": "age",
"min_doc_count": 1,
"interval": 1
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
},
"derivative_avg_salary":{
"derivative": {
"buckets_path": "avg_salary"
}
}
}
}
}
}
#Cumulative_sum（累计求和）
POST employees/_search
{
"size":0,
"aggs": {
"age": {
"histogram": {
"field": "age",
"min_doc_count": 1,
"interval": 1
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
},
"cumulative_salary":{
"cumulative_sum": {
"buckets_path":"avg_salary"
}
}
}
}
}
}
#Moving Function(移动平均)
POST employees/_search
{
"size":0,
"aggs": {
"age": {
"histogram": {
"field": "age",
"min_doc_count": 1,
"interval": 1
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
},
"moving_avg_salary":{
"moving_fn": {
"buckets_path":"avg_salary",
"window":10,
"script":"MovingFunctions.min(values)"
}
}
}
}
}
}
五、作用范围与排序
1.聚合的作用范围
- ES聚合分析的默认作用范围是query的查询结果集
- 同时ES还支持以下方式改变聚合的作用范围
-
- Filter
- Post Filter
- Global
#Query
POST employees/_search
{
"size": 0,
"query": {
"range": {
"age": {
//20
"gte": 40
}
}
},
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword"
}
}
}
}
#Filter
POST employees/_search
{
"size": 0,
"aggs": {
"order_person": {
"filter": {
"range": {
"age": {
"from":35
}
}
},
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword"
}
}
}},
"all_jobs":{
"terms":{
"field":"job.keyword"
}
}
}
}
#Post field,一条语句，找出所有的job类型。还能找到聚合后符合条件的结果
POST employees/_search
{
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword"
}
}
},
"post_filter": {
"match":{
"job.keyword":"Dev Manager"
}
}
}
#global
POST employees/_search
{
"size": 0,
"query": {
"range": {
"age": {
"gte": 40
}
}
},
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword"
}
},
"all":{
"global": {},
"aggs": {
"salary_avg": {
"avg": {
"field": "salary"
}
}
}
}
}
}
2.排序
- 指定order，按照count和key进行排序
-
- 默认情况，按照count降序排序
- 指定size，就能返回相应的桶
#排序 order
#count and key
POST employees/_search
{
"size": 0,
"query": {
"range": {
"age": {
"gte": 20
}
}
},
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword",
"order": [
//升序
{"_count":"asc"},
{"_key":"desc"}
]
}
}
}
}
六、聚合分析的原理及精准度问题
1.分布式系统的近似统计算法
2.Min聚合分析的执行流程
3.Terms Aggregation的返回值
- 在Terns Aggregation的返回中有两个特殊的数值
-
- doc_count_error_upper_bound:被遗漏的term分桶，包含的文档，有可能的最大值
- sum_other_doc_count:除了返回结果bucket的terms以外，其他terms的文档总数（总数-返回的总数）
4.Terms聚合分析的执行
5.如何解决Terms不准的问题：提升shard_size的参数
- Terms聚合分析不准的原因，数据分散在多个分片上，CoordinatingNode无法获取数据全貌
- 解决方法1：当数据量不大时，设置OrimaryShard为1；实现准确性
- 方案2：在分布式数据上，设置shard-size参数，提高精准度
-
- 原理：每次从Shard上额外多获取数据，提升准确率
6.shard_size设定
- 调整shard_size大小，降低doc_count_error_upper_bound来提升准确度
-
- 增加整体计算量，提高了准确度，但会降低相应时间
- Shard Size默认大小设定
-
- shard_size=size*1.5+10
POST _reindex
{
"source": {
"index": "kibana_sample_data_flights"
},
"dest": {
"index": "my_flights"
}
}
GET kibana_sample_data_flights/_count
GET my_flights/_count
get kibana_sample_data_flights/_search
GET kibana_sample_data_flights/_search
{
"size": 0,
"aggs": {
"weather": {
"terms": {
"field":"OriginWeather",
"size":5,
"show_term_doc_count_error":true
}
}
}
}
GET my_flights/_search
{
"size": 0,
"aggs": {
"weather": {
"terms": {
"field":"OriginWeather",
"size":1,
"shard_size":1,
"show_term_doc_count_error":true
}
}
}
}
![k8s部署es集群和kibana[图]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)