天天看點

elasticsearch api client使用

一、基本概念

1、Node 與 Cluster

Elastic 本質上是一個分布式資料庫,允許多台伺服器協同工作,每台伺服器可以運作多個 Elastic 執行個體。

單個 Elastic 執行個體稱為一個節點(node)。一組節點構成一個叢集(cluster)。

2、Index

Elastic 會索引所有字段,經過處理後寫入一個反向索引(Inverted Index)。查找資料的時候,直接查找該索引。

是以,Elastic 資料管理的頂層機關就叫做 Index(索引)。它是單個資料庫的同義詞。每個 Index (即資料庫)的名字必須是小寫。

下面的指令可以檢視目前節點的所有 Index。

$ curl -X GET 'http://localhost:9200/_cat/indices?v'      
elasticsearch api client使用

3、Document

Index 裡面單條的記錄稱為 Document(文檔)。許多條 Document 構成了一個 Index。

Document 使用 JSON 格式表示,下面是一個例子。

{
    "user": "張三",
    "title": "工程師",
    "desc": "資料庫管理"}      

同一個 Index 裡面的 Document,不要求有相同的結構(scheme),但是最好保持相同,這樣有利于提高搜尋效率。

4、Type

Document 可以分組,比如weather這個 Index 裡面,可以按城市分組(北京和上海),也可以按氣候分組(晴天和雨天)。這種分組就叫做 Type,它是虛拟的邏輯分組,用來過濾 Document。

不同的 Type 應該有相似的結構(schema),舉例來說,id字段不能在這個組是字元串,在另一個組是數值。這是與關系型資料庫的表的一個差別。性質完全不同的資料(比如products和logs)應該存成兩個 Index,而不是一個 Index 裡面的兩個 Type(雖然可以做到)。

下面的指令可以列出每個 Index 所包含的 Type。

$ curl 'localhost:9200/_mapping?pretty=true'      

根據規劃,Elastic 6.x 版隻允許每個 Index 包含一個 Type,7.x 版将會徹底移除 Type。

以上部分摘自:​​http://www.ruanyifeng.com/blog/2017/08/elasticsearch.html​​

二、通過Http請求排序

1、資料格式

{
    "_scroll_id": "DnF1ZXJ5VGhlbkZldGNoAwAAAAAATaBwFklfYTRhdy0wVHJxQUNpcm5sWVBHeHcAAAAAAEvhqhYwNTgtVi1xT1FUNlkxMl9CVldWM1lnAAAAAACXzBgWVlhBRnRfd2xRd09HdlduY2tRNXpmQQ==",
    "took": 3,
    "timed_out": false,
    "_shards": {
        "total": 3,
        "successful": 3,
        "failed": 0},
    "hits": {
        "total": 9564,
        "max_score": 1,
        "hits": [
            {
                "_index": "alert-201712s",
                "_type": "HISTORY",
                "_id": "000E94E15DA381A680F9C0E0C14F1E7F-1513323398",
                "_score": 1,
                "_source": {
                    "duration": 120,
                    "times": 2,
                    "status": "resolve",
                    "level": "warning",
                    "project": "AAAA"}
            },
            {
                "_index": "alert-201712s",
                "_type": "HISTORY",
                "_id": "00A70A194DCF6DE937BC97610715DDCE-1513320277",
                "_score": 1,
                "_source": {
                    "duration": 120,
                    "times": 54,
                    "level": "critical",
                    "project": "BBBB"}
            },
            ..........
        ]
    }
}      

想要先按照project聚合,再按照level聚合,再把聚合後的各個項目、各個level的duration求和(類似與sql中的select sum(duration) ….group by project,level)

2、聚合排序

通過postman請求:

請求方式:Post

url:

ip:9200/index名稱/Type名稱/_search      

此處應該是:

localhost:9200/alert-201712s/HISTORY/_search      

body參數:

{
    "size": 0,
    "query": {
        "bool": {
            "filter": {
                "terms": {
                    "project": ["AAAA", "BBBB"] }
            }
        }
    },
    "aggs": {
        "projects": {
            "terms": {
                "field": "project",
                "size": 10000},
            "aggs": {
                "levels": {
                    "terms": { "field": "level"},
                    "aggs": { "durations": { "sum": { "field": "duration"} } } }
            }
        }
    }
}      

body參數注意aggs的嵌套結構(層級)

查詢結果:

{
   "took": 3,
    "timed_out": false,
    "_shards": {
        "total": 3,
        "successful": 3,
        "failed": 0},
    "hits": {
        "total": 8768,
        "max_score": 0,
        "hits": []
    },
    "aggregations": {
        "types_count": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
                {
                    "key": "AAA",
                    "doc_count": 2077,
                    "types_count": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [ { "key": "serious", "doc_count": 789, "durations": { "value": 18720} }, { "key": "null", "doc_count": 456, "durations": { "value": 23} }, { "key": "warning", "doc_count": 401, "durations": { "value": 234} }, { "key": "critical", "doc_count": 4, "durations": { "value": 78} } ] }
                },
                {
                    "key": "BBB",
                    "doc_count": 1225,
                    "types_count": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [ { "key": "serious", "doc_count": 966, "durations": { "value": 56} }, { "key": "null", "doc_count": 258, "durations": { "value": 34} }, { "key": "critical", "doc_count": 1, "durations": { "value": 2343} } ] }
                }
    }
}      

三、java http 請求

1、pom依賴

<dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>5.6.4</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>transport</artifactId>
            <version>5.1.1</version>
        </dependency>      

2、代碼

import com.google.gson.Gson;
import com.google.gson.JsonObject;
import org.apache.http.Header;
import org.apache.http.HttpHost;
import org.apache.http.entity.StringEntity;
import org.apache.http.message.BasicHeader;
import org.apache.http.util.EntityUtils;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.client.Response;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.transport.TransportClient;

import java.util.HashMap;
import java.util.Map;

public class Test {

    private static String es_url = "localhost:9200";
    private TransportClient client;
    private IndexRequest source;

    //将postman中參數直接複制到idea中自動轉義的
     private static String str = "{\n" +
            "\t\"size\": 0,\n" +
            "\t\"query\": {\n" +
            "\t\t\"bool\": {\n" +
            "\t\t\t\"filter\": {\n" +
            "\t\t\t\t\"terms\": {\n" +
            "\t\t\t\t\t\"project\": [\"AA\",\n" +
            "\t\t\t\t\t\"BB\"]\n" +
            "\t\t\t\t}\n" +
            "\t\t\t}\n" +
            "\t\t}\n" +
            "\t},\n" +
            "\t\"aggs\": {\n" +
            "\t\t\"projects\": {\n" +
            "\t\t\t\"terms\": {\n" +
            "\t\t\t\t\"field\": \"project\",\n" +
            "\t\t\t\t\"size\": 10000\n" +
            "\t\t\t},\n" +
            "\t\t\t\"aggs\": {\n" +
            "\t\t\t\t\"levels\": {\n" +
            "\t\t\t\t\t\"terms\": {\n" +
            "\t\t\t\t\t\t\"field\": \"level\",\n" +
            "\t\t\t\t\t\t\"size\": 10000\n" +
            "\t\t\t\t\t},\n" +
            "\t\t\t\t\t\"aggs\": {\n" +
            "\t\t\t\t\t\t\"durations\": {\n" +
            "\t\t\t\t\t\t\t\"sum\": {\n" +
            "\t\t\t\t\t\t\t\t\"field\": \"duration\"\n" +
            "\t\t\t\t\t\t\t}\n" +
            "\t\t\t\t\t\t}\n" +
            "\t\t\t\t\t}\n" +
            "\t\t\t\t}\n" +
            "\t\t\t}\n" +
            "\t\t}\n" +
            "\t}\n" +
            "}";


    public static void main(String[] args) throws Exception {
        HttpHost[] hosts = new HttpHost[1];
        hosts[0] = HttpHost.create(es_url);
        //建立ES請求用戶端
        RestClient restClient = RestClient.builder(hosts).build();
        String index = "alert-201712s";
        String type = "HISTORY";
        String endpoint = "/" + index + "/" + type + "/_search";
        Map params = new HashMap();
        StringEntity queryBody = new StringEntity(str, "UTF-8");
        Header header = new BasicHeader("content-type", "application/json");

        Response response = restClient.performRequest("GET", endpoint, params, queryBody, header);
        //System.out.println(response);

        String resultJson = EntityUtils.toString(response.getEntity());
        Gson gson = new Gson();
        //擷取到傳回的資料
        JsonObject resultObj = gson.fromJson(resultJson, JsonObject.class);

    }

}      

個人微信公衆号: