ELKStack å®æä¹ Elasticsearch [ä¸]
ELKStack å®æä¹ Elasticsearch [ä¸]
ELKStack
ELKStackç®ä»
ELK Stack æ¯ ElasticsearchãLogstashãKibana ä¸ä¸ªå¼æºè½¯ä»¶çç»åãå¨å®æ¶æ°æ®æ£ç´¢ååæåºåï¼ä¸è é常æ¯é åå ±ç¨ï¼èä¸åé½å åå½äº
Elastic.co
å ¬å¸åä¸ï¼æ ææ¤ç®ç§°ã
ELK Stack å¨æè¿ä¸¤å¹´è¿ éå´èµ·ï¼æ为æºå¨æ°æ®åæï¼æè 说å®æ¶æ¥å¿å¤çé¢åï¼å¼æºçç第ä¸éæ©ãåä¼ ç»çæ¥å¿å¤çæ¹æ¡ç¸æ¯ï¼ELK Stack å ·æå¦ä¸å 个ä¼ç¹ï¼
å¤çæ¹å¼çµæ´»ãElasticsearch æ¯å®æ¶å ¨æç´¢å¼ï¼ä¸éè¦å storm é£æ ·é¢å ç¼ç¨æè½ä½¿ç¨ï¼ Â
 é ç½®ç®æä¸æãElasticsearch å ¨é¨éç¨ JSON æ¥å£ï¼Logstash æ¯ Ruby DSL 设计ï¼é½æ¯ç®åä¸çæéç¨çé ç½®è¯æ³è®¾è®¡ï¼ Â
 æ£ç´¢æ§è½é«æãè½ç¶æ¯æ¬¡æ¥è¯¢é½æ¯å®æ¶è®¡ç®ï¼ä½æ¯ä¼ç§ç设计åå®ç°åºæ¬å¯ä»¥è¾¾å°å ¨å¤©æ°æ®æ¥è¯¢çç§çº§ååºï¼ Â
 é群线æ§æ©å±ãä¸ç®¡æ¯ Elasticsearch é群è¿æ¯ Logstash é群é½æ¯å¯ä»¥çº¿æ§æ©å±çï¼ Â
 å端æä½ç«ä¸½ãKibana çé¢ä¸ï¼åªéè¦ç¹å»é¼ æ ï¼å°±å¯ä»¥å®ææç´¢ãèååè½ï¼çæç«ä¸½ç仪表æ¿ã
ELKå°åï¼https://www.elastic.co/
Logstash æä½³å®è·µï¼http://udn.yyuap.com/doc/logstash-best-practice-cn/index.html
Elasticsearch æå¨æåï¼http://www.learnes.net/index.html
ELKStackä¸æ社åºï¼https://kibana.logstash.es/content/
ãã对äºæ¥å¿æ¥è¯´ï¼æ常è§çéæ±å°±æ¯æ¶éãåå¨ãæ¥è¯¢ãå±ç¤ºï¼å¼æºç¤¾åºæ£å¥½æç¸å¯¹åºçå¼æºé¡¹ç®ï¼logstashï¼æ¶éï¼ãelasticsearchï¼åå¨+æç´¢ï¼ãkibanaï¼å±ç¤ºï¼ï¼æ们å°è¿ä¸ä¸ªç»åèµ·æ¥çææ¯ç§°ä¹ä¸ºELKStackï¼æ以说ELKStackæçæ¯Elasticsearch(java)ãLogstash(jruby)ãKibanaææ¯æ çç»åï¼ä¸ä¸ªé ç¨çæ¶æå¦ä¸å¾æ示ï¼
![](https://img.laitimes.com/img/_0nNw4CM6IyYiwiM6ICdiwiIn5GcukjdyFWNpZ3avFmdr5WMzV3NxkmdkV2MiJWMfhGdhB3YpB3X39FcvwlZ1JTanZTd5lXMykmd6dGdxZXc5RWe6R2LcJXZrN2bkNmYh9CXt92Yu8WdsVnY5pnLjlGdhR3cvw1LcpDc0RHaiojIsJye.png)
å¾ç解éï¼elk åé¢ä¸»è¦é logstashæ¥è¿è¡æ¶éæ¥å¿ï¼logstashå°æ¥å¿ä¸ä¼ å°brokerä¸ï¼åé¢è¿æä¸ä¸ªlogstashç¨æ¥è¯»åbrokerä¸çæ¥å¿ï¼å¨å°æ¥å¿åå¨å°eséé¢ï¼æåç¨kibanaç»å°esä¸è¿è¡å±ç¤º
æ示ï¼æ们å¯ä»¥å°Elasticsearchç®ç§°ä¸ºes
Elasticsearchä»ç»
ElasticSearchæ¯ä¸ä¸ªåºäº
Lucene
çæç´¢æå¡å¨ãå®æä¾äºä¸ä¸ªåå¸å¼å¤ç¨æ·è½åçå ¨ææç´¢å¼æï¼åºäºRESTful webæ¥å£ãElasticsearchæ¯ç¨Javaå¼åçï¼å¹¶ä½ä¸ºApache许å¯æ¡æ¬¾ä¸çå¼æ¾æºç åå¸ï¼æ¯ç¬¬äºæµè¡çä¼ä¸æç´¢å¼æã设计ç¨äºäºè®¡ç®ä¸ï¼è½å¤è¾¾å°å®æ¶æç´¢ï¼ç¨³å®ï¼å¯é ï¼å¿«éï¼å®è£ 使ç¨æ¹ä¾¿ã
Elasticsearché¨ç½²
ããElasticsearché¦å éè¦Javaç¯å¢ï¼æ以éè¦æåå®è£ 好JDKï¼å¯ä»¥ç´æ¥ä½¿ç¨yumå®è£ ãä¹å¯ä»¥ä»Oracleå®ç½ä¸è½½JDKè¿è¡å®è£ ãå¼å§ä¹åè¦ç¡®ä¿JDKæ£å¸¸å®è£ 并ä¸ç¯å¢åéä¹é ç½®æ£ç¡®ï¼
ç¯å¢ä»ç»
[[email protected] ~]#
[[email protected] ~]# df -hFilesystem Size Used Avail Use% Mounted on/dev/sda1 20G 2.4G 18G 12% /devtmpfs 903M 0 903M 0% /devtmpfs 912M 0 912M 0% /dev/shmtmpfs 912M 8.6M 904M 1% /runtmpfs 912M 0 912M 0% /sys/fs/cgrouptmpfs 183M 0 183M 0% /run/user/0
[[email protected] ~]# free -m total used free shared buff/cache availableMem: 1823 328 1113 8 381 1322Swap: 0 0 0[[email protected] ~]# cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) æ¬æ¬¡ä½¿ç¨2å°æå¡å¨æ¥è¿è¡æ¨¡æé群ï¼æ以请åå¤2å°æå¡å¨
å®è£ JDK
[[email protected] ~]# yum install -y java
[[email protected] ~]# java -versionopenjdk version "1.8.0_65"OpenJDK Runtime Environment (build 1.8.0_65-b17)OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
æºç å®è£ JDK
ä¸è½½å®è£
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.htmlé
ç½®Javaç¯å¢# tar zxf jdk-8u91-linux-x64.tar.gz -C /usr/local/# ln âs /usr/local/jdk1.8.0_91 /usr/local/jdk# vim /etc/profileexport JAVA_HOME=/usr/local/jdkexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport PATH=$PATH:$JAVA_HOME/bin# source /etc/profileçå°å¦ä¸ä¿¡æ¯ï¼javaç¯å¢é
ç½®æå# java -versionjava version "1.8.0_91"Java(TM) SE Runtime Environment (build 1.8.0_91-b14)Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
é ç½®å®è£ ElasticSearch
å¯ä»¥ä½¿ç¨æºç ï¼æè yum
ä¸ãyumå®è£
1.ä¸è½½å¹¶å®è£
GPG key[[email protected] ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch2.æ·»å yumä»åº[[email protected] ~]# cat /etc/yum.repos.d/elasticsearch.repo[elasticsearch-2.x]name=Elasticsearch repository for 2.x packagesbaseurl=http://packages.elastic.co/elasticsearch/2.x/centosgpgcheck=1gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearchenabled=13.å®è£
elasticsearch[[email protected] ~]# yum install -y elasticsearch
äºãæºç å®è£
[[email protected] home]# wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.tar.gz #ä¸è½½elasticsearch[[email protected] home]# tar xf elasticsearch-1.7.2.tar.gz -C /usr/local/ #解åelasticsearch[[email protected] home]# ln -s /usr/local/elasticsearch-1.7.2 /usr/local/elasticsearch #å建软è¿æ¥
ä¸ãé ç½®elasticsearch
ä¿®æ¹é
ç½®æ件[[email protected] ~]# vim /etc/elasticsearch/elasticsearch.ymlcluster.name: myes ã #ESé群å称node.name: abcdocker-node-1 ã#èç¹å称path.data: /data/es-date #æ°æ®åå¨çç®å½ï¼å¤ä¸ªç®å½ä½¿ç¨éå·åéï¼path.logs: /var/log/elasticsearch #æ¥å¿æ ¼å¼bootstrap.memory_lock: true #éä½eså
åï¼ä¿è¯å
åä¸åé
è³äº¤æ¢ååºnetwork.host: 192.168.56.11 #设置æ¬æºIPå°åhttp.port: 9200 #端å£é»è®¤9200
åã设置æ°æ®ç®å½æé
[[email protected] ~]# chown -R elasticsearch:elasticsearch /data/es-date/#è¿ä¸ªæ¯æ们åæ¾æ°æ®çç®å½ï¼æå¨å建
esé»è®¤åç°æç»æååæï¼ç»æå°±æ¯é½å å ¥å°ä¸ä¸ªç»éé¢ï¼åæå°±æ¯ä¸å¯¹ä¸éä¿¡
æç¤ºï¼ yumå®è£ æ¶ä¼å°äºè®¸å¤é ç½®ï¼å¦æç¼è¯å®è£ å°±ä¼äº§çå¾å¤ã
äºãå¯å¨
å¯å¨es[[email protected] ~]# systemctl start elasticsearch.service[[email protected] ~]# netstat -lntupActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 532/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 724/master tcp6 0 0 192.168.56.11:9200 :::* LISTEN 2125/java tcp6 0 0 192.168.56.11:9300 :::* LISTEN 2125/java tcp6 0 0 :::22 :::* LISTEN 532/sshd tcp6 0 0 ::1:25 :::* LISTEN 724/master 端å£é»è®¤ï¼9200
æ¬æ¬¡ç¯å¢æ们使ç¨2å°æå¡å¨ï¼è¿2å°æå¡å¨çæå¡æ建å¯ä»¥è·ä¸é¢çæ¥éª¤ç¸åå³å¯
访é®æµè¯ï¼http://IP:9200
Elasticsearchæä¾äºé常å¤çæ件ï¼è¿å¯ä»¥ä½¿ç¨
curl
è¿è¡é讯
æ们å¯ä»¥ä½¿ç¨curlæ¥æ¥çeséé¢æä»ä¹å 容
[[email protected] ~]# curl -i -XGET 'http://192.168.56.11:9200/_count?'HTTP/1.1 200 OKContent-Type: application/json; charset=UTF-8Content-Length: 59{"count":0,"_shards":{"total":0,"successful":0,"failed":0}}[解éï¼è¿å头é¨200ï¼æ§è¡æå0个ï¼è¿å0个
Elasticsearchæå¾å¤æ件ï¼ä½æ¯æçæ件好ç¨ä½æ¯æ¶è´¹
Elasticsearchæ件ä»ç»
ä¸ãHaedæ件
æ件ä½ç¨ï¼ä¸»è¦æ¯åesé群管ççæ件
Githubä¸è½½å°åï¼https://github.com/mobz/elasticsearch-head
ä¸è½½æ¹å¼ï¼[[email protected] ~]# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head-> Installing mobz/elasticsearch-head...Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...Downloading ...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONEVerifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)Installed head into /usr/share/elasticsearch/plugins/headä¸è½½å®ä¼å¨/usr/share/elasticsearch/plugins/ç®å½ä¸äº§çæ件ç®å½
访é®ï¼http://ipå°å:9200/_plugin/head/
äºãBigdeskæ件
æ件ä½ç¨ï¼æ§è½çæ§
Githubä¸è½½ï¼https://github.com/lukas-vlcek/bigdesk
æ示ï¼å 为æ们使ç¨yumå®è£ çææ°çæ¬ï¼bigdeskææ¶ä¸æ¯æææ°çæ¬
[[email protected] ~]# /usr/share/elasticsearch/bin/elasticsearch -VOpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=NVersion: 2.4.4, Build: fcbb46d/2017-01-03T11:33:16Z, JVM: 1.8.0_121
æ£å¸¸è®¿é®å°åï¼http://localhost:9200/_plugin/bigdesk/
Bigdesk 1.x
Bigdesk 2.X
ä¸ãKopfæ件
æ件ä½ç¨ï¼kopfæ¯ä¸ä¸ªç®åçç½ç»ç®¡çå·¥å ·
Kopfä¸åç»´æ¤ãå·²ç»å¼åäºæ¿ä»£åï¼cerebroï¼ï¼ç®åç»´æ¤å¨https://github.com/lmenezes/cerebroãå¨è¿ä¸ç¹ä¸ï¼cerebroåºè¯¥æç¸å½äºkopfçåè½ï¼é¡¶é¨æä¸äºæ°çåè½ã
Githubå°åï¼https://github.com/lmenezes/elasticsearch-kopf
å®è£
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf-> Installing lmenezes/elasticsearch-kopf...Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip ...Downloading ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONEVerifying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip checksums if available ...NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)Installed kopf into /usr/share/elasticsearch/plugins/kopf
访é®å°åï¼http://192.168.56.11:9200/_plugin/kopf/
æ件使ç¨
1ãHeadæ件使ç¨ä»ç»
ç°å¨æ们已ç»å°æ件å®è£ å®æï¼è¿æ¶åéé¢è¿æ²¡ææ°æ®ãæ们ç°å¨å¾éé¢åä¸äºæ°æ®
ç¹å»æ交
_indexï¼å建äºä¸ä¸ªç´¢å¼index-demo
_typeï¼å建äºä¸ä¸ªç±»åtest
totalï¼åç2个
Successfulï¼æå1个
Failedï¼å¤±è´¥0个
Createdï¼ç¶ææå
æ们ç¹å»æ¦è§ï¼ç¹å»è¿æ¥
æ们åä¸ç¯ææ¡£ä¼å¸®æ们åæ5çï¼0-4ï¼å¯ä»¥ä¿®æ¹æå¤ä¸ªï¼ï¼ç²çº¿ä»£è¡¨ä¸»åçç»çº¿ä»£è¡¨å¯æ¬åçï¼å¯ä»¥ç解为ä¸ä¸»ä¸å¤ï¼æ£å¸¸æ åµä¸ä¼å°ä¸»åçåå¯æ¬åçæ¾å¨2å°æºå¨ä¸
é群å¥åº·å¼ä»ç»ï¼
ÃÃÃ代表没æ主åçæ°æ®ä¸¢å¤±ï¼ä½æ¯ç°å¨ä¸æ¯å¥åº·çç¶æï¼è¦åï¼åºè¯¥æ10个åçï¼ç°å¨åªæ5个ã
红è²ä»£è¡¨ææ°æ®ä¸¢å¤±
绿è²ä»£è¡¨æ£å¸¸
æ示ï¼esæ¯æä¸ä¸ªç±»ä¼¼äºå¿«ç §çåè½ï¼æ¹ä¾¿æ们ç¨äºæ°æ®å¤ä»½
Esç´¢å¼ä¿¡æ¯
è¿éç´¢å¼ä¿¡æ¯æ¯open代表æ们å¯ä»¥å°å®ç»å ³éæï¼è¿æ ·å°±ä¸ä¼ç»§ç»ä½¿ç¨è¿ä¸ªç´¢å¼è¿è¡æç´¢
å ³éç´¢å¼
é¤äºæ¯æå ³éåå¼å¯ç´¢å¼ï¼åæ ·ä¹æ¯æå é¤
æ示ï¼å é¤ä¹åç´¢å¼æ°æ®æ¯æ æ³æ¢å¤ç
Headæ件å°ç¼ºç¹ï¼ å½æ们索å¼ç¹å«å¤çæ¶åï¼æå¼headè³å°éè¦äºåéãå 为å®è¦æææçç´¢å¼é½æ«æä¸éè¿è¡å±ç¤ºï¼è¿æ¶åæå¼ä½¿ç¨ç带宽ä¹ä¼ç¹å«å¤§ï¼ä¸ä¼åºç°è¶ æ¶ï¼ä¸ç´çå¾ å°±å¯ä»¥ï¼
2ãkopfæ件使ç¨ä»ç»
访é®å°åï¼http://192.168.56.11:9200/_plugin/kopf/
æ们å¯ä»¥çå°ç´¢å¼ãåçãææ¡£æ°éã大å°ä»¥åjvm使ç¨æ åµãdiskãcpuãloadç
æ们ç¹å»èç¹ï¼å°±å¯ä»¥çå°æ´è¯¦ç»ç硬件信æ¯ï¼å®æ¶ååï¼
åæ ·kopfä¹æ¯æheadçæ件ï¼æ¯æ请æ±
ï¼ç产åºæ¯è¿ä¸ªä¿¡æ¯ä¸»è¦ç»å¼å使ç¨ï¼
æ示ï¼esæ¯æèªå·±çæ¥è¯¢è¯è¨ï¼å¯ä»¥åçå¾å¤æã
å ¶ä»åè½
â² Create indexï¼ Â å建索å¼: Â
 ⲠCluster settingsï¼ é群设置: Â
 ⲠAliasesï¼ Â Â Â Â Â å«å:
 ⲠAnalysisï¼ Â Â åæ: Â
 ⲠPercolatorï¼ Â Â Â è¿æ»¤å¨: Â
 ⲠSnapshotï¼ Â Â Â Â å¿«ç §
 ⲠIndex templatesï¼ Â ç´¢å¼æ¨¡æ¿: Â
 ⲠCat apisï¼ Â Â Â Â Â Â api Â
 ⲠHot threadsï¼ Â Â Â ç线ç¨:
ESä»ç»å®æ
æ¬æåèï¼èµµçé¿ï¼èµµæ¢¦æ¢¦ï¼å´ä¾ç¸å ³æç«
转载äº:https://blog.51cto.com/abcdocker/1907687