2015-10-26 1 閱讀在上周的上海Gopher Meetup的聚會(huì )上，聽(tīng)了 ASTA謝 的演講。然后公司最近也需要實(shí)現一個(gè)日志集中分析平臺。
ASTA謝 恰好也講了他使用了Elasticsearch + Logstash + Kibana這個(gè)組合進(jìn)行日志分析?；貋?lái)之后就買(mǎi)了一本書(shū)然后各種google把它配置好了，當然只是把框架搭好了。這三個(gè)組建還有很多功能并沒(méi)有熟悉。本文只是簡(jiǎn)單的介紹在Centos如果配置ELK(因為公司的服務(wù)器是Centos的,個(gè)人比較喜歡Ubuntu 哈哈)
什么是ELK：
Elasticsearch + Logstash + Kibana（ELK）是一套開(kāi)源的日志管理方案，分析網(wǎng)站的訪(fǎng)問(wèn)情況時(shí)我們一般會(huì )借助Google/百度/CNZZ等方式嵌入JS做數據統計，但是當網(wǎng)站訪(fǎng)問(wèn)異?；蛘弑还魰r(shí)我們需要在后臺分析如Nginx的具體日志，而Nginx日志分割/GoAccess/Awstats都是相對簡(jiǎn)單的單節點(diǎn)解決方案，針對分布式集群或者數據量級較大時(shí)會(huì )顯得心有余而力不足，而ELK的出現可以使我們從容面對新的挑戰。
Logstash：負責日志的收集，處理和儲存
Elasticsearch：負責日志檢索和分析
Kibana：負責日志的可視化
官方網(wǎng)站：
JDK –
http://www.oracle.com/technetwork/java/javase/downloads/index.html Elasticsearch –
https://www.elastic.co/downloads/elasticsearch Logstash –
https://www.elastic.co/downloads/logstash Kibana –
https://www.elastic.co/downloads/kibana Nginx-
https://www.nginx.com/ 服務(wù)端配置：
安裝Java JDK：
cat /etc/redhat-release//這是我linux的版本CentOS Linux release 7.1.1503 (Core) //我們通過(guò)yum 方式安裝Java Jdkyum install java-1.7.0-openjdk Elasticsearch安裝：
#下載安裝wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpmyum localinstall elasticsearch-1.7.1.noarch.rpm #啟動(dòng)相關(guān)服務(wù)service elasticsearch startservice elasticsearch status#查看Elasticsearch的配置文件rpm -qc elasticsearch/etc/elasticsearch/elasticsearch.yml/etc/elasticsearch/logging.yml/etc/init.d/elasticsearch/etc/sysconfig/elasticsearch/usr/lib/sysctl.d/elasticsearch.conf/usr/lib/systemd/system/elasticsearch.service/usr/lib/tmpfiles.d/elasticsearch.conf#查看端口使用情況netstat -nltpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1817/master tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 27369/node tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 31848/nginx: master tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 16567/sshd tcp6 0 0 127.0.0.1:8005 :::* LISTEN 8263/java tcp6 0 0 :::5000 :::* LISTEN 2771/java tcp6 0 0 :::8009 :::* LISTEN 8263/java tcp6 0 0 :::3306 :::* LISTEN 28839/mysqld tcp6 0 0 :::80 :::* LISTEN 31848/nginx: master tcp6 0 0 :::8080 :::* LISTEN 8263/java tcp6 0 0 :::9200 :::* LISTEN 25808/java tcp6 0 0 :::9300 :::* LISTEN 25808/java tcp6 0 0 :::9301 :::* LISTEN 2771/java tcp6 0 0 :::22 :::* LISTEN 16567/sshd
我們看到9200端口了說(shuō)明我們安裝成功了，我們可以在終端輸入
#測試訪(fǎng)問(wèn)curl -X GET http://localhost:9200/
或者直接瀏覽器打開(kāi)我們可以看到
{status: 200,name: "Pip the Troll",cluster_name: "elasticsearch",version: {number: "1.7.2",build_hash: "e43676b1385b8125d647f593f7202acbd816e8ec",build_timestamp: "2015-09-14T09:49:53Z",build_snapshot: false,lucene_version: "4.10.4"},tagline: "You Know, for Search"}
說(shuō)明我們的程序是運行正常的。
Kibana的安裝：
#下載tar包wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz#解壓tar zxf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/cd /usr/local/mv kibana-4.1.1-linux-x64 kibana#創(chuàng )建kibana服務(wù)vim /etc/rc.d/init.d/kibana#!/bin/bash### BEGIN INIT INFO# Provides: kibana# Default-Start: 2 3 4 5# Default-Stop: 0 1 6# Short-Description: Runs kibana daemon# Description: Runs the kibana daemon as a non-root user### END INIT INFO# Process nameNAME=kibanaDESC="Kibana4"PROG="/etc/init.d/kibana"# Configure location of Kibana binKIBANA_BIN=/usr/local/kibana/bin# PID InfoPID_FOLDER=/var/run/kibana/PID_FILE=/var/run/kibana/$NAME.pidLOCK_FILE=/var/lock/subsys/$NAMEPATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BINDAEMON=$KIBANA_BIN/$NAME# Configure User to run daemon processDAEMON_USER=root# Configure logging locationKIBANA_LOG=/var/log/kibana.log# Begin ScriptRETVAL=0if [ `id -u` -ne 0 ]; then echo "You need root privileges to run this script" exit 1fi# Function library. /etc/init.d/functions start() { echo -n "Starting $DESC : "pid=`pidofproc -p $PID_FILE kibana` if [ -n "$pid" ] ; then echo "Already running." exit 0 else # Start Daemonif [ ! -d "$PID_FOLDER" ] ; then mkdir $PID_FOLDER fidaemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 & sleep 2 pidofproc node > $PID_FILE RETVAL=$? [[ $? -eq 0 ]] && success || failureecho [ $RETVAL = 0 ] && touch $LOCK_FILE return $RETVAL fi}reload(){ echo "Reload command is not implemented for this service." return $RETVAL}stop() { echo -n "Stopping $DESC : " killproc -p $PID_FILE $DAEMON RETVAL=$?echo [ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE} case "$1" in start) start;; stop) stop ;; status) status -p $PID_FILE $DAEMON RETVAL=$? ;; restart) stop start ;; reload)reload;; *)# Invalid Arguments, print the following message. echo "Usage: $0 {start|stop|status|restart}" >&2exit 2 ;;esac#修改啟動(dòng)權限chmod +x /etc/rc.d/init.d/kibana#啟動(dòng)kibana服務(wù)service kibana startservice kibana status#查看端口netstat -nltp
因為剛剛已經(jīng)執行過(guò)
netstat -nltp
所以顯示的效果我這里就不貼了，如果我們能看到5601端口就說(shuō)明我們安裝成功了。
Option 1：Generate SSL Certificates：
生成SSL證書(shū)是為了服務(wù)端和客戶(hù)端進(jìn)行驗證：
sudo vi /etc/pki/tls/openssl.cnf
Find the [ v3_ca ] section in the file, and add this line under it (substituting in the Logstash Server’s private IP address):
subjectAltName = IP: logstash_server_private_ipcd /etc/pki/tlssudo openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt Option 2: FQDN (DNS)：
cd /etc/pki/tlssudo openssl req -subj '/CN=<^>logstash_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt Logstash安裝：
Logstash Forwarder（客戶(hù)端）：
安裝Logstash Forwarderwget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpmyum localinstall logstash-forwarder-0.4.0-1.x86_64.rpm#查看logstash-forwarder的配置文件位置rpm -qc logstash-forwarder/etc/logstash-forwarder.conf#備份配置文件cp /etc/logstash-forwarder.conf /etc/logstash-forwarder.conf.save#編輯 /etc/logstash-forwarder.conf，需要根據實(shí)際情況進(jìn)行修改vim /etc/logstash-forwarder.conf { "network": { "servers": [ "這里寫(xiě)服務(wù)器的ip:5000" ], "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt", "timeout": 15 }, "files": [ { "paths": [ "/var/log/messages", "/var/log/secure" ], "fields": { "type": "syslog" } } ]} Logstash Server（服務(wù)端）：
#下載rpm包wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm#安裝yum localinstall logstash-1.5.4-1.noarch.rpm #創(chuàng )建一個(gè)01-logstash-initial.conf文件vim /etc/logstash/conf.d/01-logstash-initial.conf input { lumberjack { port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" }}filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } }}output { elasticsearch { host => localhost } stdout { codec => rubydebug }}#啟動(dòng)logstash服務(wù)service logstash startservice logstash status#訪(fǎng)問(wèn)Kibana，Time-field name 選擇 @timestamp 要在下一步操作 Nginx 日志配置之后訪(fǎng)問(wèn) 不然會(huì )沒(méi)有數據不能創(chuàng )建http://localhost:5601/#增加節點(diǎn)和客戶(hù)端配置一樣，注意同步證書(shū)(可以通過(guò)SSH的方式同步)/etc/pki/tls/certs/logstash-forwarder.crt 配置Nginx日志：
#修改客戶(hù)端配置vim /etc/logstash-forwarder.conf{ "network": { "servers": [ "自己服務(wù)器的ip地址:5000" ], "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt", "timeout": 15 }, "files": [ { "paths": [ "/var/log/messages", "/var/log/secure" ], "fields": { "type": "syslog" } }, { "paths": [ "/app/local/nginx/logs/access.log" ], "fields": { "type": "nginx" } } ]}#服務(wù)端增加patternsmkdir /opt/logstash/patternsvim /opt/logstash/patterns/nginxNGUSERNAME [a-zA-Z.@-+_%]+NGUSER %{NGUSERNAME}NGINXACCESS %{IPORHOST:remote_addr} - - [%{HTTPDATE:time_local}] "%{WORD:method} %{URIPATH:path}(?:%{URIPARAM:param})? HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}#修改logstash權限chown -R logstash:logstash /opt/logstash/patterns#修改服務(wù)端配置vim /etc/logstash/conf.d/01-logstash-initial.conf input { lumberjack { port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" }}filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } if [type] == "nginx" { grok { match => { "message" => "%{NGINXACCESS}" } } }}output { elasticsearch { host => localhost } stdout { codec => rubydebug }}
我們看一下完成配置之后的效果：
好了，我是折騰了2天才折騰出來(lái)的，感覺(jué)自己好笨。寫(xiě)篇總結為了下一次能夠快速的搭建起來(lái)。
我們可以ton
使用Elasticsearch + Logstash + Kibana搭建日志集中分析平臺實(shí)踐
作者：
Give Me Freedom,Give Me Fire ,Give Me Reason .Take Me Higher
原文地址：
使用Elasticsearch + Logstash + Kibana搭建日志集中分析平臺實(shí)踐, 感謝原作者分享。
→
經(jīng)典算法-完全平方數 ←
Golang MySQL 數據庫開(kāi)發(fā)發(fā)表評論
