搭建ELK日志分析系统（一）-Elasticsearch安装搭建ELK日志分析系统（一）-

前言系统介绍请参考
http://www.jianshu.com/p/09beacb7dbf6
搭建ELK系统有两种方式
1、组件独立安装（更深入了解ELK系统的工作流程）
2、使用docker容器安装（这种方式配置更简单，快捷方便）
本系列文章使用组件独立安装的方式，如果你想使用docker容器安装，请跳过本教程
环境需求本教程使用CentOS 6.9
Elasticsearch5.x版本不能以root用户运行，所以需增加普通用户
这里新增用户：elk

useradd elk

JDK环境安装 Elasticsearch是基于Java的，需要安装最新的JDK 1.8
如果自带低版本的JDK，执行以下命令查看

java -version

请删除再进行安装新版本

yum remove java

下载JDK1.8

wget http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz #wget命令可能下载不了，自行想办法下载

解压JDK1.8

#解压到指定目录 tar -zxvf ./jdk-8u144-linux-x64.tar.gz -C /usr/local/ #最终结果/usr/local/jdk1.8.0_144

配置环境变量

vi /etc/profile

新增

JAVA_HOME=/usr/local/jdk1.8.0_144 JRE_HOME=/usr/local/jdk1.8.0_144/jre PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin CLASSPATH=:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/libexport PATH=$PATH:$JAVA_HOME:$JRE_HOME:$CLASSPATH

安装Elasticsearch5.5.2 下载安装包

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.2.tar.gz

解压安装包

#解压到指定目录 tar -zxvf ./elasticsearch-5.5.2.tar.gz -C /usr/local/ #最终结果/usr/local/elasticsearch-5.5.2 #改变权限 chown -R elk:elk /usr/local/elasticsearch-5.5.2/

直接运行elasticsearch

#切换用户 su elk #进入目录 cd /usr/local/elasticsearch-5.5.2 #运行 ./bin/elasticsearch

此时可能会出现一大堆异常

java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:363) ~[elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:638) ~[elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:245) [elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) [elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:194) [elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) [elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.5.2.jar:5.5.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.5.2.jar:5.5.2] [2017-08-30T10:29:44,497][INFO ][o.e.n.Node] [] initializing ... [2017-08-30T10:29:45,180][INFO ][o.e.e.NodeEnvironment] [2NJyxQe] using [1] data paths, mounts [[/ (/dev/mapper/VolGroup-lv_root)]], net usable_space [14.9gb], net total_space [17.1gb], spins? [possibly], types [ext4] [2017-08-30T10:29:45,183][INFO ][o.e.e.NodeEnvironment] [2NJyxQe] heap size [1.9gb], compressed ordinary object pointers [true] [2017-08-30T10:29:45,197][INFO ][o.e.n.Node] node name [2NJyxQe] derived from node ID [2NJyxQewTSmbEGY0YlXTSg]; set [node.name] to override [2017-08-30T10:29:45,201][INFO ][o.e.n.Node] version[5.5.2], pid[1828], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/2.6.32-696.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_144/25.144-b01] [2017-08-30T10:29:45,205][INFO ][o.e.n.Node] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/local/elasticsearch-5.5.2] [2017-08-30T10:29:48,468][INFO ][o.e.p.PluginsService] [2NJyxQe] loaded module [aggs-matrix-stats] [2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService] [2NJyxQe] loaded module [ingest-common] [2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService] [2NJyxQe] loaded module [lang-expression] [2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService] [2NJyxQe] loaded module [lang-groovy] [2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService] [2NJyxQe] loaded module [lang-mustache] [2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService] [2NJyxQe] loaded module [lang-painless] [2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService] [2NJyxQe] loaded module [parent-join] [2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService] [2NJyxQe] loaded module [percolator] [2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService] [2NJyxQe] loaded module [reindex] [2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService] [2NJyxQe] loaded module [transport-netty3] [2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService] [2NJyxQe] loaded module [transport-netty4] [2017-08-30T10:29:48,470][INFO ][o.e.p.PluginsService] [2NJyxQe] no plugins loaded [2017-08-30T10:29:55,362][INFO ][o.e.d.DiscoveryModule] [2NJyxQe] using discovery type [zen] [2017-08-30T10:29:56,597][INFO ][o.e.n.Node] initialized [2017-08-30T10:29:56,597][INFO ][o.e.n.Node] [2NJyxQe] starting ... [2017-08-30T10:29:57,256][INFO ][o.e.t.TransportService] [2NJyxQe] publish_address {192.168.1.138:9300}, bound_addresses {192.168.1.138:9300} [2017-08-30T10:29:57,296][INFO ][o.e.b.BootstrapChecks] [2NJyxQe] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks [2017-08-30T10:29:57,412][ERROR][o.e.b.Bootstrap] [2NJyxQe] node validation exception [4] bootstrap checks failed [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] [2]: max number of threads [1024] for user [elk] is too low, increase to at least [2048] [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk [2017-08-30T10:29:57,435][INFO ][o.e.n.Node] [2NJyxQe] stopping ... [2017-08-30T10:29:57,534][INFO ][o.e.n.Node] [2NJyxQe] stopped [2017-08-30T10:29:57,534][INFO ][o.e.n.Node] [2NJyxQe] closing ... [2017-08-30T10:29:57,552][INFO ][o.e.n.Node] [2NJyxQe] closed

异常问题一

[4] bootstrap checks failed [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] [2]: max number of threads [1024] for user [elk] is too low, increase to at least [2048] [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

这里有4个问题，需要逐一解决
问题[1]
原因：无法创建本地文件问题,用户最大可创建文件数太小
解决方案：
切换到root用户，编辑limits.conf配置文件，添加类似如下内容：

vi /etc/security/limits.conf#添加如下内容: *soft nofile 65536 * hard nofile 131072 * soft nproc 2048 * hard nproc 4096

问题[2]
解决：切换到root用户，修改90-nproc.conf配置文件。

vi /etc/security/limits.d/90-nproc.conf #修改如下内容： * soft nproc 1024 #修改为 * soft nproc 2048

问题[3]
解决：切换到root用户修改配置sysctl.conf

vi /etc/sysctl.conf #添加下面配置： vm.max_map_count=655360 #并执行命令： sysctl -p

问题[4]

[4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk#和以下异常是统一原因 java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed ....

问题原因：因为Centos6不支持SecComp，而ES5.5.2默认bootstrap.system_call_filter为true进行检测
所以导致检测失败，失败后直接导致ES不能启动。
详见：https://github.com/elastic/elasticsearch/issues/22899
解决方法：
修改配置

vi ./config/elasticsearch.yml

bootstrap.system_call_filter为false，注意要在Memory下面:

bootstrap.memory_lock: false bootstrap.system_call_filter: false

切换root用户，重新启动elasticsearch

su root ./bin/elasticsearch

如无意外，启动正常
本机命令访问

curl http://127.0.0.1:9200

如出现以下内容，则启动正常

{ "name" : "2NJyxQe", "cluster_name" : "elasticsearch", "cluster_uuid" : "HR3s74ntRXKZi0qal3BE0A", "version" : { "number" : "5.5.2", "build_hash" : "b2f0c09", "build_date" : "2017-08-14T12:33:14.154Z", "build_snapshot" : false, "lucene_version" : "6.6.0" }, "tagline" : "You Know, for Search" }

如果在虚拟机里安装elasticsearch，想使用外部浏览器访问
需修改配置