锐客网

  1. 首页 > 睿知 > it技术 > >

ELK集群搭建简略记录

IT知识集群集群搭建ELK集群

一身转战三千里，一剑曾百万师。这篇文章主要讲述ELK集群搭建简略记录相关的知识，希望能为你提供帮助。
ELK集群搭建简略记录
1.规划三台主机搭建集群，对应关系如下

10.0.0.4 node1

10.0.0.8 node2

10.0.0.9 node3

2.添加host（各节点）
cat > > /etc/hosts < < EOF

10.0.0.4 node1

10.0.0.8 node2

10.0.0.9 node3

EOF

3.下载安装elk（各节点）
mkdir /elastic

mkdir -p /data/elasticsearch/{data,logs}

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.2-linux-x86_64.tar.gz

tar -xzf elasticsearch-7.15.2-linux-x86_64.tar.gz -C /elastic

mv /elastic/elasticsearch-7.15.2 /elastic/elasticsearch

cd /elastic/elasticsearch

useradd elastic

chown -R elastic:elastic /elastic

chown -R elastic:elastic /data/elasticsearch

3.修改jvm.options（各节点）#参考(java最大只能分配32G内存，建议分配为内存的一半)
-Xms4g

-Xmx4g

4.修改使用elk自带的jdk（各节点）#修改使用自带的java虚拟机,修改/elastic/elasticsearch/bin/elasticsearch-env配置#在if [ ! -z "$ES_JAVA_HOME" ]; then前添加配置
sed -i/ "$ES_JAVA_HOME/iES_JAVA_HOME=/elastic/elasticsearch/jdk//elastic/elasticsearch/bin/elasticsearch-env

5.生成集群间通信的SSL证书（各节点）#只有配置这个才能使用用户认证#生成证书,记得拷贝到其他节点,如果要拷贝其他节点，记得添加ipsan; 如果不添加，把证书拷贝到其他节点，单独生成证书 #先通过elasticsearch-certutil ca先生成公钥 #再通过elasticsearch-certutil cert 生成证书
cd /elastic/elasticsearch/

/elastic/elasticsearch/bin/elasticsearch-certutil ca

bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --ip "10.0.0.4","10.0.0.8","10.0.0.9"

cd /elastic/elasticsearch/

mkdir /elastic/elasticsearch/config/certs

mv elastic-certificates.p12 /elastic/elasticsearch/config/certs

chown -R elastic:elastic /elastic/elasticsearch/config/certs

scp xxx nodex:xxx

6.生成用于客户端访问的SSL证书（各节点）【ELK集群搭建简略记录】#当然也可以复用tls生成的#这个命令不仅会生成node节点的证书和配置文件，也会生成kibana所需要的配置 #具体参考? ?https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html??#如果麻烦可以不启用
cd /elastic/elasticsearch/

/elastic/elasticsearch/bin/elasticsearch-certutil http

scp xxx node1:xxxx

7.集群配置文件参考
#集群名称

cluster.name: bl-els

# 节点名称，3个节点对应名称["node-1","node-2","node-3"]

node.name: node-1

# ES数据存储路径

path.data: /data/elasticsearch/data

# ES日志存储路径

path.logs: /data/elasticsearch/logs

# 锁定内存

bootstrap.memory_lock: true

# HTTP访问IP，内网IP、外网IP都可以访问

network.host: 0.0.0.0

# HTTP访问端口

http.port: 9200

# 集群访问端口

transport.tcp.port: 9300

# 种子节点的地址列表

discovery.seed_hosts: ["node1", "node2", "node3"]

# 可以成为主节点的名称列表

cluster.initial_master_nodes: ["node1", "node2", "node3"]

#设置elk 账号和密码登录

#节点间安全通信，只有配置这个才能配置用户认证

#elasticsearch-certutil ca cert 之类的命令生成

#参考 https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup.html#generate-certificates

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true

xpack.security.transport.ssl.keystore.path: /elastic/elasticsearch/config/certs/elastic-certificates.p12

xpack.security.transport.ssl.truststore.path: /elastic/elasticsearch/config/certs/elastic-certificates.p12

#接口使用https通信(视情况使用)面向客户端，面向浏览器 可以使用 elasticsearch-certutil http生成；配置完成后需要在kinaba启用https

#参考https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html

xpack.security.http.ssl.enabled: true

xpack.security.transport.ssl.verification_mode: certificate

xpack.security.http.ssl.keystore.path: /elastic/elasticsearch/config/certs/elastic-certificates.p12

xpack.security.http.ssl.truststore.path: /elastic/elasticsearch/config/certs/elastic-certificates.p12

8.调整系统参数配置
echo "vm.max_map_count = 655300" > > /etc/sysctl.conf

echo "* - memlock unlimited" > > /etc/security/limits.conf

echo "* - nofile 655360" > > /etc/security/limits.conf

echo "* - as unlimited" > > /etc/security/limits.conf

echo "* - nproc 2056474" > > /etc/security/limits.conf

echo "* - fsize unlimited" > > /etc/security/limits.conf

echo "net.ipv4.tcp_abort_on_overflow = 1" > > /etc/sysctl.conf

echo "net.core.somaxconn = 2048" > > /etc/sysctl.conf

sysctl -p

9.调整systemd相关参数
cat > > /etc/systemd/system.conf < < EOF

DefaultLimitNOFILE=65536

DefaultLimitNPROC=32000

DefaultLimitMEMLOCK=infinity

EOF

10.设置开机启动-initd方式
cat > /etc/init.d/elasticsearch < < EOF

#!/bin/sh

#chkconfig: 2345 80 05

#description: elasticsearch

els_user=elastic

els_menu=/elastic/elasticsearch

case "$1" in

start)

su $els_user< < !

cd $els_menu

./bin/elasticsearch -d

exit

!

echo "elasticsearch startup"

; ;

stop)

els_pid=`ps aux | grep org.elasticsearch.bootstrap.Elasticsearch | grep -v grep | awk {print $2}`

kill $els_pid

echo "elasticsearch stopped"

; ;

restart)

els_pid=`ps aux | grep org.elasticsearch.bootstrap.Elasticsearch | grep -v grep | awk {print $2}`

kill $els_pid

echo "elasticsearch stopped"

su $els_user< < !

cd $els_menu

./bin/elasticsearch -d

exit

!

echo "elasticsearch startup"

; ;

*)

echo "start|stop|restart"

; ;

esac

exit $?

EOF

chmod +x /etc/init.d/elasticsearch

chkconfig --add elasticsearch

11.设置开机启动-systemd方式
cat > /lib/systemd/system/elasticsearch.service< < EOF

[Unit]

Description=elasticsearch

After=network.target

[Service]

Type=simple

LimitMEMLOCK=infinity

WorkingDirectory=/elastic/elasticsearch/

ExecStart=/elastic/elasticsearch/bin/elasticsearch

ExecReload=/bin/kill -HUP $MAINPID #重载时执行的命令

LimitNOFILE=65536

User=elastic

PrivateTmp=true

[Install]

WantedBy=multi-user.target

EOF

systemctl enable elasticsearch

systemctl restart elasticsearch

systemctl status elasticsearch

12.生成用户和密码//自动随机生成密码，并输出到控制台
/elastic/elasticsearch/bin/elasticsearch-setup-passwords auto

//生成的密码如下：
Changed password for user apm_system

PASSWORD apm_system = gmxadlfgV2KLdgW2uCmi

Changed password for user kibana_system

PASSWORD kibana_system = 0FmCua51aWuiuuS30Pbw

Changed password for user kibana

PASSWORD kibana = 0FmCua51aWuiuuS30Pbw

Changed password for user logstash_system

PASSWORD logstash_system = UaMTgwThhCkpwA9piaf5

Changed password for user beats_system

PASSWORD beats_system = 7SxJtzLeCfE53n8It3Qk

Changed password for user remote_monitoring_user

PASSWORD remote_monitoring_user = dgkVtG5SYSmj8AN7y8PQ

Changed password for user elastic

PASSWORD elastic = Jmrb6idz2EPOlajZpIvD

13.安装kibana
mkdir /elastic/kibana

curl -O https://artifacts.elastic.co/downloads/kibana/kibana-7.15.1-linux-x86_64.tar.gz

tar -xzf kibana-7.15.1-linux-x86_64.tar.gz-C /elastic/

mv/elastic/kibana-7.15.1-linux-x86_64//elastic/kibana

cd /elastic/kibana/

chown -R elastic:elastic /elastic/kibana

14.修改配置文件
修改kinaba 配置文件

/elastic/kibana/config/kibana.yml

server.port: 5601

server.host: "0.0.0.0"

server.name: "es-node1"

elasticsearch.hosts: ["http://10.0.0.4:9200"]

kibana.index: ".kibana"

i18n.locale: "zh-CN"

elasticsearch.username: "elastic"

elasticsearch.password: "kibana_passwd"

#如果启动用https访问

#这个是访问elk需要的，不是指kibana开启ssl

elasticsearch.ssl.certificateAuthorities: $KBN_PATH_CONF/elasticsearch-ca.pem

#如果kibana开启ssl,需要另外设置

15.设置开机启动
cat > /lib/systemd/system/kibana.service< < EOF

[Unit]

Description=kibana

After=network.target

[Service]

Type=simple

WorkingDirectory=/elastic/kibana/

ExecStart=/elastic/kibana/bin/kibana

ExecReload=/bin/kill -HUP $MAINPID #重载时执行的命令

LimitNOFILE=65536

User=elastic

PrivateTmp=true

[Install]

WantedBy=multi-user.target

EOF

systemctl daemon-reload

systemctl startkibana

systemctl statuskibana

systemctl enable kibana




    推荐阅读


    上一篇：Flutter 专题05 图解修改应用名称及图标 #yyds干货盘点#

    下一篇：Flutter 专题02 图解 Mac 环境下安装配置环境 #yyds干货盘点#