kubernetes|Kubernetes K8S之Helm部署EFK日志分析系统

Kubernetes K8S之Helm部署ELK日志分析系统;由于Logstash比较消耗资源,因此本次我们使用Fluentd实现日志收集(EFK)。
主机配置规划
服务器名称(hostname) 系统版本 配置 内网IP 外网IP(模拟)
k8s-master CentOS7.7 2C/4G/20G 172.16.1.110 10.0.0.110
k8s-node01 CentOS7.7 2C/4G/20G 172.16.1.111 10.0.0.111
k8s-node02 CentOS7.7 2C/4G/20G 172.16.1.112 10.0.0.112
备注:由于EFK部署比较消耗内存;所以每台机器的内存最好大于等于4G。
ELK概述 ELK是Elasticsearch、Logstash、Kibana的简称,这三者是核心套件,但并非全部。
Elasticsearch是实时全文搜索和分析引擎,提供搜集、分析、存储数据三大功能;是一套开放REST和JAVA API等接口,提供高效搜索功能,可扩展的分布式系统。它构建于Apache Lucene搜索引擎库之上。
Logstash是一个用来搜集、分析、过滤日志的工具。它支持几乎任何类型的日志,包括系统日志、错误日志和自定义应用程序日志。它可以从许多来源接收日志,这些来源包括 syslog、消息传递(例如 RabbitMQ)和JMX,它能够以多种方式输出数据,包括电子邮件、websockets和Elasticsearch。
Kibana是一个基于Web的图形界面,用于搜索、分析和可视化存储在 Elasticsearch指标中的日志数据。它利用Elasticsearch的REST接口来检索数据,不仅允许用户创建他们自己数据定制仪表板的视图,还允许他们以特殊的方式查询和过滤数据。
由于Logstash比较消耗资源,因此本次我们使用Fluentd实现日志收集(EFK)。
EFK镜像下载 由于镜像都在国外,因此我们在国内下载镜像,然后tag为对应的镜像名称。执行如下脚本【集群所有机器都执行】:
[root@k8s-node02 software]# vim download_efk_image.sh #!/bin/sh##### 在 master 节点和 worker 节点都要执行 【所有机器执行】# 加载环境变量 . /etc/profile . /etc/bashrc# 变量设置 elasticsearch_iamge="elasticsearch-oss:6.7.0" busybox_image="busybox:latest" bats_image="bats:0.4.0" fluentd_image="fluentd-elasticsearch:v2.3.2" kibana_image="kibana-oss:6.7.0"# 集群所有机器执行 # elasticsearch镜像下载 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge} docker tagregistry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge} docker.elastic.co/elasticsearch/${elasticsearch_iamge} docker rmiregistry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge} # busybox镜像下载 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image} docker tagregistry.cn-beijing.aliyuncs.com/google_registry/${busybox_image} ${busybox_image} docker rmiregistry.cn-beijing.aliyuncs.com/google_registry/${busybox_image} # bats镜像下载 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${bats_image} docker tagregistry.cn-beijing.aliyuncs.com/google_registry/${bats_image} dduportal/${bats_image} docker rmiregistry.cn-beijing.aliyuncs.com/google_registry/${bats_image} # fluentd-elasticsearch镜像下载 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image} docker tagregistry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image} gcr.io/google-containers/${fluentd_image} docker rmiregistry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image} # kibana-oss镜像下载 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image} docker tagregistry.cn-beijing.aliyuncs.com/google_registry/${kibana_image} docker.elastic.co/kibana/${kibana_image} docker rmiregistry.cn-beijing.aliyuncs.com/google_registry/${kibana_image}

Elasticsearch部署 本次部署EFK,创建一个efk名称空间。
chart下载与配置修改
# 当前目录 [root@k8s-master efk]# pwd /root/k8s_practice/efk # 创建 efk 名称空间 [root@k8s-master efk]# kubectl create namespace efk [root@k8s-master efk]# # ES版本查看,本次我们部署chart 1.30.0版本,ES 6.7.0版本 [root@k8s-master efk]# helm search stable/elasticsearch -l NAMECHART VERSION APP VERSION DESCRIPTION stable/elasticsearch1.32.56.8.6DEPRECATED Flexible and powerful open source, distributed... stable/elasticsearch1.32.46.8.6Flexible and powerful open source, distributed real-time ... stable/elasticsearch1.32.36.8.6Flexible and powerful open source, distributed real-time ... ……………… [root@k8s-master efk]# [root@k8s-master efk]# helm fetch stable/elasticsearch --version 1.30.0 [root@k8s-master efk]# tar xf elasticsearch-1.30.0.tgz # 修改配置文件1 [root@k8s-master efk]# vim elasticsearch/values.yaml initImage: repository: "busybox" tag: "latest" pullPolicy: "IfNotPresent"# 从Always 改为IfNotPresent ……………… client: name: client replicas: 1# 从2改为1,因为是在自己PC机操作的,内存有限 serviceType: ClusterIP ……………… master: name: master exposeHttp: false replicas: 3# 不要修改 heapSize: "512m" persistence: enabled: false# 没有多余的PVC,因此从true改为false accessMode: ReadWriteOnce ……………… data: name: data exposeHttp: false replicas: 1# 从2改为1,因为是在自己PC机操作的,内存有限 heapSize: "1024m"# 从1536m改为1024m,因为是在自己PC机操作的,内存有限 persistence: enabled: false# 没有多余的PVC,因此从true改为false accessMode: ReadWriteOnce [root@k8s-master efk]# # 修改配置文件2 [root@k8s-master efk]# vim elasticsearch/templates/client-deployment.yaml apiVersion: apps/v1# 从 apps/v1beta1 改为 apps/v1 kind: Deployment ……………… spec: replicas: {{ .Values.client.replicas }} # 添加信息 如下 selector: matchLabels: app: {{ template "elasticsearch.name" . }} component: "{{ .Values.client.name }}" release: {{ .Release.Name }} # 添加信息 如上 [root@k8s-master efk]# # 修改配置文件3 [root@k8s-master efk]# vim elasticsearch/templates/data-statefulset.yaml apiVersion: apps/v1# 从 apps/v1beta1 改为 apps/v1 kind: StatefulSet ……………… spec: serviceName: {{ template "elasticsearch.data.fullname" . }} replicas: {{ .Values.data.replicas }} # 添加信息 如下 selector: matchLabels: app: {{ template "elasticsearch.name" . }} component: "{{ .Values.data.name }}" release: {{ .Release.Name }} role: data # 添加信息 如上 [root@k8s-master efk]# # 修改配置文件4 [root@k8s-master efk]# vim elasticsearch/templates/master-statefulset.yaml apiVersion: apps/v1# 从 apps/v1beta1 改为 apps/v1 kind: StatefulSet ……………… spec: serviceName: {{ template "elasticsearch.master.fullname" . }} replicas: {{ .Values.master.replicas }} # 添加信息 如下 selector: matchLabels: app: {{ template "elasticsearch.name" . }} component: "{{ .Values.master.name }}" release: {{ .Release.Name }} role: master # 添加信息 如上 [root@k8s-master efk]#

Elasticsearch部署 步骤如下:
# 当前目录 [root@k8s-master efk]# pwd /root/k8s_practice/efk # 部署ES [root@k8s-master efk]# helm install --name es01 --namespace=efk elasticsearch/ [root@k8s-master efk]# # 状态查看 [root@k8s-master ~]# helm list NAMEREVISION UPDATEDSTATUSCHARTAPP VERSION NAMESPACE es011Sat Jul 25 12:18:50 2020 DEPLOYED elasticsearch-1.30.06.7.0efk [root@k8s-master efk]# # 等待一会儿后【估计几分钟】,查看pod状态信息如下 [root@k8s-master ~]# kubectl get deploy -n efk NAMEREADYUP-TO-DATEAVAILABLEAGE es01-elasticsearch-client1/1116m13s [root@k8s-master ~]# [root@k8s-master ~]# kubectl get pod -n efk NAMEREADYSTATUSRESTARTSAGE es01-elasticsearch-client-646f8f866d-rt2wp1/1Running06m21s es01-elasticsearch-data-01/1Running06m21s es01-elasticsearch-master-01/1Running06m21s es01-elasticsearch-master-11/1Running05m30s es01-elasticsearch-master-21/1Running05m3s [root@k8s-master efk]# [root@k8s-master efk]# kubectl get svc -n efk NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGE es01-elasticsearch-clientClusterIP10.100.237.1529200/TCP6m34s es01-elasticsearch-discoveryClusterIPNone9300/TCP6m42s [root@k8s-master efk]# [root@k8s-master efk]# kubectl get sts -n efk NAMEREADYAGE es01-elasticsearch-data1/17m4s es01-elasticsearch-master3/37m4s [root@k8s-master efk]#

Elasticsearch访问 其中IP来源于ES的svc。
[root@k8s-master ~]# curl 10.100.237.152:9200/ { "name" : "es01-elasticsearch-client-646f8f866d-rt2wp", "cluster_name" : "elasticsearch", "cluster_uuid" : "S4t_UDOuRye9mtK22VWxLw", "version" : { "number" : "6.7.0", "build_flavor" : "oss", "build_type" : "docker", "build_hash" : "8453f77", "build_date" : "2019-03-21T15:32:29.844721Z", "build_snapshot" : false, "lucene_version" : "7.7.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" } [root@k8s-master ~]# [root@k8s-master ~]# curl 10.100.237.152:9200/_cluster/health?pretty { "cluster_name" : "elasticsearch", "status" : "green",# 可见状态正常 "timed_out" : false, "number_of_nodes" : 5, "number_of_data_nodes" : 1, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }

至此,elasticsearch部署完毕
elasticsearch-client域名获取 根据es01-elasticsearch-client的svc信息获取到es01-elasticsearch-client的域名;用于后面的fluentd 和kibana。
启动一个pod
[root@k8s-master test]# pwd /root/k8s_practice/test [root@k8s-master test]# cat myapp_demo.yaml apiVersion: v1 kind: Pod metadata: name: myapp-demo namespace: default labels: k8s-app: myapp spec: containers: - name: myapp image: registry.cn-beijing.aliyuncs.com/google_registry/myapp:v1 imagePullPolicy: IfNotPresent ports: - name: httpd containerPort: 80 protocol: TCP [root@k8s-master test]# [root@k8s-master test]# kubectl apply -f myapp_demo.yaml pod/myapp-demo created [root@k8s-master test]# [root@k8s-master test]# kubectl get pod -o wide NAMEREADYSTATUSRESTARTSAGEIPNODENOMINATED NODEREADINESS GATES myapp-demo1/1Running06s10.244.2.84k8s-node02

进入pod并得到elasticsearch-client域名信息
# 进入一个pod容器 [root@k8s-master test]# kubectl exec -it myapp-demo sh ##### 格式 nslookup svc-cluster-ip / # nslookup 10.100.237.152 nslookup: can't resolve '(null)': Name does not resolveName:10.100.237.152 Address 1: 10.100.237.152 es01-elasticsearch-client.efk.svc.cluster.local / # / # ### 通过ping判断域名是否通畅 / # ping es01-elasticsearch-client.efk.svc.cluster.local PING es01-elasticsearch-client.efk.svc.cluster.local (10.100.237.152): 56 data bytes 64 bytes from 10.100.237.152: seq=0 ttl=64 time=0.094 ms 64 bytes from 10.100.237.152: seq=1 ttl=64 time=0.081 ms 64 bytes from 10.100.237.152: seq=2 ttl=64 time=0.243 ms

由上可得,Service中es01-elasticsearch-client的域名为:es01-elasticsearch-client.efk.svc.cluster.local
Service的域名格式为:$(service name).$(namespace).svc.cluster.local,其中 cluster.local 指定的集群的域名
Fluentd部署 chart下载与配置修改
[root@k8s-master efk]# pwd /root/k8s_practice/efk [root@k8s-master efk]# # fluentd版本信息查看 [root@k8s-master efk]# helm search stable/fluentd-elasticsearch -l NAMECHART VERSION APP VERSION DESCRIPTION stable/fluentd-elasticsearch 2.0.72.3.2DEPRECATED! - A Fluentd Helm chart for Kubernetes with El... stable/fluentd-elasticsearch 2.0.62.3.2A Fluentd Helm chart for Kubernetes with Elasticsearch ou... stable/fluentd-elasticsearch 2.0.52.3.2A Fluentd Helm chart for Kubernetes with Elasticsearch ou... ……………… # 获取fluentd-elasticsearch 并解压 [root@k8s-master efk]# helm fetch stable/fluentd-elasticsearch --version 2.0.7 [root@k8s-master efk]# tar xf fluentd-elasticsearch-2.0.7.tgz # 配置修改 [root@k8s-master efk]# vim fluentd-elasticsearch/values.yaml ### 为什么使用域名而不是IP,因此每次重启ES的svc,对应IP都会改变。而域名是不变的 elasticsearch: host: 'es01-elasticsearch-client.efk.svc.cluster.local'# 修改处,域名获取参见上文 port: 9200 scheme: 'http' [root@k8s-master efk]#

fluentd-elasticsearch部署 步骤如下:
################ 部署fluentd-elasticsearch # 当前目录 [root@k8s-master efk]# pwd /root/k8s_practice/efk # 部署fluentd-elasticsearch [root@k8s-master efk]# helm install --name fluentd-es01 --namespace=efk fluentd-elasticsearch [root@k8s-master efk]# # 状态查看 [root@k8s-master efk]#helm list NAMEREVISION UPDATEDSTATUSCHARTAPP VERSION NAMESPACE es011Sat Jul 25 12:18:50 2020 DEPLOYED elasticsearch-1.30.06.7.0efk fluentd-es011Sat Jul 25 12:36:01 2020 DEPLOYED fluentd-elasticsearch-2.0.7 2.3.2efk [root@k8s-master efk]# #查看pod状态信息如下 [root@k8s-master efk]# kubectl get ds -n efk NAMEDESIREDCURRENTREADYUP-TO-DATEAVAILABLENODE SELECTORAGE fluentd-es01-fluentd-elasticsearch22222113s [root@k8s-master efk]# [root@k8s-master efk]# kubectl get pod -n efk -o wide NAMEREADYSTATUSRESTARTSAGEIPNODENOMINATED NODEREADINESS GATES es01-elasticsearch-client-646f8f866d-rt2wp1/1Running017m10.244.2.57k8s-node02 es01-elasticsearch-data-01/1Running017m10.244.2.58k8s-node02 es01-elasticsearch-master-01/1Running017m10.244.4.241k8s-node01 es01-elasticsearch-master-11/1Running017m10.244.2.59k8s-node02 es01-elasticsearch-master-21/1Running016m10.244.4.242k8s-node01 fluentd-es01-fluentd-elasticsearch-qnmf91/1Running043s10.244.4.243k8s-node01 fluentd-es01-fluentd-elasticsearch-xmw5f1/1Running043s10.244.2.60k8s-node02

Kibana部署 kibana的主版本和大版本必须和elasticsearch(ES)一致,小版本可以不同;但两者版本最好一致,这样可以避免一些因版本不同导致的意外情况。
由于elasticsearch(ES)使用的是6.7.0,因此kibana我们也是用该版本。
chart下载与配置修改
[root@k8s-master efk]# pwd /root/k8s_practice/efk # 所有版本查看 [root@k8s-master efk]# helm search stable/kibana -l NAMECHART VERSION APP VERSION DESCRIPTION stable/kibana 3.2.76.7.0Kibana is an open source data visualization plugin for El... stable/kibana 3.2.66.7.0Kibana is an open source data visualization plugin for El... ……………… # 获取kibana,并解压缩 [root@k8s-master efk]# helm fetch stable/kibana --version 3.2.7 [root@k8s-master efk]# tar xf kibana-3.2.7.tgz # 配置修改1 [root@k8s-master efk]# vim kibana/values.yaml ### 为什么使用域名而不是IP,因此每次重启ES的svc,对应IP都会改变。而域名是不变的 files: kibana.yml: ## Default Kibana configuration from kibana-docker. server.name: kibana server.host: "0" elasticsearch.url: http://es01-elasticsearch-client.efk.svc.cluster.local:9200# 修改处,域名获取参见上文 ……………… service: type: NodePort# 修改内容从ClusterIP改为NodePort externalPort: 443 internalPort: 5601 nodePort: 30601# 添加处,Service端口范围:30000-32767 [root@k8s-master efk]# # 配置修改2 [root@k8s-master efk]# vim kibana/templates/deployment.yaml apiVersion: apps/v1# 从 apps/v1beta1 改为 apps/v1 kind: Deployment metadata: ……………… spec: replicas: {{ .Values.replicaCount }} revisionHistoryLimit: {{ .Values.revisionHistoryLimit }} # 添加信息 如下 selector: matchLabels: app: {{ template "kibana.name" . }} release: "{{ .Release.Name }}" # 添加信息 如上

kibana部署 步骤如下:
################ 部署kibana-oss # 当前目录 [root@k8s-master efk]# pwd /root/k8s_practice/efk # 部署kibana-oss [root@k8s-master efk]# helm install --name kibana01 --namespace=efk kibana [root@k8s-master efk]# # 状态查看 [root@k8s-master efk]# helm list NAMEREVISION UPDATEDSTATUSCHARTAPP VERSION NAMESPACE es011Sat Jul 25 12:18:50 2020 DEPLOYED elasticsearch-1.30.06.7.0efk fluentd-es011Sat Jul 25 12:36:01 2020 DEPLOYED fluentd-elasticsearch-2.0.7 2.3.2efk kibana011Sat Jul 25 12:38:18 2020 DEPLOYED kibana-3.2.76.7.0efk [root@k8s-master efk]# #查看pod状态信息如下 [root@k8s-master efk]# kubectl get deploy -n efk NAMEREADYUP-TO-DATEAVAILABLEAGE es01-elasticsearch-client1/11119m kibana011/11127s [root@k8s-master efk]# [root@k8s-master efk]# kubectl get pod -n efk -o wide NAMEREADYSTATUSRESTARTSAGEIPNODENOMINATED NODEREADINESS GATES es01-elasticsearch-client-646f8f866d-rt2wp1/1Running020m10.244.2.57k8s-node02 es01-elasticsearch-data-01/1Running020m10.244.2.58k8s-node02 es01-elasticsearch-master-01/1Running020m10.244.4.241k8s-node01 es01-elasticsearch-master-11/1Running019m10.244.2.59k8s-node02 es01-elasticsearch-master-21/1Running019m10.244.4.242k8s-node01 fluentd-es01-fluentd-elasticsearch-qnmf91/1Running03m10s10.244.4.243k8s-node01 fluentd-es01-fluentd-elasticsearch-xmw5f1/1Running03m10s10.244.2.60k8s-node02 kibana01-bc479f8c7-kr2n21/1Running053s10.244.4.244k8s-node01 [root@k8s-master efk]# # 查看svc信息 [root@k8s-master efk]# kubectl get svc -n efk -o wide NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGESELECTOR es01-elasticsearch-clientClusterIP10.100.237.1529200/TCP20mapp=elasticsearch,component=client,release=es01 es01-elasticsearch-discoveryClusterIPNone9300/TCP20mapp=elasticsearch,component=master,release=es01 kibana01NodePort10.101.200.177443:30601/TCP71sapp=kibana,release=kibana01

浏览器访问
http://172.16.1.110:30601/

页面访问
kubernetes|Kubernetes K8S之Helm部署EFK日志分析系统
文章图片

创建索引
kubernetes|Kubernetes K8S之Helm部署EFK日志分析系统
文章图片

kubernetes|Kubernetes K8S之Helm部署EFK日志分析系统
文章图片

数据查看
kubernetes|Kubernetes K8S之Helm部署EFK日志分析系统
文章图片

相关阅读 1、Kubernetes K8S之Helm部署、使用与示例
完毕!
———END——— 如果觉得不错就关注下呗 (-^O^-) ! 【kubernetes|Kubernetes K8S之Helm部署EFK日志分析系统】kubernetes|Kubernetes K8S之Helm部署EFK日志分析系统
文章图片

    推荐阅读