Kubernetes K8S之Helm部署ELK日志分析系统;由于Logstash比较消耗资源,因此本次我们使用Fluentd实现日志收集(EFK)。主机配置规划
服务器名称(hostname) | 系统版本 | 配置 | 内网IP | 外网IP(模拟) |
---|---|---|---|---|
k8s-master | CentOS7.7 | 2C/4G/20G | 172.16.1.110 | 10.0.0.110 |
k8s-node01 | CentOS7.7 | 2C/4G/20G | 172.16.1.111 | 10.0.0.111 |
k8s-node02 | CentOS7.7 | 2C/4G/20G | 172.16.1.112 | 10.0.0.112 |
ELK概述 ELK是Elasticsearch、Logstash、Kibana的简称,这三者是核心套件,但并非全部。
Elasticsearch是实时全文搜索和分析引擎,提供搜集、分析、存储数据三大功能;是一套开放REST和JAVA API等接口,提供高效搜索功能,可扩展的分布式系统。它构建于Apache Lucene搜索引擎库之上。
Logstash是一个用来搜集、分析、过滤日志的工具。它支持几乎任何类型的日志,包括系统日志、错误日志和自定义应用程序日志。它可以从许多来源接收日志,这些来源包括 syslog、消息传递(例如 RabbitMQ)和JMX,它能够以多种方式输出数据,包括电子邮件、websockets和Elasticsearch。
Kibana是一个基于Web的图形界面,用于搜索、分析和可视化存储在 Elasticsearch指标中的日志数据。它利用Elasticsearch的REST接口来检索数据,不仅允许用户创建他们自己数据定制仪表板的视图,还允许他们以特殊的方式查询和过滤数据。
由于Logstash比较消耗资源,因此本次我们使用Fluentd实现日志收集(EFK)。
EFK镜像下载 由于镜像都在国外,因此我们在国内下载镜像,然后tag为对应的镜像名称。执行如下脚本【集群所有机器都执行】:
[root@k8s-node02 software]# vim download_efk_image.sh
#!/bin/sh##### 在 master 节点和 worker 节点都要执行 【所有机器执行】# 加载环境变量
. /etc/profile
. /etc/bashrc# 变量设置
elasticsearch_iamge="elasticsearch-oss:6.7.0"
busybox_image="busybox:latest"
bats_image="bats:0.4.0"
fluentd_image="fluentd-elasticsearch:v2.3.2"
kibana_image="kibana-oss:6.7.0"# 集群所有机器执行
# elasticsearch镜像下载
docker pull registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge}
docker tagregistry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge} docker.elastic.co/elasticsearch/${elasticsearch_iamge}
docker rmiregistry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge}
# busybox镜像下载
docker pull registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image}
docker tagregistry.cn-beijing.aliyuncs.com/google_registry/${busybox_image} ${busybox_image}
docker rmiregistry.cn-beijing.aliyuncs.com/google_registry/${busybox_image}
# bats镜像下载
docker pull registry.cn-beijing.aliyuncs.com/google_registry/${bats_image}
docker tagregistry.cn-beijing.aliyuncs.com/google_registry/${bats_image} dduportal/${bats_image}
docker rmiregistry.cn-beijing.aliyuncs.com/google_registry/${bats_image}
# fluentd-elasticsearch镜像下载
docker pull registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image}
docker tagregistry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image} gcr.io/google-containers/${fluentd_image}
docker rmiregistry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image}
# kibana-oss镜像下载
docker pull registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image}
docker tagregistry.cn-beijing.aliyuncs.com/google_registry/${kibana_image} docker.elastic.co/kibana/${kibana_image}
docker rmiregistry.cn-beijing.aliyuncs.com/google_registry/${kibana_image}
Elasticsearch部署 本次部署EFK,创建一个efk名称空间。
chart下载与配置修改
# 当前目录
[root@k8s-master efk]# pwd
/root/k8s_practice/efk
# 创建 efk 名称空间
[root@k8s-master efk]# kubectl create namespace efk
[root@k8s-master efk]#
# ES版本查看,本次我们部署chart 1.30.0版本,ES 6.7.0版本
[root@k8s-master efk]# helm search stable/elasticsearch -l
NAMECHART VERSION APP VERSION DESCRIPTION
stable/elasticsearch1.32.56.8.6DEPRECATED Flexible and powerful open source, distributed...
stable/elasticsearch1.32.46.8.6Flexible and powerful open source, distributed real-time ...
stable/elasticsearch1.32.36.8.6Flexible and powerful open source, distributed real-time ...
………………
[root@k8s-master efk]#
[root@k8s-master efk]# helm fetch stable/elasticsearch --version 1.30.0
[root@k8s-master efk]# tar xf elasticsearch-1.30.0.tgz
# 修改配置文件1
[root@k8s-master efk]# vim elasticsearch/values.yaml
initImage:
repository: "busybox"
tag: "latest"
pullPolicy: "IfNotPresent"# 从Always 改为IfNotPresent
………………
client:
name: client
replicas: 1# 从2改为1,因为是在自己PC机操作的,内存有限
serviceType: ClusterIP
………………
master:
name: master
exposeHttp: false
replicas: 3# 不要修改
heapSize: "512m"
persistence:
enabled: false# 没有多余的PVC,因此从true改为false
accessMode: ReadWriteOnce
………………
data:
name: data
exposeHttp: false
replicas: 1# 从2改为1,因为是在自己PC机操作的,内存有限
heapSize: "1024m"# 从1536m改为1024m,因为是在自己PC机操作的,内存有限
persistence:
enabled: false# 没有多余的PVC,因此从true改为false
accessMode: ReadWriteOnce
[root@k8s-master efk]#
# 修改配置文件2
[root@k8s-master efk]# vim elasticsearch/templates/client-deployment.yaml
apiVersion: apps/v1# 从 apps/v1beta1 改为 apps/v1
kind: Deployment
………………
spec:
replicas: {{ .Values.client.replicas }}
# 添加信息 如下
selector:
matchLabels:
app: {{ template "elasticsearch.name" . }}
component: "{{ .Values.client.name }}"
release: {{ .Release.Name }}
# 添加信息 如上
[root@k8s-master efk]#
# 修改配置文件3
[root@k8s-master efk]# vim elasticsearch/templates/data-statefulset.yaml
apiVersion: apps/v1# 从 apps/v1beta1 改为 apps/v1
kind: StatefulSet
………………
spec:
serviceName: {{ template "elasticsearch.data.fullname" . }}
replicas: {{ .Values.data.replicas }}
# 添加信息 如下
selector:
matchLabels:
app: {{ template "elasticsearch.name" . }}
component: "{{ .Values.data.name }}"
release: {{ .Release.Name }}
role: data
# 添加信息 如上
[root@k8s-master efk]#
# 修改配置文件4
[root@k8s-master efk]# vim elasticsearch/templates/master-statefulset.yaml
apiVersion: apps/v1# 从 apps/v1beta1 改为 apps/v1
kind: StatefulSet
………………
spec:
serviceName: {{ template "elasticsearch.master.fullname" . }}
replicas: {{ .Values.master.replicas }}
# 添加信息 如下
selector:
matchLabels:
app: {{ template "elasticsearch.name" . }}
component: "{{ .Values.master.name }}"
release: {{ .Release.Name }}
role: master
# 添加信息 如上
[root@k8s-master efk]#
Elasticsearch部署 步骤如下:
# 当前目录
[root@k8s-master efk]# pwd
/root/k8s_practice/efk
# 部署ES
[root@k8s-master efk]# helm install --name es01 --namespace=efk elasticsearch/
[root@k8s-master efk]#
# 状态查看
[root@k8s-master ~]# helm list
NAMEREVISION UPDATEDSTATUSCHARTAPP VERSION NAMESPACE
es011Sat Jul 25 12:18:50 2020 DEPLOYED elasticsearch-1.30.06.7.0efk
[root@k8s-master efk]#
# 等待一会儿后【估计几分钟】,查看pod状态信息如下
[root@k8s-master ~]# kubectl get deploy -n efk
NAMEREADYUP-TO-DATEAVAILABLEAGE
es01-elasticsearch-client1/1116m13s
[root@k8s-master ~]#
[root@k8s-master ~]# kubectl get pod -n efk
NAMEREADYSTATUSRESTARTSAGE
es01-elasticsearch-client-646f8f866d-rt2wp1/1Running06m21s
es01-elasticsearch-data-01/1Running06m21s
es01-elasticsearch-master-01/1Running06m21s
es01-elasticsearch-master-11/1Running05m30s
es01-elasticsearch-master-21/1Running05m3s
[root@k8s-master efk]#
[root@k8s-master efk]# kubectl get svc -n efk
NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGE
es01-elasticsearch-clientClusterIP10.100.237.1529200/TCP6m34s
es01-elasticsearch-discoveryClusterIPNone9300/TCP6m42s
[root@k8s-master efk]#
[root@k8s-master efk]# kubectl get sts -n efk
NAMEREADYAGE
es01-elasticsearch-data1/17m4s
es01-elasticsearch-master3/37m4s
[root@k8s-master efk]#
Elasticsearch访问 其中IP来源于ES的svc。
[root@k8s-master ~]# curl 10.100.237.152:9200/
{
"name" : "es01-elasticsearch-client-646f8f866d-rt2wp",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "S4t_UDOuRye9mtK22VWxLw",
"version" : {
"number" : "6.7.0",
"build_flavor" : "oss",
"build_type" : "docker",
"build_hash" : "8453f77",
"build_date" : "2019-03-21T15:32:29.844721Z",
"build_snapshot" : false,
"lucene_version" : "7.7.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
[root@k8s-master ~]#
[root@k8s-master ~]# curl 10.100.237.152:9200/_cluster/health?pretty
{
"cluster_name" : "elasticsearch",
"status" : "green",# 可见状态正常
"timed_out" : false,
"number_of_nodes" : 5,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
至此,elasticsearch部署完毕
elasticsearch-client域名获取 根据es01-elasticsearch-client的svc信息获取到es01-elasticsearch-client的域名;用于后面的fluentd 和kibana。
启动一个pod
[root@k8s-master test]# pwd
/root/k8s_practice/test
[root@k8s-master test]# cat myapp_demo.yaml
apiVersion: v1
kind: Pod
metadata:
name: myapp-demo
namespace: default
labels:
k8s-app: myapp
spec:
containers:
- name: myapp
image: registry.cn-beijing.aliyuncs.com/google_registry/myapp:v1
imagePullPolicy: IfNotPresent
ports:
- name: httpd
containerPort: 80
protocol: TCP
[root@k8s-master test]#
[root@k8s-master test]# kubectl apply -f myapp_demo.yaml
pod/myapp-demo created
[root@k8s-master test]#
[root@k8s-master test]# kubectl get pod -o wide
NAMEREADYSTATUSRESTARTSAGEIPNODENOMINATED NODEREADINESS GATES
myapp-demo1/1Running06s10.244.2.84k8s-node02
进入pod并得到elasticsearch-client域名信息
# 进入一个pod容器
[root@k8s-master test]# kubectl exec -it myapp-demo sh
##### 格式 nslookup svc-cluster-ip
/ # nslookup 10.100.237.152
nslookup: can't resolve '(null)': Name does not resolveName:10.100.237.152
Address 1: 10.100.237.152 es01-elasticsearch-client.efk.svc.cluster.local
/ #
/ #
### 通过ping判断域名是否通畅
/ # ping es01-elasticsearch-client.efk.svc.cluster.local
PING es01-elasticsearch-client.efk.svc.cluster.local (10.100.237.152): 56 data bytes
64 bytes from 10.100.237.152: seq=0 ttl=64 time=0.094 ms
64 bytes from 10.100.237.152: seq=1 ttl=64 time=0.081 ms
64 bytes from 10.100.237.152: seq=2 ttl=64 time=0.243 ms
由上可得,Service中es01-elasticsearch-client的域名为:
es01-elasticsearch-client.efk.svc.cluster.local
Service的域名格式为:
$(service name).$(namespace).svc.cluster.local
,其中 cluster.local 指定的集群的域名Fluentd部署 chart下载与配置修改
[root@k8s-master efk]# pwd
/root/k8s_practice/efk
[root@k8s-master efk]#
# fluentd版本信息查看
[root@k8s-master efk]# helm search stable/fluentd-elasticsearch -l
NAMECHART VERSION APP VERSION DESCRIPTION
stable/fluentd-elasticsearch 2.0.72.3.2DEPRECATED! - A Fluentd Helm chart for Kubernetes with El...
stable/fluentd-elasticsearch 2.0.62.3.2A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
stable/fluentd-elasticsearch 2.0.52.3.2A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
………………
# 获取fluentd-elasticsearch 并解压
[root@k8s-master efk]# helm fetch stable/fluentd-elasticsearch --version 2.0.7
[root@k8s-master efk]# tar xf fluentd-elasticsearch-2.0.7.tgz
# 配置修改
[root@k8s-master efk]# vim fluentd-elasticsearch/values.yaml
### 为什么使用域名而不是IP,因此每次重启ES的svc,对应IP都会改变。而域名是不变的
elasticsearch:
host: 'es01-elasticsearch-client.efk.svc.cluster.local'# 修改处,域名获取参见上文
port: 9200
scheme: 'http'
[root@k8s-master efk]#
fluentd-elasticsearch部署 步骤如下:
################ 部署fluentd-elasticsearch
# 当前目录
[root@k8s-master efk]# pwd
/root/k8s_practice/efk
# 部署fluentd-elasticsearch
[root@k8s-master efk]# helm install --name fluentd-es01 --namespace=efk fluentd-elasticsearch
[root@k8s-master efk]#
# 状态查看
[root@k8s-master efk]#helm list
NAMEREVISION UPDATEDSTATUSCHARTAPP VERSION NAMESPACE
es011Sat Jul 25 12:18:50 2020 DEPLOYED elasticsearch-1.30.06.7.0efk
fluentd-es011Sat Jul 25 12:36:01 2020 DEPLOYED fluentd-elasticsearch-2.0.7 2.3.2efk
[root@k8s-master efk]#
#查看pod状态信息如下
[root@k8s-master efk]# kubectl get ds -n efk
NAMEDESIREDCURRENTREADYUP-TO-DATEAVAILABLENODE SELECTORAGE
fluentd-es01-fluentd-elasticsearch22222113s
[root@k8s-master efk]#
[root@k8s-master efk]# kubectl get pod -n efk -o wide
NAMEREADYSTATUSRESTARTSAGEIPNODENOMINATED NODEREADINESS GATES
es01-elasticsearch-client-646f8f866d-rt2wp1/1Running017m10.244.2.57k8s-node02
es01-elasticsearch-data-01/1Running017m10.244.2.58k8s-node02
es01-elasticsearch-master-01/1Running017m10.244.4.241k8s-node01
es01-elasticsearch-master-11/1Running017m10.244.2.59k8s-node02
es01-elasticsearch-master-21/1Running016m10.244.4.242k8s-node01
fluentd-es01-fluentd-elasticsearch-qnmf91/1Running043s10.244.4.243k8s-node01
fluentd-es01-fluentd-elasticsearch-xmw5f1/1Running043s10.244.2.60k8s-node02
Kibana部署 kibana的主版本和大版本必须和elasticsearch(ES)一致,小版本可以不同;但两者版本最好一致,这样可以避免一些因版本不同导致的意外情况。
由于elasticsearch(ES)使用的是6.7.0,因此kibana我们也是用该版本。
chart下载与配置修改
[root@k8s-master efk]# pwd
/root/k8s_practice/efk
# 所有版本查看
[root@k8s-master efk]# helm search stable/kibana -l
NAMECHART VERSION APP VERSION DESCRIPTION
stable/kibana 3.2.76.7.0Kibana is an open source data visualization plugin for El...
stable/kibana 3.2.66.7.0Kibana is an open source data visualization plugin for El...
………………
# 获取kibana,并解压缩
[root@k8s-master efk]# helm fetch stable/kibana --version 3.2.7
[root@k8s-master efk]# tar xf kibana-3.2.7.tgz
# 配置修改1
[root@k8s-master efk]# vim kibana/values.yaml
### 为什么使用域名而不是IP,因此每次重启ES的svc,对应IP都会改变。而域名是不变的
files:
kibana.yml:
## Default Kibana configuration from kibana-docker.
server.name: kibana
server.host: "0"
elasticsearch.url: http://es01-elasticsearch-client.efk.svc.cluster.local:9200# 修改处,域名获取参见上文
………………
service:
type: NodePort# 修改内容从ClusterIP改为NodePort
externalPort: 443
internalPort: 5601
nodePort: 30601# 添加处,Service端口范围:30000-32767
[root@k8s-master efk]#
# 配置修改2
[root@k8s-master efk]# vim kibana/templates/deployment.yaml
apiVersion: apps/v1# 从 apps/v1beta1 改为 apps/v1
kind: Deployment
metadata:
………………
spec:
replicas: {{ .Values.replicaCount }}
revisionHistoryLimit: {{ .Values.revisionHistoryLimit }}
# 添加信息 如下
selector:
matchLabels:
app: {{ template "kibana.name" . }}
release: "{{ .Release.Name }}"
# 添加信息 如上
kibana部署 步骤如下:
################ 部署kibana-oss
# 当前目录
[root@k8s-master efk]# pwd
/root/k8s_practice/efk
# 部署kibana-oss
[root@k8s-master efk]# helm install --name kibana01 --namespace=efk kibana
[root@k8s-master efk]#
# 状态查看
[root@k8s-master efk]# helm list
NAMEREVISION UPDATEDSTATUSCHARTAPP VERSION NAMESPACE
es011Sat Jul 25 12:18:50 2020 DEPLOYED elasticsearch-1.30.06.7.0efk
fluentd-es011Sat Jul 25 12:36:01 2020 DEPLOYED fluentd-elasticsearch-2.0.7 2.3.2efk
kibana011Sat Jul 25 12:38:18 2020 DEPLOYED kibana-3.2.76.7.0efk
[root@k8s-master efk]#
#查看pod状态信息如下
[root@k8s-master efk]# kubectl get deploy -n efk
NAMEREADYUP-TO-DATEAVAILABLEAGE
es01-elasticsearch-client1/11119m
kibana011/11127s
[root@k8s-master efk]#
[root@k8s-master efk]# kubectl get pod -n efk -o wide
NAMEREADYSTATUSRESTARTSAGEIPNODENOMINATED NODEREADINESS GATES
es01-elasticsearch-client-646f8f866d-rt2wp1/1Running020m10.244.2.57k8s-node02
es01-elasticsearch-data-01/1Running020m10.244.2.58k8s-node02
es01-elasticsearch-master-01/1Running020m10.244.4.241k8s-node01
es01-elasticsearch-master-11/1Running019m10.244.2.59k8s-node02
es01-elasticsearch-master-21/1Running019m10.244.4.242k8s-node01
fluentd-es01-fluentd-elasticsearch-qnmf91/1Running03m10s10.244.4.243k8s-node01
fluentd-es01-fluentd-elasticsearch-xmw5f1/1Running03m10s10.244.2.60k8s-node02
kibana01-bc479f8c7-kr2n21/1Running053s10.244.4.244k8s-node01
[root@k8s-master efk]#
# 查看svc信息
[root@k8s-master efk]# kubectl get svc -n efk -o wide
NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGESELECTOR
es01-elasticsearch-clientClusterIP10.100.237.1529200/TCP20mapp=elasticsearch,component=client,release=es01
es01-elasticsearch-discoveryClusterIPNone9300/TCP20mapp=elasticsearch,component=master,release=es01
kibana01NodePort10.101.200.177443:30601/TCP71sapp=kibana,release=kibana01
浏览器访问
http://172.16.1.110:30601/
页面访问
文章图片
创建索引
文章图片
文章图片
数据查看
文章图片
相关阅读 1、Kubernetes K8S之Helm部署、使用与示例
完毕!
———END——— 如果觉得不错就关注下呗 (-^O^-) ! 【kubernetes|Kubernetes K8S之Helm部署EFK日志分析系统】
文章图片
推荐阅读
- 网络|Kubernetes核心概念总结
- kubernetes|你不是不会处理多行日志,只是不会写正则表达式而已~
- 抖机灵|k8S-调度器
- K8S|k8s——排错指南
- #云原生征文#Kubernetes(k8s)临时存储
- 「云原生的进阶之路」|【Docker 那些事儿】容器数据卷的妙手
- #云原生征文# 在 Google Kubernetes Cluster 上使用 HANA Expression Database Service
- Kubernetes|Kubectl cp&port-forward&label
- Kubernetes|Kubernetes 架构核心组件工作原理解析