ceph基于VMware Workstation虚拟机Ceph集群安装配置笔记#私藏项目实操分享# _配置笔记

高斋晓开卷，独共圣人语。这篇文章主要讲述ceph基于VMware Workstation虚拟机Ceph集群安装配置笔记#私藏项目实操分享#相关的知识，希望能为你提供帮助。
1.写在前面本文主要内容为搭建三节点Ceph集群的过程。
这两天捣鼓了一下Ceph，准备做个底层基于ceph的nas存储，首先网上看了一些教程，还有对照了一下书籍，依然在安装配置时ceph还是遇到很多问题，所以写此贴记录一下自己遇到的问题和安装过程，同时也希望能够帮助一些小伙伴搭建Ceph集群。

2.安装环境linux版本，精简版

[root@node1 ~]# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)

Ceph版本 Giant

[root@node1 ~]# ceph -v
ceph version 0.87.2 (87a7cec9ab11c677de2ab23a7668a77d2f5b955e)

VMware workstation版本

网络（桥接物理主机）
node1 192.168.68.11
node2 192.168.68.12
node3 192.168.68.13
网关 192.168.68.1
3.安装步骤3.1 VMware Workstation 安装虚机步骤

3.2 centos7安装过程

node2、node3也是如此，局部将1改为2或3即可（或者使用Vmware Workstation克隆功能，这里不再演示）
给三个节点虚机分别增加3块硬盘

3.3 安装Ceph前虚机配置（三个节点都需要）
配置hosts访问

vi /etc/hosts

配置node1免密访问node2、3

ssh-keygen

ssh-copy-id node2

ssh-copy-id node3

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

关闭selinux

setenforce 0
sed -i /SELINUX/s/enforcing/disabled/ /etc/selinux/config

安装并配置ntp时钟服务

yum install -y ntp ntpdate -y
ntpdate pool.net.org
systemctl restart ntpdate.service
systemctl restart ntp.service
systemctl enable ntp.service
systemctl enable ntpdate.service

添加Ceph Giant 版本包更新yum

rpm -Uhv http://download.ceph.com/rpm-giant/el7/noarch/ceph-release-1-0.el7.noarch.rpm

修改ceph.repo配置文件（提示:%d可快速清除此页内容）

vi /etc/yum.repos.d/ceph.repo

[Ceph]
name=Ceph packages for $basearch
baseurl=http://download.ceph.com/rpm-giant/el7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=1

[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://download.ceph.com/rpm-giant/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=1

[ceph-source]
name=Ceph source packages
baseurl=http://download.ceph.com/rpm-giant/el7/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=1

更新yum

yum -y update

3.4 安装和配置Ceph
强烈建议此步骤前将三台虚机快照备份
3.4.1 在node?1上创建Ceph集群安装ceph-deploy

yum install ceph-deploy -y

创建一个ceph集群

mkdir /etc/ceph
cd /etc/ceph
ceph-deploy new node1

在node1节点给个节点安装ceph

ceph-deploy install node1 node2 node3

查看安装ceph版本

ceph -v

在node1上创建第一个ceph monitor

ceph-deploy mon create-initial

检查集群状态

ceph -s

在node1上创建OSD(在ceph目录下执行)列出可用磁盘

ceph-deploy disk list node1

删除选择磁盘的分区和内容

ceph-deploy disk zap node1:sdb node1:sdc node1:sdd

创建osd

ceph-deploy osd create node1:sdb node1:sdc node1:sdd

检查集群状态

ceph -s

此时单节点配置完毕
3.4.2 拓展Ceph集群在node1上将共有网络地址添加到文件/etc/ceph/ceph.conf

public network = 192.168.68.0/24

分别在node2、3上创建monitor

ceph-deploy mon create node2
ceph-deploy mon create node3

查看集群状态

ceph -s

此时另外两个节点已成功加入
处理node2、3节点的磁盘并创建OSD

ceph-deploy disk zap node2:sdb node2:sdc node2:sdd
ceph-deploy disk zap node3:sdb node3:sdc node3:sdd

ceph-deploy osd create node2:sdb node2:sdc node2:sdd
ceph-deploy osd create node3:sdb node3:sdc node3:sdd

调整rbd存储池pg_num和pgp_num的值，使我们的集群达到HEALTH_OK状态

ceph osd pool set rbd pg_num 256
ceph osd pool set rbd pgp_num 256

检查集群状态，如果为HEALTH_OK，集群已经完整搭建并且正常了
4.遇到的一些错误原因：修改了ceph用户里的ceph.conf文件内容，但是没有把这个文件里的最新消息发送给其他节点，所有要推送消息

ceph-deploy --overwrite-conf config push node2

或

ceph-deploy --overwrite-conf mon create node2

5.一些常用命令检查Ceph安装状态

ceph -s

ceph status

[root@node1 ceph]# ceph -s
cluster 5e563d2f-94e6-4d9b-8aaf-6b2c76f856e4
health HEALTH_OK
monmap e3: 3 mons at node1=192.168.68.11:6789/0,node2=192.168.68.12:6789/0,node3=192.168.68.13:6789/0, election epoch 4, quorum 0,1,2 node1,node2,node3
osdmap e53: 9 osds: 9 up, 9 in
pgmap v122: 256 pgs, 1 pools, 0 bytes data, 0 objects
318 MB used, 134 GB / 134 GB avail
256 active+clean

[root@node1 ceph]# ceph -s
cluster 5e563d2f-94e6-4d9b-8aaf-6b2c76f856e4
health HEALTH_OK
monmap e3: 3 mons at node1=192.168.68.11:6789/0,node2=192.168.68.12:6789/0,node3=192.168.68.13:6789/0, election epoch 4, quorum 0,1,2 node1,node2,node3
osdmap e53: 9 osds: 9 up, 9 in
pgmap v122: 256 pgs, 1 pools, 0 bytes data, 0 objects
318 MB used, 134 GB / 134 GB avail
256 active+clean

查看Ceph版本

ceph -v

[root@node1 ceph]# ceph -v
ceph version 0.87.2 (87a7cec9ab11c677de2ab23a7668a77d2f5b955e)

观察集群健康状况

ceph -

[root@node1 ceph]# ceph -w
cluster 5e563d2f-94e6-4d9b-8aaf-6b2c76f856e4
health HEALTH_OK
monmap e3: 3 mons at node1=192.168.68.11:6789/0,node2=192.168.68.12:6789/0,node3=192.168.68.13:6789/0, election epoch 4, quorum 0,1,2 node1,node2,node3
osdmap e53: 9 osds: 9 up, 9 in
pgmap v122: 256 pgs, 1 pools, 0 bytes data, 0 objects
318 MB used, 134 GB / 134 GB avail
256 active+clean

2022-04-25 10:27:09.830678 mon.0 [INF] pgmap v122: 256 pgs: 256 active+clean; 0 bytes data, 318 MB used, 134 GB / 134 GB avail

检查Ceph monitor仲裁状态

ceph quorum_status --format json-pretty

[root@node1 ceph]# ceph quorum_status --format json-pretty

"election_epoch": 4,
"quorum": [
0,
1,
2],
"quorum_names": [
"node1",
"node2",
"node3"],
"quorum_leader_name": "node1",
"monmap":"epoch": 3,
"fsid": "5e563d2f-94e6-4d9b-8aaf-6b2c76f856e4",
"modified": "2022-04-25 10:06:48.209985",
"created": "0.000000",
"mons": [
"rank": 0,
"name": "node1",
"addr": "192.168.68.11:6789\\/0",
"rank": 1,
"name": "node2",
"addr": "192.168.68.12:6789\\/0",
"rank": 2,
"name": "node3",
"addr": "192.168.68.13:6789\\/0"]

列表PG

ceph pg dump

列表Ceph存储池

ceph osd lspools

[root@node1 ceph]# ceph osd lspools
0 rbd,

列表集群的认证密钥

ceph auth list

[root@node1 ceph]# ceph auth list
installed auth entries:

osd.0
key: AQA7GGViMOKvBhAApRSMC8DDLnlOQXmAD7UUDQ==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.1
key: AQBEGGVioIu6IxAAtFI6GkzHH86f5DbZcFLP+Q==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.2
key: AQBMGGViMHbdKxAAlahPljoMpYC5gRoJBPwmcg==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.3
key: AQB9BWZiwNzrOhAADsHBX/QZgBgZ/5SbJ9wFlg==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.4
key: AQCHBWZiECT2IBAApALFn7F7IDMW/ctkL8BAsA==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.5
key: AQCPBWZisNn3NxAAiDcZGUPWY+e3lflW+7c6AQ==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.6
key: AQCdBWZiQLATHxAAc7z2NE3FmFUx28dIXeHN2g==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.7
key: AQCnBWZiIPh+CRAA/hDBfG/iwChiNcVvB4lw2Q==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.8
key: AQCvBWZiQE9vARAAroqxul/dQRsDnN7Cz9pkAA==
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQCRF2ViOAyLMBAAip1+6gqV2wJmxYUlrBzdFQ==
caps: [mds] allow
caps: [mon] allow *
caps: [osd] allow *
client.bootstrap-mds
key: AQCSF2ViiMcjChAAncKTNo7o5sGaKFvoJHEFmA==
caps: [mon] allow profile bootstrap-mds
client.bootstrap-osd
key: AQCSF2ViUB/CABAAPXRQtcShw39kI6xYr51Cdw==
caps: [mon] allow profile bootstrap-osd

检查集群使用状态

ceph df

[root@node1 ceph]# ceph df
GLOBAL:
SIZEAVAILRAW USED%RAW USED
134G134G318M0.23
POOLS:
NAMEIDUSED%USEDMAX AVAILOBJECTS
rbd00045938M0

检查OSD的Crush map

ceph osd tree

[root@node1 ceph]# ceph osd tree
# idweighttype name up/down reweight
-10.08995 root default
-20.02998host node1
0 0.009995osd.0 up1
1 0.009995osd.1 up1
2 0.009995osd.2 up1
-30.02998host node2
3 0.009995osd.3 up1
4 0.009995osd.4 up1
5 0.009995osd.5 up1
-40.02998host node3
6 0.009995osd.6 up1
7 0.009995osd.7 up1
8 0.009995osd.8 up1