CEPH-4(ceph|CEPH-4:ceph RadowGW对象存储功能详解)

ceph RadosGW对象存储使用详解 一个完整的ceph集群,可以提供块存储、文件系统和对象存储。
本节主要介绍对象存储RadosGw功能如何灵活的使用,集群背景:

$ ceph -s cluster: id:f0a8789e-6d53-44fa-b76d-efa79bbebbcf health: HEALTH_OK services: mon: 1 daemons, quorum a (age 2d) mgr: a(active, since 2d) mds: cephfs:1 {0=cephfs-a=up:active} 1 up:standby-replay osd: 1 osds: 1 up (since 2d), 1 in (since 2d) rgw: 1 daemon active (my.store.a) data: pools:10 pools, 200 pgs objects: 1.29k objects, 3.5 GiB usage:60 GiB used, 798 GiB / 858 GiB avail pgs:200 active+clean io: client:852 B/s rd, 1 op/s rd, 0 op/s wr

什么是对象存储
  1. 对象存储,又称键值存储,通过其接口指令,例如简单的GET、PUT、DEL等,向存储服务上传下载数据;
  2. 对象存储中所有数据都被认为是一个对象。所以,任何数据都可以存入对象存储中,如图片、视频、音频等;
  3. 常见的对象存储厂商有Swift、S3等,ceph就支持Swift API和AWS S3两种标准。
ceph对象存储的构成 Ceph对象存储是通过 RGW组件 来实现,什么是 rgw 呢?
  1. rgw全称Rados Gateway,是一种服务,使客户端能够利用标准对象存储API来访问ceph对象网关;
  2. ceph 0.8版本之后使用Civeweb的web服务器来响应api请求,说白了,rgw里边就是一个web服务;
  3. 客户端使用http/https协议通过RESTful API与rgw通信;
  4. rgw通过librados与ceph集群通信,利用cephx加密协议与ceph存储通信;
  5. rgw通过bucket来实现数据存储和多用户的隔离;
  6. 可以部署多个rgw,实现负载均衡及高可用。
ceph RadosGW中有一个bucket桶的概念,一般项目或者分类会使用bucket来进行隔离,bucket的权限控制,想要操作某个bucket,操作用户必须有对此bucket的对应操作权限,bucket最终的数据其实还是通过PG来落盘到后端的osd存储中的。

RadosGW存储池作用 rgw安装流程请参考之前的部署文档,此处不在赘述,默认端口7480,能够curl通就表示安装成功:
$ curl 10.153.204.13:30080 anonymous

我这里更改了默认端口,改为了30080
rgw安装完毕后,会有一些默认创建的存储池:
$ ceph osd lspools | grep rgw 1 .rgw.root 3 my-store.rgw.control 6 my-store.rgw.meta 7 my-store.rgw.log 8 my-store.rgw.buckets.index 9 my-store.rgw.buckets.non-ec 10 my-store.rgw.buckets.data

这些存储池也是分为两种类型的,一种是元数据存储池,一种是数据存储池:
  • .rgw.root : 包含realm(领域信息),比如zone和zonegroup。
  • rgw.log:存储日志信息,用户记录各种log信息。
  • rgw.control:系统控制池,在有数据更新时,通知其它RGW更新缓存。
  • rgw.meta:元数据存储池,通过不同的名称空间分别存储不同的rados对象,这些名称空间包括用户的UID,及其Bucket映射信息的名称空间users.uid,用户的密钥名称空间users.keys,用户的emai名称空间users.email,用户的subuser的名称空间 users.swift,bucket的名称空间root等。
  • rgw.buckets.index:存放bucket到object的索引信息。
  • rgw.buckets.non-ec:数据的额外信息存储池。
  • rgw.buckets.data:存放对象的数据
RadosGW常用操作详解 查看全部zone
$ radosgw-admin zone list { "default_info": "a06a6df5-68a4-47f0-9afa-2ac1c09aee58", "zones": [ "my-store" ] }

默认为default,我这里更改名字叫my-store
查看zone详情
$ radosgw-admin zone get --rgw-zone=my-store { "id": "a06a6df5-68a4-47f0-9afa-2ac1c09aee58", "name": "my-store", "domain_root": "my-store.rgw.meta:root", "control_pool": "my-store.rgw.control", "gc_pool": "my-store.rgw.log:gc", "lc_pool": "my-store.rgw.log:lc", "log_pool": "my-store.rgw.log", "intent_log_pool": "my-store.rgw.log:intent", "usage_log_pool": "my-store.rgw.log:usage", "roles_pool": "my-store.rgw.meta:roles", "reshard_pool": "my-store.rgw.log:reshard", "user_keys_pool": "my-store.rgw.meta:users.keys", "user_email_pool": "my-store.rgw.meta:users.email", "user_swift_pool": "my-store.rgw.meta:users.swift", "user_uid_pool": "my-store.rgw.meta:users.uid", "otp_pool": "my-store.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "my-store.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "my-store.rgw.buckets.data" } }, "data_extra_pool": "my-store.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "" }

radosgw创建新用户认证
$ radosgw-admin user create --uid="vfan" --display-name="my vfan"{ "user_id": "vfan", "display_name": "my vfan", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "vfan", "access_key": "Q6VGP3LYMH99D0A9GUV0", "secret_key": "NVDfq7CBJgpUnCXKqbgVuKvI3siWNbx0sRltClA4" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] }

新建一个子用户
为了给用户新建一个子用户 (Swift 接口) ,必须为该子用户指定用户的 ID(--uid={username}),子用户的 ID 以及访问级别:
$ radosgw-admin subuser create --uid=vfan --subuser=vfan:swift --access=full { "user_id": "vfan", "display_name": "my vfan", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [ { "id": "vfan:swift", "permissions": "full-control" } ], "keys": [ { "user": "vfan", "access_key": "Q6VGP3LYMH99D0A9GUV0", "secret_key": "NVDfq7CBJgpUnCXKqbgVuKvI3siWNbx0sRltClA4" } ], "swift_keys": [ { "user": "vfan:swift", "secret_key": "GrjjD8yJgr2khUCIeRmggNMWqnganFlhMKMMom9s" } ], "caps": [], "op_mask": "read, write, delete", }

--access=full并不仅仅代表读写,因为他还包括访问权限策略。
查看user列表
$ radosgw-admin user list [ "vfan", "ceph-object-user" ]

禁用或启动一个用户
创建账户后,默认是启用状态,可以将其设置为关闭状态:
## 停用一个用户 $ radosgw-admin user suspend --uid=vfan## 启用一个用户 $ radosgw-admin user enable --uid=vfan

主要是用户中的"suspended"值发生了变化,开启为0,关闭为1。
添加或删除 用户管理权限
## 添加 $ radosgw-admin caps add --uid=vfan --caps="users=*"## 删除 $ radosgw-admin caps rm --uid=vfan --caps="users=write"

--caps="[users|buckets|metadata|usage|zone]=[*|read|write|read, write]"
删除用户 或 子用户
## 删除用户 $ radosgw-admin user rm --uid=vfan## 删除子用户 $ radosgw-admin subuser rm --subuser=vfan:swift

查看所有的bucket桶
$ radosgw-admin bucket list [ "my-test-bucket" ]

查看桶内对象
$ radosgw-admin bucket list --bucket=my-test-bucket [ { "name": "hello.txt", "instance": "", "ver": { "pool": 10, "epoch": 1 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 12, "mtime": "2022-03-30T10:51:38.420295Z", "etag": "ed076287532e86365e841e92bfc50d8c", "storage_class": "", "owner": "vfan", "owner_display_name": "my vfan", "content_type": "application/octet-stream", "accounted_size": 12, "user_data": "", "appendable": "false" }, "tag": "a06a6df5-68a4-47f0-9afa-2ac1c09aee58.24132.17942", "flags": 0, "pending_map": [], "versioned_epoch": 0 } ]

查看存储桶详情
$ radosgw-admin bucket stats --bucket=my-test-bucket { "bucket": "my-test-bucket", "num_shards": 0, "tenant": "", "zonegroup": "fd710024-4ba3-41bb-9f96-579d8f03dd1b", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "a06a6df5-68a4-47f0-9afa-2ac1c09aee58.24134.1", "marker": "a06a6df5-68a4-47f0-9afa-2ac1c09aee58.24134.1", "index_type": "Normal", "owner": "vfan", "ver": "0#2", "master_ver": "0#0", "mtime": "2022-03-30T10:51:38.323147Z", "creation_time": "2022-03-30T10:51:38.321498Z", "max_marker": "0#", "usage": { "rgw.main": { "size": 12, "size_actual": 4096, "size_utilized": 12, "size_kb": 1, "size_kb_actual": 4, "size_kb_utilized": 1, "num_objects": 1 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } }

查看用户配额
$ radosgw-admin user info --uid=vfan | grep -A 5 "quota" "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 -- "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1

默认这些配额都是未激活的,处于false状态。
激活用户配额
$ radosgw-admin quota enable --quota-scope=user --uid=vfan$ radosgw-admin user info --uid=vfan | grep -A 5 "quota" "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 -- "user_quota": { "enabled": true, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1

已激活用户配额,此时可以修改最大限额,默认是不限制。
更新配额
$ radosgw-admin quota set --uid=vfan --quota-scope=user --max-objects=10000 --max-size=107374182400 [cephadmin@yq01-aip-aikefu10.yq01.baidu.com ~]$ radosgw-admin user info --uid=vfan | grep -A 5 "quota" "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 -- "user_quota": { "enabled": true, "check_on_raw": false, "max_size": 107374182400, "max_size_kb": 104857600, "max_objects": 10000

max_size单位是bytes,max_size_kb单位是kb。
操纵radosgw 一般对象存储都由开发在代码层面控制,几乎不需要我们运维人员操作什么,只需要把用户权限和集群维护好就没啥问题了,接下来用一段python代码来演示其bucket以及增删文件的操作。也有一些命令可以实现,例如s3cmd等。
需要先安装好python3环境,以及python的boto模块
# pip3 install boto-2.41.0-py2.py3-none-any.whl

如果没有pip源,离线下载地址:https://pypi.org/simple/boto/
python脚本编写
这里测试使用上边演示新创建的用户vfan
vi ceph-s3.py
import boto.s3.connectionaccess_key = 'Q6VGP3LYMH99D0A9GUV0' #创建S3用户时返回的AK secret_key = 'NVDfq7CBJgpUnCXKqbgVuKvI3siWNbx0sRltClA4' #S3用户的SK host = '10.153.204.13' # RWG节点IP和端口 port = 30080 # 新建一个连接 conn = boto.connect_s3( aws_access_key_id=access_key, aws_secret_access_key=secret_key, host=host, port=port, is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(), ) # 新建一个Bucket bucket = conn.create_bucket('my-vfan-bucket')# 列出用户的所有Bucket for bucket in conn.get_all_buckets(): print("桶名称: %s, 创建时间: %s" %(bucket.name,bucket.creation_date))# 列出Bucket内容 for key in bucket.list(): print("key名称: %s, 文件大小: %s, 修改时间: %s" %(key.name,key.size,key.last_modified))# 新建一个对象 key = bucket.new_key('hi.txt') key.set_contents_from_string('Hello World!')# 下载一个对象到文件 key = bucket.get_key('hi.txt') key.get_contents_to_filename('/tmp/hi.txt')

执行py脚本
# python3 ceph-s3.py 桶名称: my-test-bucket, 创建时间: 2022-03-30T10:51:38.321Z 桶名称: my-vfan-bucket, 创建时间: 2022-04-01T07:32:54.671Z# cat /tmp/hi.txt Hello World!

已经新创建了一个名为my-vfan-bucket的bucket,并新建了一个对象hi.txt,并下载到了本地的/tmp目录下。
可以优化一下脚本,使其可以单项操作
#!/usr/bin/python # -*- coding: utf-8 -*-""" @Time: 2021-12-22 19:14 @Author: xxxxxx @Email: xxxxxx @File: bucket.py @Software: PyCharm """ import boto import boto.s3.connectionclass Bucket(): """ ceph中bucket相关的类 boto s3 api手册:http://boto.readthedocs.org/en/latest/ref/s3.html boto s3 api用法:https://docs.ceph.com/en/latest/radosgw/s3/python/# """ def __init__(self, ak, sk, host, port): self.ak = ak self.sk = sk self.host = host self.port = port self.conn = boto.connect_s3(aws_access_key_id=self.ak, aws_secret_access_key=self.sk, host=self.host, port=self.port, is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat()) print self.conndef bucketList(self): """ 获取所有的bucketList :return: """ for bucket in self.conn.get_all_buckets(): print("{name}\t{created}".format(name=bucket.name, created=bucket.creation_date))def bucketCreate(self, bucketName): """ 创建bucket :return: """ createRes = self.conn.create_bucket(bucketName) print createResdef bucketDelete(self): """ 删除bucket :return: """ passif __name__ == "__main__": """ 主函数 """ access_key = "FHPC3HED7P7J8ADFQVOD" secret_key = "Zgf01sjynnAbNS6yCO99VFphDQ6sOlmPBRRd7P2E" host = "xxxxx" port = 8000 bucketName = 'share' bucket = Bucket(access_key, secret_key, host, port) # 创建bucket bucket.bucketCreate(bucketName) # 查看bucket列表 # bucket.bucketList()

可以再基于此脚本优化,增加其他功能。
【CEPH-4(ceph|CEPH-4:ceph RadowGW对象存储功能详解)】RadosGW相关操作至此已演示介绍完毕,后续会陆续介绍一些自定义crush规则、pg及一些常用的参数配置。

    推荐阅读