一、Ceph技术栈概述
1，Ceph组件介绍
| 组件名称 | 解释说明 |
|---|---|
| MON管理节点 | 【主要功能】： :one:记录OSD的集群地br>:two:记录OSD集群状态，谁在线？谁故障？ |
| OSD组件 | 【主要功能】： :one:直接管理磁盘，读写数据、复制备份数据、数据恢复； :two:每新增一个物理磁盘，自动启动一个OSD进程自动管理； |
| MGR组件（非必须） | Ceph集群的监控，就是Ceph的仪表盘，监控其他所有组件的存活状态，主要用于运维使用； 【主要功能】： :one:【故障自愈】:标记故障的OSD，触发PG的副本重平衡； :two:【负载均衡】:动态优化PG的分布位置； :three:【存储分层】:自动迁移冷热数据到固态硬盘和机械硬盘； :four:【dashboad仪表盘】:web页面可视化监控； :five:【prometheus指标暴露】:暴露集群metrics指标给prometheus； |
| MDS组件 | 【主要功能：CephFS】: :one:主要用于CephFS网络文件系统的存储方式； |
| RGW组件 | 【主要功能：对象存储】rados gateway: :one:对象存储网关：对象存储的核心组件； :two:存储对象（文件、音视频等）； |
2，Ceph存储分类
| 存储分类 | 解释说明 |
|---|---|
| RBD块存储 | 模拟成物理磁盘，供业务当做硬件磁盘来使用； |
| CephFS文件系统 | 类似NFS网络文件系统，NFS是单点网络文件系统，CephFS是集群网络文件系统 |
| RGW对象存储 | 用于业务对象存储：音视频、图片等文件的使用；基于http请求实现； |
3，Ceph存储原理:star:
| 存储步骤 | 说明 |
|---|---|
| :one:用户客户端提交 | 用户使用ceph客户端存储数据到集群 |
| :two:文件对象拆分 | ceph将用户存储的对象拆分为每个存储对象4M； 再生成一个相同的对象id用于证明多个对象同属一个文件 |
| :three:计算存储到目标PG | ceph通过【hash（文件id）% pg数】选中一个PG进行归属存储； |
| :four:Crush算法计算存储目标OSD | ceph通过Crush算法，和需求副本数，计算出要存储的目标OSD，存储成功； |
# 概念；
- pg：逻辑分组；决定归属分组；
- pgp: 物理映射；决定最终存储到那个osd硬盘；
4，集群规模介绍
· 企业级集群数量
| 企业级集群规模 | MON管理 | RGW网关 | MDS元数据 | OSD存储 | 说明 |
|---|---|---|---|---|---|
| 最小集群 | 3台 | 2台 | 2台 | 3台 | 满足基础高可用 |
| 中型集群 | 3台 | 2-4台 | 2-4台 | 6-15台 | 支持PB级别数据 |
| 大型集群 | 5台 | 4+台 | 4+台 | 21+台 | 多机房、多区域存储 |
· 企业级性能要求
| 节点类型 | CPU | 内存 | 磁盘 | 注意 |
|---|---|---|---|---|
| MON管理节点 | 4-8核 | 16-32GB | 1-2TB | 每增加1w台OSD需要增加8GB内存 |
| RGW网管节点 | 16-32核 | 64-128GB | 1TB | 每增加一个RGW进程需要增加4-8核CPU |
| MDS元数据节点 | 16-64核 | 128-512GB | 1-2TB | 每增加100万个文件需新增30GB缓存内存 |
| OSD存储节点 | 1-2核 | 4-8GB | 12-36块磁盘 | 建议80%容量利用率留缓冲 |
· 学习环境
| 节点名称 | ip地址 | 性能 | 磁盘 | 各节点作用 |
|---|---|---|---|---|
| ceph01 | 10.0.0.141 | 2c4g | 100GB：系统盘 300GB：存储磁盘 500GB：存储磁盘 1TB：存储磁盘 |
【MON管理节点】 【MGR监控节点】 【RGW对象存储节点】 【OSD节点】 |
| ceph02 | 10.0.0.142 | 2c4g | 100GB：系统盘 300GB：存储磁盘 500GB：存储磁盘 1TB：存储磁盘 |
【MON管理节点】 【MGR监控节点】 【RGW对象存储节点】 【OSD节点】 |
| ceph03 | 10.0.0.143 | 2c4g | 100GB：系统盘 300GB：存储磁盘 500GB：存储磁盘 1TB：存储磁盘 |
【MON管理节点】 【OSD节点】 |
| ceph04 | 10.0.0.144 | 2c4g | 100GB：系统盘 300GB：存储磁盘 500GB：存储磁盘 1TB：存储磁盘 |
【OSD节点】 |
[root@ceph01:~]# ll /dev/sd*
brw-rw---- 1 root disk 8, 0 Jul 25 13:39 /dev/sda
brw-rw---- 1 root disk 8, 1 Jul 25 13:38 /dev/sda1
brw-rw---- 1 root disk 8, 2 Jul 25 13:38 /dev/sda2
brw-rw---- 1 root disk 8, 3 Jul 25 13:38 /dev/sda3
brw-rw---- 1 root disk 8, 16 Jul 25 13:39 /dev/sdb
brw-rw---- 1 root disk 8, 32 Jul 25 13:39 /dev/sdc
brw-rw---- 1 root disk 8, 48 Jul 25 13:39 /dev/sdd
[root@ceph01:~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 63.9M 1 loop /snap/core20/2105
loop1 7:1 0 40.4M 1 loop /snap/snapd/20671
loop2 7:2 0 87M 1 loop /snap/lxd/27037
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 2G 0 part /boot
└─sda3 8:3 0 98G 0 part
├─ubuntu--vg-lv--0 253:0 0 4G 0 lvm [SWAP]
└─ubuntu--vg-lv--1 253:1 0 94G 0 lvm /
sdb 8:16 0 300G 0 disk
sdc 8:32 0 500G 0 disk
sdd 8:48 0 1T 0 disk
sr0 11:0 1 2G 0 rom
5，ceph版本选择
ceph版本选择：
- Active版本：长期维护稳定版；
- 版本号选择：17.2、18.2、19.2等；
- 版本号中间位是偶数的版本为稳定版；
- 如官网（下图）所示，
- 18.2.7版本到2025年8月停止维护；
- 19.2.2版本到2026年9与停止维护；
- 所以，我们选择【19.2.2版本】。
#官网：https://docs.ceph.com
二、安装部署ceph集群
1，所有节点部署docker
官方推荐：
- ceph节点组件建议使用容器进行维护；
- 所以我们需要在每一台机器上部署docker或者podman；
· ceph01部署docker
1，上传解压docker
[root@ceph01:~]# rz -E
[root@ceph01:~]# ls -l
total 65428
-rw-r--r-- 1 root root 66990280 Jul 18 14:07 docker-20.10.24.tgz
[root@ceph01:~]# tar xf docker-20.10.24.tgz
[root@ceph01:~]# chown root.root docker/*
[root@ceph01:~]# cp docker/* /usr/bin/
[root@ceph01:~]# echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
[root@ceph01:~]# sysctl -p
2，编辑启动文件
[root@ceph01:~]# vim /lib/systemd/system/docker.service
[Unit]
Description=bakwite docker
Documentation=https://www.bakwite.com
#启动docker时，需要什么程序已经准备好？？？（没启动成功页没关系）
After=network-online.target containerd.service
#如果这个程序没准备好，依旧可以启动docker
Wants=network-online.target
[Service]
Type=notify #systemd配置，表示，docker进程启动好running/active之后再进行下一步操作；
#服务启动时执行的命令
ExecStart=/usr/bin/dockerd
# 重启进程，$MAINPID表示docker的pid变量（systemd生成的）
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0 #docker停止时等待时间，0表示伊吾县等待其优雅停止，避免强制杀死
LimitNOFILE=infinity #不限制文件打开数
LimitNPROC=infinity #不限制系统进程打开数
[Install]
WantedBy=multi-user.target
3，启动docker
[root@ceph01:~]# systemctl daemon-reload
[root@ceph01:~]# systemctl enable docker.service
[root@ceph01:~]# systemctl start docker.service
[root@ceph01:~]# systemctl status docker.service
4，测试docker运行情况
[root@ceph01:~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[root@ceph01:~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
· 同步其他节点
[root@ceph01:~]# scp docker/* 10.0.0.142:/usr/bin/
[root@ceph01:~]# scp docker/* 10.0.0.143:/usr/bin/
[root@ceph01:~]# scp docker/* 10.0.0.144:/usr/bin/
[root@ceph01:~]# scp /lib/systemd/system/docker.service 10.0.0.142:/lib/systemd/system/
[root@ceph01:~]# scp /lib/systemd/system/docker.service 10.0.0.143:/lib/systemd/system/
[root@ceph01:~]# scp /lib/systemd/system/docker.service 10.0.0.144:/lib/systemd/system/
· 其他节点启动
systemctl daemon-reload
systemctl enable docker.service
systemctl start docker.service
2，所有节点hosts解析
[root@ceph01:~]# cat >> /etc/hosts<<EOF
10.0.0.141 ceph01
10.0.0.142 ceph02
10.0.0.143 ceph03
10.0.0.144 ceph04
EOF
#将hosts文件同步其他主机
[root@ceph01:~]# scp /etc/hosts 10.0.0.142:/etc/
[root@ceph01:~]# scp /etc/hosts 10.0.0.143:/etc/
[root@ceph01:~]# scp /etc/hosts 10.0.0.144:/etc/
3，下载cephadm文件
· 官网地址
https://download.ceph.com/rpm-19.2.2/el9/noarch/cephadm
· 配置cephadm文件
1，上传cephadm
[root@ceph01:~]# rz -E
[root@ceph01:~]# ls -l
-rw-r--r-- 1 root root 787576 Jul 25 17:16 cephadm
2，给cephadm执行权限
[root@ceph01:~]# chmod +x cephadm
[root@ceph01:~]# ls -l
-rwxr-xr-x 1 root root 787576 Jul 25 17:16 cephadm
3，将cephadm移动到PATH路径
[root@ceph01:~]# cp cephadm /usr/local/bin/
4，基于cephadm部署
· 开启科学上网代理(不用开)
[root@ceph01:~]# export http_proxy=http://10.0.0.1:7897
[root@ceph01:~]# export https_proxy=http://10.0.0.1:7897
· 初始化主节点
1，安装python3环境（不用做）
注意：ubuntu系统默认自带python3，所以不需要安装；
[root@ceph01:~]# python3 --version
Python 3.10.12
#如果不存在再安装
- ubuntu
[root@ceph01:~]# apt -y install python3
- centos/kylin
[root@ceph01:~]# yum -y install python3
2，初始化集群MON主节点
初始化命令含义：在这个节点上部署【MON管理组件】和【MGR监控组件】两个组件
- 【cephadm bootstrap】：初始化命令；
- 【--mon-ip 10.0.0.141】：指定mon进程绑定的ip地址；
- 【--cluster-network 10.0.0.0/24】：表示所有节点所在的集群网段（特别是OSD）；
- 【--allow-fqdn-hostname】：允许主机名称包含【.】，也就是允许主机名可以设置成域名的形式；
[root@ceph01:~]# cephadm bootstrap --mon-ip 10.0.0.141 --cluster-network 10.0.0.0/24 --allow-fqdn-hostname
Creating directory /etc/ceph for ceph.conf
......
Host looks OK
Cluster fsid: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
Verifying IP 10.0.0.141 port 3300 ...
Verifying IP 10.0.0.141 port 6789 ...
Mon IP `10.0.0.141` is in CIDR network `10.0.0.0/24`
Mon IP `10.0.0.141` is in CIDR network `10.0.0.0/24`
Pulling container image quay.io/ceph/ceph:v19...
Ceph version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
.....
Ceph Dashboard is now available at:
URL: https://ceph01:8443/
User: admin
Password: bxxwes697g
...
Bootstrap complete.
提示：Kylin系统记得删除podman
[root@docker01 ~]# rm -rf /usr/bin/podman
如果之前初始化失败过，需要重新初始化那么需要删除残留的元数据
# 查看集群ID
[root@docker01 ~]# cephadm ls
# 清理集群元数据
[root@docker01 ~]# cephadm rm-cluster --force --zap-osds --fsid <集群ID>
4，访问web页面
https://10.0.0.141:8443/
#
URL: https://ceph01:8443/
User: admin
Password: bxxwes697g
#首次登录 修改密码为
- cephadmin
5，查看cephadm拉取的镜像
[root@ceph01:~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/ceph/ceph v19 4892a7ef541b 3 months ago 1.31GB
quay.io/prometheus/prometheus v2.51.0 1d3b7f56885b 16 months ago 262MB
quay.io/ceph/grafana 10.4.0 c8b91775d855 16 months ago 430MB
quay.io/prometheus/node-exporter v1.7.0 72c9c2088986 20 months ago 22.7MB
quay.io/prometheus/alertmanager v0.25.0 c8568f914cd2 2 years ago 65.1MB
6，查看当前Ceph版本
[root@ceph01:~]# cephadm version
cephadm version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
# 扩展：卸载集群
cephadm shell
ceph fsid
cephadm rm-cluster --force --zap-osds --fsid <你的FSID>
5，客户端管理ceph集群
· 方法一：安装ceph客户端命令
1，cephadm下载软件源
[root@ceph01 ~ ]# ./cephadm add-repo --release squid --gpg-url https://download.ceph.com/keys/release.asc
Installing repo GPG key from https://download.ceph.com/keys/release.asc...
Installing repo file at /etc/apt/sources.list.d/ceph.list...
Updating package list...
Completed adding repo.
2，查看客户端秘钥与配置文件
[root@ceph01:~]# ls -l /etc/ceph/
total 12
-rw------- 1 root root 151 Jul 28 12:39 ceph.client.admin.keyring
-rw-r--r-- 1 root root 171 Jul 28 12:39 ceph.conf
-rw-r--r-- 1 root root 595 Jul 25 17:35 ceph.pub
4，安装ceph管理命令
[root@ceph01:~]# apt update
[root@ceph01:~]# apt -y install ceph-common
# 如果嫌弃慢，可以换成国内源
# 备份原源文件
cp /etc/apt/sources.list.d/ceph.list /etc/apt/sources.list.d/ceph.list.bak
# 替换为国内源（以 Ubuntu 22.04 为例，squid 发行版）
echo "deb https://mirrors.ustc.edu.cn/ceph/debian-squid/ jammy main" > /etc/apt/sources.list.d/ceph.list
# 重新更新
apt update
5，使用命令测试
#1，查看集群状态
[root@ceph01:~]# ceph -s
cluster:
# 整个ceph集群的唯一标识；
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
# 整个集群的健康状态:
# [HEALTH_WARN]:表示健康告警；
# OSD count 0 < osd_pool_default_size 3 表示OSD数量小于3个（默认要大于3）；
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
# 【mon】：集群的管理节点，当前1个（生产环境最少3个）
mon: 1 daemons, quorum ceph01 (age 27m)
# 【mgr】：集群的manager节点，负责收集集群数据指标信息、监控信息，页面展示、告警；
mgr: ceph01.vaajsm(active, since 29m)
# 【osd】：集群中的可用磁盘；
osd: 0 osds: 0 up, 0 in
data:
# 存储池信息，
# pools：存储池数量。pgs：归置组（数据分布的最小单元多个pg组成的pgs组）
pools: 0 pools, 0 pgs
# 对象数量：文件数
objects: 0 objects, 0 B
# 总空间使用情况；
usage: 0 B used, 0 B / 0 B avail
pgs:
# 2,查看命令版本，发现版本比较低，没办法，正常不替换apt源，下载就好了；没事，也能用。
[root@ceph01:~]# ceph --version
ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)
#3，查看主机列表
[root@ceph01:~]# ceph orch host ls
HOST ADDR LABELS STATUS
ceph01 10.0.0.141 _admin
1 hosts in cluster
· 方法二：客户端容器方式
1，cephadm进入客户端容器
[root@ceph01:~]# cephadm shell
Inferring fsid 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
Inferring config /var/lib/ceph/53b83466-693a-11f0-ae5c-3bb1aa4e0ea6/mon.ceph01/config
Not using image 'sha256:4892a7ef541bbfe6181ff8fd5c8e03957338f7dd73de94986a5f15e185dacd51' as it's not in list of non-dangling images with ceph=True label
root@ceph01:/#
2，客户端容器中使用命令
# 容器中查看集群状态
root@ceph01:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum ceph01 (age 35m)
mgr: ceph01.vaajsm(active, since 37m)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
# 容器查看主机列表
root@ceph01:/# ceph orch host ls
HOST ADDR LABELS STATUS
ceph01 10.0.0.141 _admin
1 hosts in cluster
三、集群节点管理
1，添加集群节点
1，免秘钥配置集群节点
注意：这个命令将秘钥发送给目标主机的~/.ssh/authorized_keys文件中了；
[root@ceph01:~]# ssh-copy-id -f -i /etc/ceph/ceph.pub ceph02
[root@ceph01:~]# ssh-copy-id -f -i /etc/ceph/ceph.pub ceph03
2，导出主节点镜像（其实不用做）
[root@ceph01:~]# docker save quay.io/ceph/ceph:v19 > ceph-v19.tar.gz
[root@ceph01:~]# docker save quay.io/prometheus/prometheus:v2.51.0 > ceph-prometheus-2510.tar.gz
[root@ceph01:~]# docker save quay.io/ceph/grafana:10.4.0 > ceph-grafana-1040.tar.gz
[root@ceph01:~]# docker save quay.io/prometheus/node-exporter:v1.7.0 > ceph-exporter-v170.tar.gz
[root@ceph01:~]# docker save quay.io/prometheus/alertmanager:v0.25.0 > ceph-alertmanager-v0250.tar.gz
[root@ceph01:~]# ls -l
total 2140880
-rwxr-xr-x 1 root root 787576 Jul 25 17:16 cephadm
-rw-r--r-- 1 root root 66556416 Jul 28 13:31 ceph-alertmanager-v0250.tar.gz
-rw-r--r-- 1 root root 23868416 Jul 28 13:30 ceph-exporter-v170.tar.gz
-rw-r--r-- 1 root root 438047232 Jul 28 13:29 ceph-grafana-1040.tar.gz
-rw-r--r-- 1 root root 262766080 Jul 28 13:28 ceph-prometheus-2510.tar.gz
-rw-r--r-- 1 root root 1333207552 Jul 28 13:23 ceph-v19.tar.gz
3，其他节点导入镜像（其实不用做）
[root@ceph02:~]# docker load < ceph-v19.tar.gz
[root@ceph02:~]# docker load < ceph-prometheus-2510.tar.gz
[root@ceph02:~]# docker load < ceph-grafana-1040.tar.gz
[root@ceph02:~]# docker load < ceph-exporter-v170.tar.gz
[root@ceph02:~]# docker load < ceph-alertmanager-v0250.tar.gz
4，将镜像传输给其他节点（其实不用做）
[root@ceph01:~]# scp ./* 10.0.0.142:~
[root@ceph01:~]# scp ./* 10.0.0.143:~
[root@ceph01:~]# scp ./* 10.0.0.144:~
5，主节点添加集群主机
root@ceph01:/# ceph orch host add ceph02 10.0.0.142
Added host 'ceph03' with addr '10.0.0.143'
root@ceph01:/# ceph orch host add ceph03 10.0.0.143
Added host 'ceph03' with addr '10.0.0.143'
#查看集群节点；
root@ceph01:/# ceph orch host ls
HOST ADDR LABELS STATUS
ceph01 10.0.0.141 _admin
ceph02 10.0.0.142
ceph03 10.0.0.143
3 hosts in cluster
添加后，查看被添加节点容器列表
页面查看集群节点
https://10.0.0.141:8443/#/hosts
· 移除/驱逐集群主机
1，驱逐集群节点
#1，先驱逐集群一个节点
root@ceph01:/# ceph orch host drain ceph03
Scheduled to remove the following daemons from host 'ceph03'
type id
-------------------- ---------------
ceph-exporter ceph03
node-exporter ceph03
mon ceph03
crash ceph03
# 驱逐后，查看，发现ceph03后面显示：_no_schedule,_no_conf_keyring
# [_no_schedule] 表示不可调度
# [_no_conf_keyring] 表示没有连接秘钥
root@ceph01:/# ceph orch host ls
HOST ADDR LABELS STATUS
ceph01 10.0.0.141 _admin
ceph02 10.0.0.142
ceph03 10.0.0.143 _no_schedule,_no_conf_keyring
3 hosts in cluster
# 删除集群节点
root@ceph01:/# ceph orch host rm ceph03
Removed host 'ceph03'
# 删除后再次查看，才真正消失
root@ceph01:/# ceph orch host ls
HOST ADDR LABELS STATUS
ceph01 10.0.0.141 _admin
ceph02 10.0.0.142
2 hosts in cluster
页面查看，只剩下两个集群节点了
2，删除被驱逐/移除节点的数据
注意：学习环境，我们先不删除，先看一眼
#1,删除集群数据
[root@ceph03:~]# ls -l /var/lib/ceph/
total 4
drwx------ 4 167 167 4096 Jul 28 13:54 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
#2，删除日志数据
[root@ceph03:~]# ls -l /var/log/ceph/
total 108
drwxrwx--- 2 167 167 4096 Jul 28 13:49 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
-rw-r--r-- 1 root root 98555 Jul 28 14:00 cephadm.log
3，添加回来
root@ceph01:/# ceph orch host add ceph03 10.0.0.143
Added host 'ceph03' with addr '10.0.0.143'
root@ceph01:/# ceph orch host ls
HOST ADDR LABELS STATUS
ceph01 10.0.0.141 _admin
ceph02 10.0.0.142
ceph03 10.0.0.143
3 hosts in cluster
2，集群组件高可用配置
· MON高可用
1，将mon主节点秘钥分发给其他mon
[root@ceph01:~]# scp -r /etc/ceph 10.0.0.142:/etc/
[root@ceph01:~]# scp -r /etc/ceph 10.0.0.143:/etc/
2，将三个节点都配置成mon
# 1，将三个节点都设置成MON
root@ceph01:/# ceph orch apply mon --placement="ceph01 ceph02 ceph03"
Scheduled mgr update...
# 2，查看集群状态
root@ceph01:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 25m)
mgr: ceph02.mjbyul(active, since 24m), standbys: ceph01.vaajsm, ceph03.bocrhz
osd: 9 osds: 9 up (since 24m), 9 in (since 23h)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 870 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 1 active+clean
# 3.检查集群MON状态
root@ceph01:/# ceph orch ps --daemon-type mon
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID
mon.ceph01 ceph01 running (26m) 4m ago 3d 67.0M 2048M 19.2.2 4892a7ef541b
mon.ceph02 ceph02 running (24h) 4m ago 24h 424M 2048M 19.2.2 4892a7ef541b
mon.ceph03 ceph03 running (23h) 4m ago 23h 412M 2048M 19.2.2 4892a7ef541b
# 4，查看mon状态
root@ceph01:/# ceph mon stat
e5: 3 mons at {
ceph01=[v2:10.0.0.141:3300/0,v1:10.0.0.141:6789/0],
ceph02=[v2:10.0.0.142:3300/0,v1:10.0.0.142:6789/0],
ceph03=[v2:10.0.0.143:3300/0,v1:10.0.0.143:6789/0]
} removed_ranks: {} disallowed_leaders:
{}, election epoch 42, leader 0 ceph01, quorum 0,1,2 ceph01,ceph02,ceph03
3，启动客户端测试查看
[root@ceph03:~]# ./cephadm shell
root@ceph03:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 34m)
mgr: ceph02.mjbyul(active, since 34m), standbys: ceph01.vaajsm, ceph03.bocrhz
osd: 9 osds: 9 up (since 34m), 9 in (since 23h)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 870 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 1 active+clean
· MGR高可用
# 1，将三个节点都设置成MGR
root@ceph01:/# ceph orch apply mgr --placement="ceph01 ceph02 ceph03"
Scheduled mgr update...
# 2，查看集群状态
root@ceph01:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 3h)
mgr: ceph01.vaajsm(active, since 24h), standbys: ceph02.mjbyul, ceph03.bocrhz
osd: 9 osds: 9 up (since 22h), 9 in (since 22h)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 1.1 GiB used, 5.3 TiB / 5.3 TiB avail
pgs: 1 active+clean
# 3，检查集群mgr状态
root@ceph01:/# ceph orch ps --daemon-type mgr
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION
mgr.ceph01.vaajsm ceph01 *:9283,8765,8443 running (24h) 2m ago 3d 342M - 19.2.2
mgr.ceph02.mjbyul ceph02 *:8443,9283,8765 running (23h) 48s ago 23h 508M - 19.2.2
mgr.ceph03.bocrhz ceph03 *:8443,9283,8765 running (3m) 3m ago 3m 107M - 19.2.2
# 4，查看MGR状态
root@ceph01:/# ceph mgr stat
{
"epoch": 28,
"available": true,
"active_name": "ceph01.vaajsm", #表示当前活跃（主节点、可访问）
"num_standby": 2
}
3，网络配置（了解）
· 添加OSD集群网落
OSD 间同步的后端网络（建议物理隔离）
root@ceph01:/# ceph config set osd cluster_network "10.0.0.0/24,172.16.1.0/16"
· 添加公共网络
客户端访问的 前端网络白名单设置
root@ceph01:/# ceph config set mon public_network "10.0.0.0/24,172.16.1.0/16"
三、OSD磁盘管理
1，添加OSD到集群
前提条件：存储磁盘想要加入集群管理；
- 首先磁盘必须大于5GB；
- 其次设备当前未被使用【此时此刻未被挂载到文件系统】；
- 注意：首次加入速度很慢哦~
· 查看集群可用的磁盘
root@ceph01:/# ceph orch device ls
# 如果没显示，手动刷新
root@ceph01:/# ceph orch device ls --refresh
· 添加ceph01节点的磁盘到集群
| 列 | ||
|---|---|---|
root@ceph01:/# ceph orch daemon add osd ceph01:/dev/sdb
root@ceph01:/# ceph orch daemon add osd ceph01:/dev/sdc
root@ceph01:/# ceph orch daemon add osd ceph01:/dev/sdd
root@ceph01:/# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.78130 root default
-3 1.78130 host ceph01
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
· 添加ceph02节点的磁盘到集群
root@ceph01:/# ceph orch daemon add osd ceph02:/dev/sdb
root@ceph01:/# ceph orch daemon add osd ceph02:/dev/sdc
root@ceph01:/# ceph orch daemon add osd ceph02:/dev/sdd
root@ceph01:/# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 3.56259 root default
-3 1.78130 host ceph01
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
-5 1.78130 host ceph02
3 hdd 0.29300 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
· 添加ceph03节点的磁盘到集群
root@ceph01:/# ceph orch daemon add osd ceph03:/dev/sdb
Created osd(s) 6 on host 'ceph03'
root@ceph01:/# ceph orch daemon add osd ceph03:/dev/sdc
Created osd(s) 7 on host 'ceph03'
root@ceph01:/# ceph orch daemon add osd ceph03:/dev/sdd
Created osd(s) 8 on host 'ceph03'
root@ceph01:/# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 5.34389 root default
-3 1.78130 host ceph01
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
-5 1.78130 host ceph02
3 hdd 0.29300 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
-7 1.78130 host ceph03
6 hdd 0.29300 osd.6 up 1.00000 1.00000
7 hdd 0.48830 osd.7 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
2，查看OSD信息
· 查看集群可用磁盘硬件
只要集群中有没有使用的磁盘，集群都会监控的到；
root@ceph01:/# ceph orch device ls
· 查看目前OSD列表
[root@ceph01:~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 7.68745 root default
-3 2.56248 host ceph01
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
-5 2.56248 host ceph02
3 hdd 0.29300 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
-7 2.56248 host ceph03
6 hdd 0.29300 osd.6 up 1.00000 1.00000
7 hdd 0.48830 osd.7 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
· 查看OSD状态
# 查看OSD状态间接
[root@ceph01:~]# ceph osd stat
12 osds: 12 up (since 61m), 12 in (since 62m); epoch: e1369
# 查看OSD详细状态
[root@ceph01:~]# ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 ceph01 96.4M 299G 0 0 0 0 exists,up
1 ceph01 111M 499G 0 0 0 0 exists,up
2 ceph01 184M 1023G 0 0 0 0 exists,up
3 ceph02 97.7M 299G 0 0 0 0 exists,up
4 ceph02 117M 499G 0 0 0 0 exists,up
5 ceph02 167M 1023G 0 0 0 0 exists,up
6 ceph03 114M 299G 0 0 0 0 exists,up
7 ceph03 124M 499G 0 0 0 0 exists,up
8 ceph03 192M 1023G 0 0 1 0 exists,up
# 查看OSD详细使用信息
[root@ceph01:~]# ceph osd df
· 查看OSD延迟信息
| 字段 | 解释说明 |
|---|---|
| commit_latency(ms) | 【提交延迟】：写入数据到【Journal日志】所花费的平均延迟； 【注意】：ceph集群中，数据优先写入Journal日志，在写入到OSD磁盘； |
| apply_latency(ms) | 【写入延迟】：写入数据到磁盘所花费的平均延迟； |
[root@ceph01:~]# ceph osd perf
osd commit_latency(ms) apply_latency(ms)
11 0 0
10 0 0
9 0 0
8 0 0
7 0 0
6 0 0
5 0 0
4 0 0
3 0 0
2 0 0
1 0 0
0 0 0
· 查看集群状态
| 命令：【ceph -s】 | 字段信息 | 解释说明 |
|---|---|---|
| cluster | 集群信息 | |
| id | 集群的唯一id | |
| health | 健康状态 | |
| services | 集群服务信息 | |
| mon | 【MON】管理节点数量 | |
| mgr | 【MGR】监控服务进程信息 | |
| osd | 【OSD】磁盘进程数量信息 | |
| data | 数据信息 | |
| pools | 存储池信息 | |
| objects | 对象存储信息 | |
| usage | 集群容量与使用情况 | |
| pgs |
root@ceph01:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 6m)
mgr: ceph01.vaajsm(active, since 112m), standbys: ceph02.mjbyul
osd: 9 osds: 9 up (since 2m), 9 in (since 2m)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 243 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 1 active+clean
· 查看物理磁盘列表
查看各个节点的磁盘列表可以看到：
- 整个磁盘被LVM占用了；
- 说明，ceph底层采用了lvm技术；
[root@ceph01:~]# lsblk
LVM逻辑卷原理
#查看【逻辑卷】LVS列表
[root@ceph01:~]# lvs
#查看【卷组】vg
[root@ceph01:~]# vgs
VG #PV #LV #SN Attr VSize VFree
ceph-1a176fce-899a-4a43-951e-2acea4182ee8 1 1 0 wz--n- <500.00g 0
ceph-1b645e89-5393-44d3-91e8-3dea04af39b8 1 1 0 wz--n- <300.00g 0
ceph-840e89cb-b0c7-4384-8242-a1108d2db807 1 1 0 wz--n- <1024.00g 0
ubuntu-vg 1 2 0 wz--n- <98.00g 0
#查看【物理卷】pv
[root@ceph01:~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 ubuntu-vg lvm2 a-- <98.00g 0
/dev/sdb ceph-1b645e89-5393-44d3-91e8-3dea04af39b8 lvm2 a-- <300.00g 0
/dev/sdc ceph-1a176fce-899a-4a43-951e-2acea4182ee8 lvm2 a-- <500.00g 0
/dev/sdd ceph-840e89cb-b0c7-4384-8242-a1108d2db807 lvm2 a-- <1024.00g 0
· 查看各节点容器列表
【docker ps】会发现：
- 每个OSD都单独起了一个容器；
- 每个容器代表添加的一块磁盘；
[root@ceph02:~]# docker ps
3，暂停/恢复OSD写入
· 暂停OSD写入
[root@ceph01:~]# ceph osd pause
pauserd,pausewr is set
# 查看集群状态
[root@ceph01:~]# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_WARN
pauserd,pausewr flag(s) set
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 3h)
mgr: ceph01.vaajsm(active, since 3h), standbys: ceph03.bocrhz, ceph02.mjbyul
mds: 1/1 daemons up, 2 standby
osd: 12 osds: 12 up (since 3h), 12 in (since 3h)
flags pauserd,pausewr
rgw: 3 daemons active (3 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 11 pools, 545 pgs
objects: 433 objects, 294 MiB
usage: 1.7 GiB used, 7.7 TiB / 7.7 TiB avail
pgs: 545 active+clean
· 恢复OSD写入
[root@ceph01:~]# ceph osd unpause
pauserd,pausewr is unset
# 查看状态验证
[root@ceph01:~]# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 3h)
mgr: ceph01.vaajsm(active, since 3h), standbys: ceph03.bocrhz, ceph02.mjbyul
mds: 1/1 daemons up, 2 standby
osd: 12 osds: 12 up (since 3h), 12 in (since 3h)
rgw: 3 daemons active (3 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 11 pools, 545 pgs
objects: 433 objects, 294 MiB
usage: 1.7 GiB used, 7.7 TiB / 7.7 TiB avail
pgs: 545 active+clean
4，OSD写入权重设置:x:
· 查看OSD写入权重
查看OSD列表：
- 【weight】字段：数值越大，表示写入权重越高；写入数据越多，PG数量也就越高；
[root@ceph01:~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 7.68745 root default
-3 2.56248 host ceph01
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
-5 2.56248 host ceph02
3 hdd 0.29300 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
-7 2.56248 host ceph03
6 hdd 0.29300 osd.6 up 1.00000 1.00000
7 hdd 0.48830 osd.7 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
· 设置OSD写入权重
# 将OSD.0的写入权重设置为0，表示不写入数据；
[root@ceph01:~]# ceph osd crush reweight osd.0 0
reweighted item id 0 name 'osd.0' to 0 in crush map
# 可以看到，weight权重已经为0
[root@ceph01:~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
......
0 hdd 0 osd.0 up 1.00000 1.00000
......
# 记得改回去
[root@ceph01:~]# ceph osd crush reweight osd.0 0.29300
5，重启OSD磁盘
OSD下线：本质上就是重启OSD的容器进程；
# 下线3号OSD
[root@ceph01:~]# ceph osd down 3
marked down osd.3.
# 查看OSD列表，会显示状态为down
[root@ceph01:~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
.....
3 hdd 0.29300 osd.3 down 1.00000 1.00000
.....
# 稍等1分钟，集群会自动重启，变回up
[root@ceph01:~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
......
3 hdd 0.29300 osd.3 up 1.00000 1.00000
......
6，关闭/启动OSD磁盘
就是直接关闭OSD的进程
# 关闭OSD.3磁盘进程；
[root@ceph01:~]# ceph orch daemon stop osd.3
Scheduled to stop osd.3 on host 'ceph02'
# 查看OSD列表，发现抑制是down的状态；
[root@ceph01:~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
......
3 hdd 0.29300 osd.3 down 1.00000 1.00000
......
# 启动OSD.3磁盘进程；
[root@ceph01:~]# ceph orch daemon start osd.3
Scheduled to start osd.3 on host 'ceph02'
# 查看OSD列表，发现抑制是up的状态；
[root@ceph01:~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
......
3 hdd 0.29300 osd.3 up 1.00000 1.00000
7，删除OSD节点
· 查看要删除的节点crush映射列表
[root@ceph01:~]# ceph osd crush ls ceph01
osd.0
osd.1
osd.2
[root@ceph01:~]# ceph osd crush ls ceph02
osd.3
osd.4
osd.5
[root@ceph01:~]# ceph osd crush ls ceph03
osd.6
osd.7
osd.8
· 迁移被删除节点OSD磁盘数据
[root@ceph01:~]# ceph osd out osd.3
[root@ceph01:~]# ceph osd out osd.4
[root@ceph01:~]# ceph osd out osd.5
# 这个时候后悔了，就把out改成in，重新写入；
# 数据迁移走后查看(reweight权重为0)
[root@ceph01:~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
......
-9 1.78130 host ceph02
10 hdd 0.29300 osd.3 up 0 1.00000
12 hdd 0.48830 osd.4 up 0 1.00000
13 hdd 1.00000 osd.5 up 0 1.00000
· 停止被删除节点OSD进程
[root@ceph02:~]# systemctl stop ceph-53b83466-693a-11f0-ae5c-3bb1aa4e0ea6@osd.3.service
[root@ceph02:~]# systemctl stop ceph-53b83466-693a-11f0-ae5c-3bb1aa4e0ea6@osd.4.service
[root@ceph02:~]# systemctl stop ceph-53b83466-693a-11f0-ae5c-3bb1aa4e0ea6@osd.5.service
· 移除集群crush map中的osd映射
[root@ceph01:~]# ceph osd crush remove osd.3
[root@ceph01:~]# ceph osd crush remove osd.4
[root@ceph01:~]# ceph osd crush remove osd.5
· 删除osd节点认证秘钥
[root@ceph01:~]# ceph auth del osd.3
[root@ceph01:~]# ceph auth del osd.4
[root@ceph01:~]# ceph auth del osd.5
# 删除秘钥后，权重为0
[root@ceph01:~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
......
-9 0 host ceph02
10 0 osd.3 down 0 1.00000
12 0 osd.4 down 0 1.00000
13 0 osd.5 down 0 1.00000
· 从集群osd磁盘列表中删除
[root@ceph01:~]# ceph osd rm osd.3
[root@ceph01:~]# ceph osd rm osd.4
[root@ceph01:~]# ceph osd rm osd.5
# 从集群的osd列表中删除后，列表中只剩接了主机名,osd已经没有了
[root@ceph01:~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
......
-9 0 host ceph04
· 清理被删除节点
1，删除主机桶（节点在集群的映射）
[root@ceph01:~]# ceph osd crush remove ceph02
2，移除集群主机列表
[root@ceph01:~]# ceph orch host rm ceph02 --force
# 此时查看集群主机列表，节点彻底删除
[root@ceph01:~]# ceph orch host ls
3，被删除主机删除lvm
vgremove -f $( pvs --noheadings -o vg_name /dev/sdb) 2>/dev/null
pvremove /dev/sdb -ff
vgremove -f $( pvs --noheadings -o vg_name /dev/sdc) 2>/dev/null
pvremove /dev/sdc -ff
vgremove -f $( pvs --noheadings -o vg_name /dev/sdd) 2>/dev/null
pvremove /dev/sdd -ff
8，集群新增/缩减磁盘案例
· 集群节点插入新磁盘
三台集群，都插入一块800GB的磁盘，模拟案例
| 节点 | 新增磁盘大小 |
|---|---|
| ceph01 | 800g |
| ceph02 | 800g |
| ceph03 | 800g |
所有节点重启
[root@ceph01:~]# reboot
[root@ceph02:~]# reboot
[root@ceph03:~]# reboot
重启后查看新增磁盘sde
[root@ceph01:~]# lsblk | grep sde
sde 8:64 0 800G 0 disk
· 添加新磁盘OSD
1，查看集群监控到的磁盘
如果查看不到，那么就等一会儿~等ceph集群监控到它；
# 查看集群监控到的磁盘
[root@ceph01:~]# ceph orch device ls
HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS
ceph01 /dev/sdb hdd 300G No 52s ago
ceph01 /dev/sdc hdd 500G No 52s ago
ceph01 /dev/sdd hdd 1024G No 52s ago
ceph01 /dev/sde hdd 800G Yes 52s ago #新增的磁盘
ceph01 /dev/sr0 hdd 2006M No 52s ago
ceph02 /dev/sdb hdd 300G No 52s ago
ceph02 /dev/sdc hdd 500G No 52s ago
ceph02 /dev/sdd hdd 1024G No 52s ago
ceph02 /dev/sde hdd 800G Yes 52s ago #新增的磁盘
ceph02 /dev/sr0 hdd 2006M No 52s ago
ceph03 /dev/sdb hdd 300G No 53s ago
ceph03 /dev/sdc hdd 500G No 53s ago
ceph03 /dev/sdd hdd 1024G No 53s ago
ceph03 /dev/sde hdd 800G Yes 53s ago #新增的磁盘
ceph03 /dev/sr0 hdd 2006M No 53s ago
2，添加新增磁盘到OSD
[root@ceph01:~]# ceph orch daemon add osd ceph01:/dev/sde
[root@ceph01:~]# ceph orch daemon add osd ceph02:/dev/sde
[root@ceph01:~]# ceph orch daemon add osd ceph03:/dev/sde
3，查看添加结果
# 查看OSD列表（已经添加成功）
[root@ceph01:~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 7.68745 root default
-3 2.56248 host ceph01
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
9 hdd 0.78119 osd.9 up 1.00000 1.00000
-5 2.56248 host ceph02
3 hdd 0.29300 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
10 hdd 0.78119 osd.10 up 1.00000 1.00000
-7 2.56248 host ceph03
6 hdd 0.29300 osd.6 up 1.00000 1.00000
7 hdd 0.48830 osd.7 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
11 hdd 0.78119 osd.11 up 1.00000 1.00000
# 查看物理磁盘列表(发现sde已经被lvm挂起了)
[root@ceph01:~]# lsblk
.......
sde 8:64 0 800G 0 disk
└─ceph--50a0e26d--d1ef--43c4--8c34--4c26e37936e3-osd--block--b536b32d--cded--4cd8--b7cb--42dd952fb366
# 查看ceph集群（发现osd变成了12块了）
[root@ceph01:~]# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 19m)
mgr: ceph01.vaajsm(active, since 19m), standbys: ceph03.bocrhz, ceph02.mjbyul
mds: 1/1 daemons up, 2 standby
osd: 12 osds: 12 up (since 2m), 12 in (since 3m)
rgw: 3 daemons active (3 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 11 pools, 545 pgs
objects: 433 objects, 294 MiB
usage: 1.6 GiB used, 7.7 TiB / 7.7 TiB avail
pgs: 545 active+clean
9，新增磁盘怎么加入新集群:star:
1.安装部署docker
#按照上面的步骤
2，添加新节点
cat >> /etc/hosts<<EOF
10.0.0.141 ceph01
10.0.0.142 ceph02
10.0.0.143 ceph03
10.0.0.144 ceph04
EOF
[root@ceph01:~]# ssh-copy-id -f -i /etc/ceph/ceph.pub ceph04
[root@ceph01 ~ ]# ceph orch host add ceph04 10.0.0.144
[root@ceph01 ~ ]# ceph orch host ls
HOST ADDR LABELS STATUS
ceph01 10.0.0.141 _admin
ceph02 10.0.0.142
ceph03 10.0.0.143
ceph04 10.0.0.144
4 hosts in cluster
3，添加osd到集群
[root@ceph01 ~ ]# ceph orch device ls | grep ceph04
ceph04 /dev/sdb hdd 500G Yes 13s ago
ceph04 /dev/sdc hdd 500G Yes 13s ago
ceph04 /dev/sdd hdd 500G Yes 13s ago
...
[root@ceph01 ~ ]# ceph orch daemon add osd ceph04:/dev/sdb
[root@ceph01 ~ ]# ceph orch daemon add osd ceph04:/dev/sdc
[root@ceph01 ~ ]# ceph orch daemon add osd ceph04:/dev/sdd
[root@ceph01 ~ ]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 5.85956 root default
-3 1.46489 host ceph01
0 hdd 0.48830 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 0.48830 osd.2 up 1.00000 1.00000
-5 1.46489 host ceph02
3 hdd 0.48830 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 0.48830 osd.5 up 1.00000 1.00000
-7 1.46489 host ceph03
6 hdd 0.48830 osd.6 up 1.00000 1.00000
7 hdd 0.48830 osd.7 up 1.00000 1.00000
8 hdd 0.48830 osd.8 up 1.00000 1.00000
-9 1.46489 host ceph04
9 hdd 0.48830 osd.9 up 1.00000 1.00000
10 hdd 0.48830 osd.10 up 1.00000 1.00000
11 hdd 0.48830 osd.11 up 1.00000 1.00000
4. 删除旧的OSD节点
[root@ceph01 ~ ]# ceph osd crush ls ceph04
osd.9
osd.10
osd.11
[root@ceph01:~]# ceph osd out osd.9
[root@ceph01:~]# ceph osd out osd.10
[root@ceph01:~]# ceph osd out osd.11
[root@ceph04:~]# systemctl stop ceph-53b83466-693a-11f0-ae5c-3bb1aa4e0ea6@osd.9.service
[root@ceph04:~]# systemctl stop ceph-53b83466-693a-11f0-ae5c-3bb1aa4e0ea6@osd.10.service
[root@ceph04:~]# systemctl stop ceph-53b83466-693a-11f0-ae5c-3bb1aa4e0ea6@osd.11.service
[root@ceph01 ~ ]# ceph orch daemon stop osd.9
[root@ceph01 ~ ]# ceph orch daemon stop osd.10
[root@ceph01 ~ ]# ceph orch daemon stop osd.11
[root@ceph01:~]# ceph auth del osd.9
[root@ceph01:~]# ceph auth del osd.10
[root@ceph01:~]# ceph auth del osd.11
[root@ceph01:~]# ceph osd crush remove osd.9
[root@ceph01:~]# ceph osd crush remove osd.10
[root@ceph01:~]# ceph osd crush remove osd.11
[root@ceph01:~]# ceph osd rm osd.9
[root@ceph01:~]# ceph osd rm osd.10
[root@ceph01:~]# ceph osd rm osd.11
[root@ceph01:~]# ceph osd crush remove ceph04
[root@ceph01 ~ ]# ceph orch host drain ceph04
[root@ceph01 ~ ]# ceph orch host rm ceph04 --force
# lvm解除
vgremove -f $( pvs --noheadings -o vg_name /dev/sdb) 2>/dev/null
pvremove /dev/sdb -ff
vgremove -f $( pvs --noheadings -o vg_name /dev/sdc) 2>/dev/null
pvremove /dev/sdc -ff
vgremove -f $( pvs --noheadings -o vg_name /dev/sdd) 2>/dev/null
pvremove /dev/sdd -ff
四、存储池pool管理
| 名词 | 解释说明 |
|---|---|
| pool存储池 | 存储数据的逻辑空间； |
| size | 副本数量（默认为3），表示存储池中的数据，有多少备份，有几份数据；size=3表示，一个文件，复制了3份进行存储 |
| min_size | 集群最小副本数（默认为2），表示最小可用副本数，当写入数据时，size必须大于等于min_size才能写入； 当3台OSD，故障2台，之前的副本数3变成了1时，集群将无法写入，只能读取； 【生产环境】：采用默认，不设置即可； |
| pg | 放置组，是pool存储池的一个逻辑单元（分片），一个存储池会有多个pg； 就是将整个pool存储池，分成了多个小的逻辑存储空间； |
| pgp | 是pg与osd之间的映射关系的计算参数； 数据通过【pg与副本数+pgp数量】crush算法，计算得出要存储到哪些OSD中？ |
1，存储池副本类型
| pool类型 | 解释说明 | 建议 |
|---|---|---|
| 副本池【replicated】 | 创建pool存储池时，自定义数据存储的副本； | 建议生产环境使用 |
| 纠删码池【erasure】 | 类似于raid5，根据磁盘数量进行单一备份，允许损失一块磁盘； | 不建议使用 |
2，创建pool存储池
· 创建存储池指定副本类型
#1，创建存储池，制定存储池类型为副本池
root@ceph01:/# ceph osd pool create wa01 replicated
pool 'wa01' created
#2，不指定，默认就是副本池
root@ceph01:/# ceph osd pool create wa02
pool 'wa02' created
#3，创建存储池，指定存储池类型为纠删码池
root@ceph01:/# ceph osd pool create wa03 erasure
pool 'wa03' created
· 创建存储池指定副本数
副本数不指定，默认是3
root@ceph01:/# ceph osd pool create wa04 --size 5
pool 'wa04' created
# 查看存储池的副本数
root@ceph01:/# ceph osd pool get wa04 size
size: 5
· 创建存储池指定pg和pgp数量
1，存储池pg大小设置规范
| pg数量设置公式 | 解释说明 |
|---|---|
| OSD数量 * 100 / 副本size数量 | 假设我们得OSD块设备是9个，副本设置为3，那么我们得pg数量：9*100/3==300 官方建议，需要接近2的n次方，那么举例300最近的2的n次方应该是256； 所以：我们的环境需要256个pg最完美； |
2，pg数量自动伸缩案例
不指定pg与pgp数量，默认值是32
- 注意：不论你指定pg数量是多大，pg数量都会自动变成32；
- 因为：查看详细信息时，可以看到有一个参数：
- --autoscale_mode=on 这个是自动伸缩pg的模块，显示处于开启状态；
- 只要这个模块开启，就会自动回溯到32个pg数量上；
# 创建存储池，指定pg和pgp数量为128
root@ceph01:/# ceph osd pool create wa06 128 128
pool 'wa06' created
# 查看存储池详细信息
root@ceph01:/# ceph osd pool ls detail
....
pool 8 'wa06' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 127 pgp_num 122 pg_num_target 32 pgp_num_target 32 autoscale_mode on last_change 571 lfor 0/571/569 flags hashpspool stripe_width 0 read_balance_score 1.98
# 多次查看pg数量，发现pg数量越来越小，直到降低到32个为止；
root@ceph01:/# ceph osd pool get wa06 pg_num
pg_num: 115
root@ceph01:/# ceph osd pool get wa06 pg_num
pg_num: 114
root@ceph01:/# ceph osd pool get wa06 pg_num
pg_num: 113
3，关闭pg自动伸缩
# 创建存储池，关闭弹性伸缩pg模块--autoscale-mode=off
root@ceph01:/# ceph osd pool create wa07 128 128 --autoscale-mode=off
pool 'wa07' created
# 再次多次查看pg数量，发现，没有变化了；
root@ceph01:/# ceph osd pool get wa07 pg_num
pg_num: 128
root@ceph01:/# ceph osd pool get wa07 pgp_num
pgp_num: 128
root@ceph01:/# ceph osd pool get wa07 pg_num
pg_num: 128
root@ceph01:/# ceph osd pool get wa07 pgp_num
pgp_num: 128
# 查看存储池详细信息[autoscale_mode off]发现已关闭
root@ceph01:/# ceph osd pool ls detail
......
pool 9 'wa07' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode off last_change 903 flags hashpspool stripe_width 0 read_balance_score 2.25
3，查看存储池
· 查看存储池列表详细信息
root@ceph01:/# ceph osd pool ls detail
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 9.09
pool 2 'bakwite_pool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 69 lfor 0/0/67 flags hashpspool stripe_width 0 read_balance_score 1.97
pool 3 'wa01' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 84 lfor 0/0/81 flags hashpspool stripe_width 0 read_balance_score 2.25
pool 4 'wa02' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 84 lfor 0/0/81 flags hashpspool stripe_width 0 read_balance_score 2.25
pool 5 'wa03' erasure profile default size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 89 lfor 0/0/87 flags hashpspool stripe_width 8192
· 查看存储池列表
# 正常人类写法
root@ceph01:/# ceph osd pool ls
.mgr
bakwite_pool
wa01
wa02
wa_03
# 简写方式
root@ceph01:/# ceph osd lspools
1 .mgr
2 bakwite_pool
3 wa01
4 wa02
5 wa03
· 查看一个存储池副本数
root@ceph01:/# ceph osd pool get wa02 size
size: 5
· 查看存储池的pg数量
root@ceph01:/# ceph osd pool get wa05 pg_num
pg_num: 128
· 查看存储池使用状态
root@ceph01:/# ceph osd pool stats
pool .mgr id 1
nothing is going on
pool bakwite_pool id 2
nothing is going on
pool wa01 id 3
nothing is going on
pool wa02 id 4
nothing is going on
pool wa_03 id 5
nothing is going on
4，修改存储池
· 修改存储池副本数
# 将存储池副本数修改为：5个副本
root@ceph01:/# ceph osd pool set wa02 size 5
set pool 4 size to 5
# 查看详细pool列表,发现size从原来的默认3个副本，变成了5个副本
root@ceph01:/# ceph osd pool ls detail
......
pool 4 'wa02' replicated size 5 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 92 lfor 0/0/81 flags hashpspool stripe_width 0 read_balance_score 2.25
...
· 修改存储池名称
root@ceph01:/# ceph osd pool rename wa03 wa_03
pool 'wa03' renamed to 'wa_03'
# 修改后查看列表，名称已经修改；
root@ceph01:/# ceph osd pool ls
1 .mgr
2 bakwite_pool
3 wa01
4 wa02
5 wa_03
· 修改pg数量
root@ceph01:/# ceph osd pool set wa01 pg_num 128
set pool 3 pg_num to 128
# 查看pg数量
root@ceph01:/# ceph osd pool get wa01 pg_num
pg_num: 128
5，删除存储池
注意：
- 因此ceph设置了两个安全机制：【nodelete】和【mon_allow_pool_delete】
- 想要允许存储池pool被删除：
- 首先【nodelete】必须设置为：false
- false，表示存储池允许被删除
- true，表示存储池不允许被删除；
- 同时通过【mon_allow_pool_delete】标记告诉mon集群，允许删除存储池；
- false（默认），表示不允许被删除，
- true，表示允许删除存储池；
- 这两种机制，缺一不可，必须全部为true，存储池才允许被删除；
· 查看存储池nodelete删除标记
true表示不允许删除存储池，所以必须为false才能删除；
root@ceph01:/# ceph osd pool get wa01 nodelete
nodelete: false #false表示允许删除；
· 删除存储池报错案例
演示删除报错
# 删除一个存储池，查看【报错】提示：
root@ceph01:/# ceph osd pool delete wa01
Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool wa01. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it.
# 根据提示：需要将名称写两次，并追加参数--yes-i-really-really-mean-it，【但是，依旧报错】；
root@ceph01:/# ceph osd pool delete wa01 wa01 --yes-i-really-really-mean-it
Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool
· 彻底删除存储池
#1，告诉mon集群节点，允许删除存储池pool；
root@ceph01:/# ceph tell mon.* injectargs --mon_allow_pool_delete=true
mon.ceph01: {}
mon.ceph01: mon_allow_pool_delete = ''
mon.ceph02: {}
mon.ceph02: mon_allow_pool_delete = ''
mon.ceph03: {}
mon.ceph03: mon_allow_pool_delete = ''
#2，再次执行删除命令；
root@ceph01:/# ceph osd pool delete wa01 wa01 --yes-i-really-really-mean-it
pool 'wa01' removed
#3，查看存储池列表，发现删除成功；
root@ceph01:/# ceph osd pool ls
.mgr
bakwite_pool
wa02
wa_03
wa04
wa05
· 设置存储池nodelete删除标记
root@ceph01:/# ceph osd pool set wa02 nodelete true
set pool 4 nodelete to true
root@ceph01:/# ceph osd pool get wa02 nodelete
nodelete: true
· 再次尝试删除存储池
设置nodelete为true之后，表示，不论如何也无法删除存储池了；
root@ceph01:/# ceph osd pool delete wa02 wa02 --yes-i-really-really-mean-it
Error EPERM: pool deletion is disabled; you must unset nodelete flag for the pool first
· 再次设置nodelete标记为false
root@ceph01:/# ceph osd pool set wa02 nodelete false
set pool 4 nodelete to false
#再次删除，删除成功；
root@ceph01:/# ceph osd pool delete wa02 wa02 --yes-i-really-really-mean-it
pool 'wa02' removed
· 清空创建的存储池
root@ceph01:/# ceph osd pool ls
.mgr
bakwite_pool
wa_03
wa04
wa05
wa06
wa07
root@ceph01:/# ceph osd pool delete wa_03 wa_03 --yes-i-really-really-mean-it
pool 'wa_03' removed
root@ceph01:/# ceph osd pool delete wa04 wa04 --yes-i-really-really-mean-it
pool 'wa04' removed
root@ceph01:/# ceph osd pool delete wa05 wa05 --yes-i-really-really-mean-it
pool 'wa05' removed
root@ceph01:/# ceph osd pool delete wa06 wa06 --yes-i-really-really-mean-it
pool 'wa06' removed
root@ceph01:/# ceph osd pool delete wa07 wa07 --yes-i-really-really-mean-it
pool 'wa07' removed
root@ceph01:/# ceph osd pool delete bakwite_pool bakwite_pool --yes-i-really-really-mean-it
pool 'bakwite_pool' removed
root@ceph01:/# ceph osd pool ls
.mgr
6，存储池资源限制/配额
· 存储池资源限制概念
对象限制【max object】含义：
- 存储数据无非三种：rbd块设备的存储文件、CephFS的存储文件、对象存储的S3存储桶中的文件；
- 最后，都会被拆分为4m大小的多个RADOS对象映射存储到真实磁盘中；
- 那么我们就可以通过限制这个对象的数量，来控制存储池的最大存储数据量；进而实现资源限制；
- 设置为0，表示不限制；
字节限制【max bytes】含义：
- 就是允许存储池使用的最大真实存储空间；
- 设置为0，表示不限制；
· 查看存储池资源限制信息
root@ceph01:/# ceph osd pool get-quota wa01
quotas for pool 'wa01':
max objects: N/A # 允许存储多少个【对象】？N/A或者设置为0，表示不限制
max bytes : N/A # 允许存储池使用的最大存储空间（以字节为单位）。
· 设置存储池存储对象数量
# 设置存储池的最大存储对象数量为3w
root@ceph01:/# ceph osd pool set-quota wa01 max_objects 30000
set-quota max_objects = 30000 for pool wa01
# 查看存储池限制
root@ceph01:/# ceph osd pool get-quota wa01
quotas for pool 'wa01':
max objects: 30k objects (current num objects: 0 objects) #当前0个对象
max bytes : N/A
· 设置存储池存储数据大小
# 设置存储池存储数据最大10M,（注意，单位是byte）
root@ceph01:/# ceph osd pool set-quota wa01 max_bytes 10485760
set-quota max_bytes = 10485760 for pool wa01
# 查看存储池存储限制信息
root@ceph01:/# ceph osd pool get-quota wa01
quotas for pool 'wa01':
max objects: 30k objects (current num objects: 0 objects)
max bytes : 10 MiB (current num bytes: 0 bytes)
五、Ceph集群用户管理
1，ceph用户介绍
· 用户格式
| 用户格式 | 解释说明 |
|---|---|
| TYPEID.USERID | 用户类型.用户名称 |
用户类型【TYPEID】
| 用户类型 | 解释说明 |
|---|---|
| mon类型 | 管理mon（管理）组件的用户 |
| mgr类型 | 管理mgr（监控）组件的用户 |
| rgw类型 | 管理rgw（对象存储）组件的用户 |
| osd类型 | 管理osd（磁盘管理）组件的用户 |
| mds类型 | 管理mds（CephFS存储）组件的用户 |
| client（客户端）类型:star: | 【普通用户】客户端（存储使用者）用户 |
# 例如
- client.zhangsan
- mon.bakwite
......
· 用户权限说明
| 用户权限 | 解释说明 |
|---|---|
| r | 读权限 |
| w | 写权限 |
| x | 执行权限：就是调用ceph中的方法（类）的权限。可以执行mon的auth相关命令； |
| * | 所有权限:rwx； |
| class-read | x权限的子权限：授予用户调用类的【读取】方法的能力； |
| class-write | x权限的子权限：授予用户调用类的【写入】方法的能力； |
| profile osd | 授予用户以某一个osd的身份，链接到其他osd或者监控器的权限； |
| profile mds | 授予用户以某一个mds的身份，链接到其他mds或者监控器的权限； |
| profile bootstrap-osd | 授予用户引导osd的权限；（在部署集群的时候产生的用户）； |
| profile bootstrap-mds | 授予用户引导元数据服务器的权限；（在部署集群的时候产生的用户）； |
2，查看用户及其权限
· 客户端配置文件查看
[root@ceph01:~]# cat /etc/ceph/ceph.client.admin.keyring
# 客户端普通用户，名字叫admin；
[client.admin]
key = AQCYT4No/zQtHxAAmqdVBLxJcInCqtqg/WiQvQ==
caps mds = "allow *" # 表示对于mds，允许所有rwx权限
caps mgr = "allow *" # 表示对于mgr，允许所有rwx权限
caps mon = "allow *" # 表示对于mon，允许所有rwx权限
caps osd = "allow *" # 表示对于osd，允许所有rwx权限
· 服务端命令查看
1，查看单个用户
# 查看一个用户
root@ceph01:/# ceph auth get client.admin
[client.admin]
key = AQCYT4No/zQtHxAAmqdVBLxJcInCqtqg/WiQvQ==
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"
2，查看系统所有用户
# 查看系统集群所有用户
root@ceph01:/# ceph auth list
osd.0
key: AQDuFodoWcbGOhAAKA2YD8ZWLCk/3Jh1uTRkbA==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.1
key: AQAYF4dota8gGRAA9vfUM/c4Xmx79cSzKeND4Q==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.2
key: AQBCF4doBj3IAhAArMRe0WyDfLQTNvp1hlHOxw==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.3
key: AQCKF4doiqe+BxAA6UcDLrlu1Ba/NGu78PL0YA==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.4
key: AQC6F4do+c8JGhAAF8Wv2auSmOZc0Al2FXdAbA==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.5
key: AQDvF4dobdpZIhAA28YjO8qp9DLgtXydhDuvPg==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.6
key: AQBMGIdoKsxjDRAAcDtbG7Q4tIkAwvWYQpsRQA==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.7
key: AQBxGIdoKzjAFhAA3/UBN6Oj9yF39/cx9g9K7Q==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.8
key: AQCZGIdolvQ1HBAAPWLqDm7TWzxBlweIJl839w==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQCYT4No/zQtHxAAmqdVBLxJcInCqtqg/WiQvQ==
caps: [mds] allow *
caps: [mgr] allow *
caps: [mon] allow *
caps: [osd] allow *
client.bootstrap-mds
key: AQCbT4No+8+WMxAATgQkMiosKj7RVX3iCUVrDQ==
caps: [mon] allow profile bootstrap-mds
client.bootstrap-mgr
key: AQCbT4Nof9uWMxAApPijUKBGeSnhWBXH6+FbxA==
caps: [mon] allow profile bootstrap-mgr
client.bootstrap-osd
key: AQCbT4NofeeWMxAAAkcO718BSl3HuPo7waCvug==
caps: [mon] allow profile bootstrap-osd
client.bootstrap-rbd
key: AQCbT4NoHPqWMxAAGj0eEqTFYFoGTBZ3ZDPxdg==
caps: [mon] allow profile bootstrap-rbd
client.bootstrap-rbd-mirror
key: AQCbT4No7QaXMxAAWlUMJNeBAtaNi4tGTH9opg==
caps: [mon] allow profile bootstrap-rbd-mirror
client.bootstrap-rgw
key: AQCbT4No1hGXMxAAXD7jWQly62mkYTpVXjgFpg==
caps: [mon] allow profile bootstrap-rgw
client.ceph-exporter.ceph01
key: AQD/T4NoMUnzGBAAPaK7aw7Ck5FwitDrbMHhag==
caps: [mgr] allow r
caps: [mon] profile ceph-exporter
caps: [osd] allow r
client.ceph-exporter.ceph02
key: AQCoDodoszL1ARAAAWBY0tfhozEYv3mYb0eL8w==
caps: [mgr] allow r
caps: [mon] profile ceph-exporter
caps: [osd] allow r
client.ceph-exporter.ceph03
key: AQBPFIdoInbEGBAAsEKBbSFcRLmipkYp8hYnrA==
caps: [mgr] allow r
caps: [mon] profile ceph-exporter
caps: [osd] allow r
client.crash.ceph01
key: AQABUINoNlZPEhAAK/nu1KfxaNcAHpaCQna4ow==
caps: [mgr] profile crash
caps: [mon] profile crash
client.crash.ceph02
key: AQCsDodog1RZIBAARizwJ+3yd3+a7HbiMcDdQA==
caps: [mgr] profile crash
caps: [mon] profile crash
client.crash.ceph03
key: AQBRFIdoJLTqHRAAdS5l1oqy1lm4m4QgtFr+4w==
caps: [mgr] profile crash
caps: [mon] profile crash
mgr.ceph01.vaajsm
key: AQCYT4NorNcuLBAAC/ut3DvcSVbPEeX5wkTE8w==
caps: [mds] allow *
caps: [mon] profile mgr
caps: [osd] allow *
mgr.ceph02.mjbyul
key: AQCxDodoskTLBBAAx26L8BzrLxC+NZzsfpdQMA==
caps: [mds] allow *
caps: [mon] profile mgr
caps: [osd] allow *
mgr.ceph03.bocrhz
key: AQBrWohofuEPBBAAzqxuqKZS6TCR6+Kokpkymw==
caps: [mds] allow *
caps: [mon] profile mgr
caps: [osd] allow *
3，创建用户
# 创建一个客户端普通用户，对mon有读取权限，对osd有读写执行命令的权限（只局限于pool=wa01的存储池）；
root@ceph01:/# ceph auth add client.bakwite mon "allow r" osd "allow rwx pool=wa01"
added key for client.bakwite
# 创建完成后，查看这个用户
root@ceph01:/# ceph auth get client.bakwite
[client.bakwite]
key = AQDt+Ilo6u32FxAA5/3pQEBOpm1CzfvNF2XLEQ==
caps mon = "allow r"
caps osd = "allow rwx pool=wa01"
4，修改用户权限
root@ceph01:/# ceph auth caps client.bakwite mon 'allow rx' osd "allow r pool=wa01"
[client.bakwite]
key = AQDt+Ilo6u32FxAA5/3pQEBOpm1CzfvNF2XLEQ==
caps mon = "allow rx"
caps osd = "allow r pool=wa01"
updated caps for client.bakwite
root@ceph01:/# ceph auth get client.bakwite
[client.bakwite]
key = AQDt+Ilo6u32FxAA5/3pQEBOpm1CzfvNF2XLEQ==
caps mon = "allow rx"
caps osd = "allow r pool=wa01"
5，删除用户
root@ceph01:/# ceph auth del client.bakwite
6，导出授权文件秘钥keyring
· 创建用户
root@ceph01:/# ceph auth add client.wa mon "allow r" osd "allow rwx pool=wa01"
added key for client.wa
root@ceph01:/# ceph auth get client.wa
[client.wa]
key = AQA1+opoUfdbGRAAvHr3mr612gzBRax1h39aiA==
caps mon = "allow r"
caps osd = "allow rwx pool=wa01"
· 导出秘钥环keyring
root@ceph01:/# ceph auth export client.wa -o ceph.client.wa.keyring
root@ceph01:/# cat ceph.client.wa.keyring
[client.wa]
key = AQA1+opoUfdbGRAAvHr3mr612gzBRax1h39aiA==
caps mon = "allow r"
caps osd = "allow rwx pool=wa01"
· 将秘钥环发送给客户端
root@ceph01:/# scp ceph.client.wa.keyring 10.0.0.144:/etc/ceph/
[root@ceph01:~]# scp /etc/ceph/ceph.conf 10.0.0.144:/etc/ceph/
· 使用用户执行命令
[root@ceph04:~]# ceph -s --id wa
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 5h)
mgr: ceph02.mjbyul(active, since 47h), standbys: ceph01.vaajsm, ceph03.bocrhz
osd: 9 osds: 9 up (since 47h), 9 in (since 2d)
data:
pools: 2 pools, 33 pgs
objects: 27 objects, 12 MiB
usage: 878 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 33 active+clean
[root@ceph04:~]# rbd -p wa01 ls -l --id wa
NAME SIZE PARENT FMT PROT LOCK
snap02_clone_rbd 10 GiB 2 excl
7，秘钥环keyring备份恢复用户
· 创建用户
root@ceph01:/# ceph auth add client.bakwite mon "allow r" osd "allow rwx pool=wa01"
added key for client.bakwite
root@ceph01:/# ceph auth get client.bakwite
[client.bakwite]
key = AQB/94popHJFChAAfmnAYMv+SqDYGE5+UzPEew==
caps mon = "allow r"
caps osd = "allow rwx pool=wa01"
· 导出用户为秘钥环keyring
root@ceph01:/# ceph auth export client.bakwite -o bakwite.keyring
root@ceph01:/# ll
......
-rw-r--r-- 1 root root 125 Jul 31 04:57 bakwite.keyring
· 模拟删除用户
root@ceph01:/# ceph auth del client.bakwite
· 文件恢复用户
root@ceph01:/# ceph auth import -i bakwite.keyring
# 再次查看用户，发现已经恢复；
root@ceph01:/# ceph auth get client.bakwite
[client.bakwite]
key = AQB/94popHJFChAAfmnAYMv+SqDYGE5+UzPEew==
caps mon = "allow r"
caps osd = "allow rwx pool=wa01"
六、RBD块存储
使用场景：云厂商；
1，创建存储池
root@ceph01:/# ceph osd pool create wa01 32 32 --autoscale_mode off --size 3
pool 'wa01' created
root@ceph01:/# ceph osd pool ls detail
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 9.09
pool 10 'wa01' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 1021 flags hashpspool stripe_width 0 read_balance_score 2.25
# 查看集群状态
root@ceph01:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
# 发现是WARN，表示存储池未被定义类型；
health: HEALTH_WARN
#存储池未被定义类型；
1 pool(s) do not have an application enabled
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 40m)
mgr: ceph02.mjbyul(active, since 40m), standbys: ceph01.vaajsm, ceph03.bocrhz
osd: 9 osds: 9 up (since 40m), 9 in (since 23h)
data:
pools: 2 pools, 33 pgs
objects: 2 objects, 449 KiB
usage: 867 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 33 active+clean
2，初始化存储池为RBD存储类型
# 初始化存储池为rbd类型
root@ceph01:/# rbd pool init wa01
- 或者
root@ceph01:/# ceph osd pool application enable wa01 rbd
# 查看存储池的详细信息
root@ceph01:/# ceph osd pool ls detail
......
pool 10 'wa01' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 1024 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.25
removed_snaps_queue [2~1]
初始化完毕后，查看存储池pool详细信息：
- application字段出现了，后面写着：rbd，表示这是一个rbd块设备存储池；
定义存储池的存储类型后：
- 再次查看状态，发现【HEALTH_OK】
root@ceph01:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 66m)
mgr: ceph02.mjbyul(active, since 66m), standbys: ceph01.vaajsm, ceph03.bocrhz
osd: 9 osds: 9 up (since 66m), 9 in (since 24h)
data:
pools: 2 pools, 33 pgs
objects: 4 objects, 449 KiB
usage: 858 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 33 active+clean
3，创建磁盘RBD块设备
#1，创建一个块设备叫：wa01_rbd01
root@ceph01:/# rbd create -s 5G wa01/wa01_rbd01
#2，查看存储池下的块设备列表
root@ceph01:/# rbd ls -l wa01
NAME SIZE PARENT FMT PROT LOCK
wa01_rbd01 5 GiB 2
#3，查看块设备的详细信息
root@ceph01:/# rbd info wa01/wa01_rbd01
# RBD块设备的镜像名称
rbd image 'wa01_rbd01':
# 镜像总大小为5GB，被分割成1280个存储对象
size 5 GiB in 1280 objects
# 每个对象2^22字节大小（4M）
order 22 (4 MiB objects)
# 镜像快照的数量：0
snapshot_count: 0
# 镜像的唯一ID
id: fb0a8c41d50
# 在底层存储（如RADOS）中，每个对象的名字会以此字符串为前缀
block_name_prefix: rbd_data.fb0a8c41d50
# 表示该镜像使用第二代RBD格式（即RBD格式2）。第二代格式支持更多的特性（features）。
format: 2
# 该镜像启用的特性
# 【layering】支持分层（创建克隆镜像的基础能力）
# 【exclusive-lock】独占锁，表示同一时间，只有一个客户端可以写入数据；
# 【object-map】对象映射（优化操作，例如：复制、导出等；前提是需要独占锁）
# 【fast-diff】快速差异，可以快速计算两个快照之间的差异（需要对象映射）
# 【deep-flatten】深度扁平化，使得克隆的镜像不再依赖父快照（即使克隆镜像有快照）
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
# 操作特性，这里为空，表示没有额外的操作特性。
op_features:
# 标志位，这里为空
flags:
# 创建时间
create_timestamp: Tue Jul 29 06:46:29 2025
# 访问时间
access_timestamp: Tue Jul 29 06:46:29 2025
# 修改时间
modify_timestamp: Tue Jul 29 06:46:29 2025
4，RBD块设备扩/缩容
· resize扩容
root@ceph01:/# rbd ls -l wa01
NAME SIZE PARENT FMT PROT LOCK
wa01_rbd01 5 GiB 2
# 重新设置rbd块设备大小；
root@ceph01:/# rbd resize -s 10G wa01/wa01_rbd01
Resizing image: 100% complete...done.
root@ceph01:/# rbd ls -l wa01
NAME SIZE PARENT FMT PROT LOCK
wa01_rbd01 10 GiB 2
· resize缩容--allow-shrink
生产环境不允许缩容，容易数据丢失
# 模拟缩容，发现不被允许,提示需要加参数--allow-shrink
root@ceph01:/# rbd resize -s 9G wa01/wa01_rbd01 --allow-shrink
rbd: shrinking an image is only allowed with the --allow-shrink flag
# 加参数--allow-shrink缩容成功
root@ceph01:/# rbd resize -s 9G wa01/wa01_rbd01 --allow-shrink
Resizing image: 100% complete...done.
root@ceph01:/# rbd ls -l wa01
NAME SIZE PARENT FMT PROT LOCK
wa01_rbd01 9 GiB 2
5，修改RBD块设备名称
root@ceph01:/# rbd rename -p wa01 wa01_rbd01 wa01_rbd-01
root@ceph01:/# rbd ls -l wa01
NAME SIZE PARENT FMT PROT LOCK
wa01_rbd-01 9 GiB 2
6，删除块设备
root@ceph01:/# rbd rm -p wa01 wa01_rbd-01
Removing image: 100% complete...done.
root@ceph01:/# rbd ls -l wa01
7，块设备应用:star:
· 准备环境
| 环境 | 作用 |
|---|---|
| Ceph集群创建RBD块设备 | 生成块设备 |
| 一台虚拟机服务器 | 挂载块设备 |
1，集群创建块设备
# 创建块设备
root@ceph01:/# rbd create -s 5G wa01/wa01_rbd-01
# 查看块设备详细信息
root@ceph01:/# rbd info wa01/wa01_rbd-01
rbd image 'wa01_rbd-01':
size 5 GiB in 1280 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: fb6161ce85e
block_name_prefix: rbd_data.fb6161ce85e
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Tue Jul 29 08:41:44 2025
access_timestamp: Tue Jul 29 08:41:44 2025
modify_timestamp: Tue Jul 29 08:41:44 2025
# 查看存储池块设备列表
root@ceph01:/# rbd ls -l wa01
NAME SIZE PARENT FMT PROT LOCK
wa01_rbd-01 5 GiB 2
2，客户端安装ceph-common
[root@ceph04:~]# apt -y install ceph-common
· 拷贝集群客户端认证文件
ceph.conf 文件中记录了集群的地址信息哦；
ceph.client.admin.keyring 文件是访问集群的秘钥哦；
[root@ceph01:~]# scp /etc/ceph/{ceph.client.admin.keyring,ceph.conf} 10.0.0.144:/etc/ceph/
[root@ceph04:~]# ls -l /etc/ceph/
-rw------- 1 root root 151 Jul 29 16:47 ceph.client.admin.keyring
-rw-r--r-- 1 root root 259 Jul 29 16:47 ceph.conf
· 客户端添加rbd块设备
1，添加映射ceph块设备
[root@ceph04:~]# rbd map -p wa01 wa01_rbd-01
/dev/rbd0
2，查看块设备
| 【rbd -p 存储池名称 ls -l】命令字段 | 解释说明 |
|---|---|
| NAME | 块设备名称 |
| SIZE | 块设备的磁盘大小 |
| PARENT | 父镜像是谁？只有克隆出来的块设备，才有这个字段；后面会讲 |
| FMT | 【2】表示ceph所使用的rbd块设备格式，第二代格式 |
| PROT | 【yes】表示：写保护；不允许删除； |
| LOCK | 是否被使用？显示excl表示正在使用（挂载中）； |
# 查看存储池块设备列表
[root@ceph04:~]# rbd -p wa01 ls
wa01_rbd-01
#详细查看本地块设备
[root@ceph04:~]# rbd -p wa01 ls -l
NAME SIZE PARENT FMT PROT LOCK
snap02_clone_rbd 10 GiB 2
wa01_rbd-01 10 GiB 2
# 查看块设备状态
[root@ceph04:~]# rbd status wa01/snap02_clone_rbd
Watchers:
watcher=10.0.0.144:0/2660493618 client.64614 cookie=18446462598732840961
# 查看本地块设备映射
[root@ceph04:~]# rbd showmapped
id pool namespace image snap device
0 wa01 wa01_rbd-01 - /dev/rbd0
# 查看/dev/下的块设备
[root@ceph04:~]# ll /dev/rbd*
brw-rw---- 1 root disk 252, 0 Jul 29 16:49 /dev/rbd0
# 查看系统磁盘列表
[root@ceph04:~]# lsblk
......
rbd0 252:0 0 5G 0 disk
· 客户端格式化磁盘
# 将rbd0格式化为
[root@ceph04:~]# mkfs.ext4 /dev/rbd0
mke2fs 1.46.5 (30-Dec-2021)
Discarding device blocks: done
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 4aaebf60-e1e2-492f-94a7-962e6cef2c11
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
· 客户端挂载RBD磁盘
[root@ceph04:~]# mount /dev/rbd0 /mnt
[root@ceph04:~]# df -h
......
/dev/rbd0 4.9G 24K 4.6G 1% /mnt
· 客户端写入数据测试
[root@ceph04:~]# cp /etc/services /mnt/`date +%F_%T`.txt
[root@ceph04:~]# cp /etc/services /mnt/`date +%F_%T`.txt
......
# 查看/mnt目录
[root@ceph04:~]# ll /mnt/
total 168
drwxr-xr-x 3 root root 4096 Jul 29 17:06 ./
drwxr-xr-x 20 root root 4096 Jul 25 20:13 ../
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:01.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:02.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:03.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:04.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:05.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:06.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:07.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:08.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:09.txt
drwx------ 2 root root 16384 Jul 29 16:53 lost+found/
· 客户端扩容案例
1，将客户端块设备扩容
root@ceph01:/# rbd ls -l wa01
NAME SIZE PARENT FMT PROT LOCK
wa01_rbd-01 5 GiB 2 excl
root@ceph01:/# rbd resize -s 10G -p wa01 wa01_rbd-01
- 或者
root@ceph01:/# rbd resize -s 10G wa01/wa01_rbd-01
Resizing image: 100% complete...done.
# 查看发现扩容完成
root@ceph01:/# rbd ls -l wa01
NAME SIZE PARENT FMT PROT LOCK
wa01_rbd-01 10 GiB 2 excl
2，查看客户端df挂载
虽然服务端更新扩容了磁盘RBD块设备，但是客户端发现没有变化；
[root@ceph04:~]# df -h
...
/dev/rbd0 4.9G 168K 4.6G 1% /mnt
3，客户端热更新磁盘
# 客户端热更新块设备磁盘
[root@ceph04:~]# resize2fs /dev/rbd0
resize2fs 1.46.5 (30-Dec-2021)
Filesystem at /dev/rbd0 is mounted on /mnt; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 2
The filesystem on /dev/rbd0 is now 2621440 (4k) blocks long.
# 再次查看df磁盘挂载，发现扩容成功
[root@ceph04:~]# df -h
......
/dev/rbd0 9.8G 168K 9.3G 1% /mnt
8，块设备的快照管理
· 准备客户端环境
[root@ceph04:~]# df -h
Filesystem Size Used Avail Use% Mounted on
......
/dev/rbd0 9.8G 168K 9.3G 1% /mnt
[root@ceph04:~]# ll /mnt/
......
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:01.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:02.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:03.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:04.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:05.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:06.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:07.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:08.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:09.txt
drwx------ 2 root root 16384 Jul 29 16:53 lost+found/
· 查看本地RBD块设备映射
| 【rbd showmapped】字段含义 | 解释说明 |
|---|---|
| id | 映射的id号码 |
| pool | 归属存储池名称 |
| namespace | RBD镜像所在的名称空间 |
| image | RBD镜像的名称 |
| snap | 映射的镜像使用的“快照”，如果显示【-】表示没有映射快照； |
| device | 映射到本地块设备的路径信息 |
[root@ceph04:~]# rbd showmapped
id pool namespace image snap device
0 wa01 wa01_rbd-01 - /dev/rbd0
· 创建RBD磁盘快照
[root@ceph04:~]# rbd snap create -p wa01 --image wa01_rbd-01 --snap wa01_rbd01_snap01
- 或者
[root@ceph04:~]# rbd snap create wa01/wa01_rbd-01@wa01_rbd01_snap01
Creating snap: 100% complete...done.
· 查看本地快照
| 【rbd snap ls 存储池/块设备镜像】字段含义 | 解释说明 |
|---|---|
| SNAPID | 快照的id号码 |
| NAME | 快照的名称 |
| SIZE | 快照的大小 |
| PROTECTED | 是否设置了【写保护】？【yes】表示：写保护； |
| TIMESTAMP | 创建这个快照的具体时间； |
[root@ceph04:~]# rbd snap ls -p wa01 wa01_rbd-01
- 或者
[root@ceph04:~]# rbd snap ls wa01/wa01_rbd-01
SNAPID NAME SIZE PROTECTED TIMESTAMP
3 wa01_rbd01_snap01 10 GiB Tue Jul 29 17:50:53 2025
· 模拟磁盘故障删除数据
删除/mnt下数据
[root@ceph04:~]# rm -rf /mnt/*
[root@ceph04:~]# ls -l /mnt/
· 客户端恢复快照
1，客户端取消挂载
[root@ceph04:~]# umount /mnt
2，客户端取消rbd映射
[root@ceph04:~]# rbd unmap /dev/rbd0
[root@ceph04:~]# rbd showmapped
[root@ceph04:~]#
3，客户端查看存储池块设备
[root@ceph04:~]# rbd ls -p wa01
wa01_rbd-01
[root@ceph04:~]# rbd -p wa01 ls -l
NAME SIZE PARENT FMT PROT LOCK
wa01_rbd-01 10 GiB 2
wa01_rbd-01@wa01_rbd01_snap01 10 GiB 2
4，客户端根据快照回滚块设备
[root@ceph04:~]# rbd snap rollback -p wa01 --image wa01_rbd-01 --snap wa01_rbd01_snap01
- 或者
[root@ceph04:~]# rbd snap rollback wa01/wa01_rbd-01@wa01_rbd01_snap01
Rolling back to snapshot: 100% complete...done.
5，客户端重新映射
[root@ceph04:~]# rbd map -p wa01 wa01_rbd-01
- 或者
[root@ceph04:~]# rbd map wa01/wa01_rbd-01
/dev/rbd0
[root@ceph04:~]# rbd showmapped
id pool namespace image snap device
0 wa01 wa01_rbd-01 - /dev/rbd0
6，客户端重新挂载
[root@ceph04:~]# mount /dev/rbd0 /mnt
[root@ceph04:~]# df -h
Filesystem Size Used Avail Use% Mounted on
......
/dev/rbd0 9.8G 168K 9.3G 1% /mnt
7，客户端验证数据
# 再次查看挂载的目录，发现数据已经恢复回来；
[root@ceph04:~]# ll /mnt/
total 168
drwxr-xr-x 3 root root 4096 Jul 29 17:06 ./
drwxr-xr-x 20 root root 4096 Jul 25 20:13 ../
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:01.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:02.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:03.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:04.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:05.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:06.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:07.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:08.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:09.txt
drwx------ 2 root root 16384 Jul 29 16:53 lost+found/
· 克隆快照为rbd块设备
快照克隆的块设备，用它来实现数据恢复，速度比rollback快很多，生产环境常用；
1，克隆快照为rbd块设备
提示，克隆快照时，快照必须是受保护的状态，否则无法克隆，会报错；
快照保护：protect
- 就是将快照保护起来，不允许被删除；
# 创建快照
[root@ceph04:~]# rbd snap create -p wa01 --image wa01_rbd-01 --snap snap02
- 或者
[root@ceph04:~]# rbd snap create wa01/wa01_rbd-01@snap02
# 设置快照保护
[root@ceph04:~]# rbd snap protect -p wa01 --image wa01_rbd-01 --snap snap02
[root@ceph04:~]# rbd info wa01/wa01_rbd-01@snap02
rbd image 'wa01_rbd-01':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 1
id: fb6161ce85e
block_name_prefix: rbd_data.fb6161ce85e
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Tue Jul 29 16:41:44 2025
access_timestamp: Tue Jul 29 16:41:44 2025
modify_timestamp: Tue Jul 29 16:41:44 2025
protected: True
# 克隆快照
[root@ceph04:~]# rbd clone wa01/wa01_rbd-01@snap02 wa01/snap02_clone_rbd
[root@ceph04:~]# rbd -p wa01 ls -l
NAME SIZE PARENT FMT PROT LOCK
snap02_clone_rbd 10 GiB wa01/wa01_rbd-01@snap02 2
wa01_rbd-01 10 GiB 2
wa01_rbd-01@snap02 10 GiB 2 yes
2，查看快照的克隆（子快照）列表
就是查看哪些rbd块设备是基于这个快照克隆出来的
# 查看快照是否有【子快照】
[root@ceph04:~]# rbd children wa01/wa01_rbd-01@snap02
wa01/snap02_clone_rbd
3，基于克隆恢复数据
# 模拟磁盘数据丢失，删除rbd挂载目录下文件
[root@ceph04:~]# rm -rf /mnt/*
# 卸载块设备
[root@ceph04:~]# umount /mnt
# 取消映射rbd块设备
[root@ceph04:~]# rbd unmap /dev/rbd0
# 映射快照的克隆rbd块设备
[root@ceph04:~]# rbd map wa01/snap02_clone_rbd
/dev/rbd0
# 查看本地映射列表
[root@ceph04:~]# rbd showmapped
id pool namespace image snap device
0 wa01 snap02_clone_rbd - /dev/rbd0
# 挂载快照的克隆rbd设备
[root@ceph04:~]# mount /dev/rbd0 /mnt
# 查看挂载的文件夹，发现数据已经恢复
[root@ceph04:~]# ll /mnt/
total 168
drwxr-xr-x 3 root root 4096 Jul 29 17:06 ./
drwxr-xr-x 20 root root 4096 Jul 25 20:13 ../
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:01.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:02.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:03.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:04.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:05.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:06.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:07.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:08.txt
-rw-r--r-- 1 root root 12813 Jul 29 17:06 2025-07-29_17:06:09.txt
drwx------ 2 root root 16384 Jul 29 16:53 lost+found/
· RBD快照的管理
1，创建快照
# 创建快照
[root@ceph04:~]# rbd snap create -p wa01 --image wa01_rbd-01 --snap snap03
- 或者
[root@ceph04:~]# rbd snap create wa01/wa01_rbd-01@snap03
Creating snap: 100% complete...done.
# 查看快照列表
[root@ceph04:~]# rbd snap ls -p wa01 wa01_rbd-01
- 或者
[root@ceph04:~]# rbd snap ls wa01/wa01_rbd-01
SNAPID NAME SIZE PROTECTED TIMESTAMP
3 wa01_rbd01_snap01 10 GiB Tue Jul 29 17:50:53 2025
4 snap02 10 GiB Tue Jul 29 18:33:11 2025
5 snap03 10 GiB Tue Jul 29 18:39:11 2025
# 查看块设备镜像列表（包含快照）
[root@ceph04:~]# rbd -p wa01 ls -l
NAME SIZE PARENT FMT PROT LOCK
wa01_rbd-01@wa01_rbd01_snap01 10 GiB
snap02_clone_rbd 10 GiB wa01/wa01_rbd-01@snap02 2 excl
wa01_rbd-01 10 GiB 2
wa01_rbd-01@snap02 10 GiB 2 yes
wa01_rbd-01@snap03 10 GiB 2
2，删除快照
# 删除快照wa01_rbd01_snap01
[root@ceph04:~]# rbd snap rm -p wa01 --image wa01_rbd-01 --snap wa01_rbd01_snap01
- 或者
[root@ceph04:~]# rbd snap rm wa01/wa01_rbd-01@wa01_rbd01_snap01
Removing snap: 100% complete...done.
# 查看快照列表，发现删除成功
[root@ceph04:~]# rbd snap ls wa01/wa01_rbd-01
SNAPID NAME SIZE PROTECTED TIMESTAMP
4 snap02 10 GiB yes Tue Jul 29 18:33:11 2025
5 snap03 10 GiB Wed Jul 30 08:14:23 2025
# 删除快照snap02，会报错，无法删除；因为被保护了
[root@ceph04:~]# rbd snap rm -p wa01 --image wa01_rbd-01 --snap snap02
- 或者
[root@ceph04:~]# rbd snap rm wa01/wa01_rbd-01@snap02
2025-07-29T18:41:07.755+0800 7f62e7fff640 -1 librbd::Operations: snapshot is protected
# 清空快照
[root@ceph04:~]# rbd snap purge wa01/wa01_rbd-01
3，保护快照避免被删除
保护快照之后，就无法被删除
# 查看快照列表
[root@ceph04:~]# rbd snap ls -p wa01 wa01_rbd-01
- 或者
[root@ceph04:~]# rbd snap ls wa01/wa01_rbd-01
SNAPID NAME SIZE PROTECTED TIMESTAMP
4 snap02 10 GiB yes Tue Jul 29 18:33:11 2025
5 snap03 10 GiB yes Tue Jul 29 18:33:11 2025
# 设置快照保护
[root@ceph04:~]# rbd snap protect -p wa01 --image wa01_rbd-01 --snap snap03
- 或者
[root@ceph04:~]# rbd snap protect wa01/wa01_rbd-01@snap03
# 再次查看快照，发现PROTECTED字段设置为了yes
[root@ceph04:~]# rbd snap ls -p wa01 wa01_rbd-01
SNAPID NAME SIZE PROTECTED TIMESTAMP
4 snap02 10 GiB yes Tue Jul 29 18:33:11 2025
5 snap03 10 GiB yes Tue Jul 29 18:33:11 2025
# 尝试删除快照（发现，删除失败）
[root@ceph04:~]# rbd snap rm -p wa01 --image wa01_rbd-01 --snap snap03
- 或者
[root@ceph04:~]# rbd snap rm wa01/wa01_rbd-01@snap03
2025-07-29T18:41:07.755+0800 7f62e7fff640 -1 librbd::Operations: snapshot is protected
Removing snap: 0% complete...failed.
rbd: snapshot 'snap03' is protected from removal.
# 查看快照的详细信息
[root@ceph04:~]# rbd info -p wa01 --image wa01_rbd-01 --snap snap03
- 或者
[root@ceph04:~]# rbd info wa01/wa01_rbd-01@snap03
rbd image 'wa01_rbd-01':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 1 #快照数量1
id: fb6161ce85e
block_name_prefix: rbd_data.fb6161ce85e
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Tue Jul 29 16:41:44 2025
access_timestamp: Tue Jul 29 16:41:44 2025
modify_timestamp: Tue Jul 29 16:41:44 2025
protected: True #快照是否被保护？？
4，去除无克隆的快照保护
注意：已经有了克隆的快照，无法去除快照保护
# 没有克隆的快照，设置取消快照保护；
[root@ceph04:~]# rbd snap unprotect -p wa01 --image wa01_rbd-01 --snap snap03
- 或者
[root@ceph04:~]# rbd snap unprotect wa01/wa01_rbd-01@snap03
# 再次查看快照详细信息
[root@ceph04:~]# rbd info -p wa01 --image wa01_rbd-01 --snap snap03
- 或者
[root@ceph04:~]# rbd info wa01/wa01_rbd-01@snap03
rbd image 'wa01_rbd-01':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 1
id: fb6161ce85e
block_name_prefix: rbd_data.fb6161ce85e
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Tue Jul 29 16:41:44 2025
access_timestamp: Tue Jul 29 16:41:44 2025
modify_timestamp: Tue Jul 29 16:41:44 2025
protected: False # 快照保护已经删除
# 查看快照列表
[root@ceph04:~]# rbd snap ls -p wa01 wa01_rbd-01
SNAPID NAME SIZE PROTECTED TIMESTAMP
4 snap02 10 GiB yes Tue Jul 29 18:33:11 2025
5 snap03 10 GiB Tue Jul 29 18:33:11 2025
# 再次删除快照，删除成功
[root@ceph04:~]# rbd snap rm wa01/wa01_rbd-01@snap03
Removing snap: 100% complete...done.
5，去除有克隆的快照保护
# 有克隆的快照，去除快照保护，会报错
[root@ceph04:~]# rbd snap unprotect -p wa01 --image wa01_rbd-01 --snap snap02
2025-07-30T12:03:08.899+0800 7f0bad827640 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [fc51d4a1389c] in pool 'wa01'
2025-07-30T12:03:08.907+0800 7f0bae028640 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy
2025-07-30T12:03:08.907+0800 7f0bae028640 -1 librbd::SnapshotUnprotectRequest: 0x5562b5ecd2a0 should_complete_error: ret_val=-16
rbd: unprotecting snap failed: (16) Device or resource busy
2025-07-30T12:03:08.919+0800 7f0bad827640 -1 librbd::SnapshotUnprotectRequest: 0x5562b5ecd2a0 should_complete_error: ret_val=-16
# 查看快照的子克隆快照（发现有克隆，所以无法去除快照保护）
[root@ceph04:~]# rbd children wa01/wa01_rbd-01
wa01/snap02_clone_rbd
# flatten取消克隆快照与快照之间的关联，独立出来
[root@ceph04:~]# rbd flatten wa01/snap02_clone_rbd
Image flatten: 100% complete...done.
# 此时再次查看父快照的子克隆快照，发现快照已经没有子快着了；
[root@ceph04:~]# rbd children wa01/wa01_rbd-01
# 再次查看快照列表，发现，快照保护已经被取消
[root@ceph04:~]# rbd snap ls -p wa01 wa01_rbd-01
SNAPID NAME SIZE PROTECTED TIMESTAMP
4 snap02 10 GiB Tue Jul 29 18:33:11 2025
# 此时删除快照，就没有任何关联了
[root@ceph04:~]# rbd snap rm wa01/wa01_rbd-01@snap02
Removing snap: 100% complete...done.
# 查看块设备（证明已经独立出来了）
[root@ceph04:~]# rbd ls l -p wa01
snap02_clone_rbd
wa01_rbd-01
· 限制快照数量
1，服务端限制快照数量
# 服务端限制快照数量为5个
root@ceph01:/# rbd snap limit set wa01/wa01_rbd-01 --limit 5
# 查看块设备详细信息（最下面多出来一行的快照限制）
root@ceph01:/# rbd info wa01/wa01_rbd-01
rbd image 'wa01_rbd-01':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: fb6161ce85e
block_name_prefix: rbd_data.fb6161ce85e
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Tue Jul 29 08:41:44 2025
access_timestamp: Wed Jul 30 04:08:03 2025
modify_timestamp: Tue Jul 29 08:41:44 2025
snapshot_limit: 5 # 快照限制是5个
2，客户端创建快照测试
[root@ceph04:~]# rbd snap create wa01/wa01_rbd-01@snap01 # 第1个快照创建成功
Creating snap: 100% complete...done.
[root@ceph04:~]# rbd snap create wa01/wa01_rbd-01@snap02 # 第2个快照创建成功
Creating snap: 100% complete...done.
[root@ceph04:~]# rbd snap create wa01/wa01_rbd-01@snap03 # 第3个快照创建成功
Creating snap: 100% complete...done.
[root@ceph04:~]# rbd snap create wa01/wa01_rbd-01@snap04 # 第4个快照创建成功
Creating snap: 100% complete...done.
[root@ceph04:~]# rbd snap create wa01/wa01_rbd-01@snap05 # 第5个快照创建成功
Creating snap: 100% complete...done.
[root@ceph04:~]# rbd snap create wa01/wa01_rbd-01@snap06 # 第6个快照创建失败
Creating snap: 10% complete...failed.
rbd: failed to create snapshot: (122) Disk quota exceeded
# 查看快照列表
[root@ceph04:~]# rbd snap ls wa01/wa01_rbd-01
SNAPID NAME SIZE PROTECTED TIMESTAMP
6 snap01 10 GiB Wed Jul 30 14:50:18 2025
7 snap02 10 GiB Wed Jul 30 14:50:20 2025
8 snap03 10 GiB Wed Jul 30 14:50:25 2025
9 snap04 10 GiB Wed Jul 30 14:50:28 2025
10 snap05 10 GiB Wed Jul 30 14:50:31 2025
# 清空快照
[root@ceph04:~]# rbd snap purge wa01/wa01_rbd-01
Removing all snapshots: 100% complete...done.
[root@ceph04:~]# rbd snap ls wa01/wa01_rbd-01
3，服务端清除快照数量限制
# 服务端限制快照数量为5个
root@ceph01:/# rbd snap limit clear wa01/wa01_rbd-01
9，存储池块设备回收站
在生产环境中：
- 直接删除块设备是比较危险的操作；
- 一般来说，都是先移动到回收站，时间久了，在进一步删除；
· 查看存储池回收站列表
[root@ceph04:~]# rbd trash ls -p wa01
[root@ceph04:~]#
[root@ceph04:~]# rbd ls -l -p wa01
NAME SIZE PARENT FMT PROT LOCK
snap02_clone_rbd 10 GiB 2
wa01_rbd-01 10 GiB 2
· 将磁盘移动至回收站
# 将块设备，移动到存储池回收站中；
[root@ceph04:~]# rbd trash move wa01/wa01_rbd-01
# 再次查看存储池回收站
[root@ceph04:~]# rbd trash ls -l -p wa01
ID NAME SOURCE DELETED_AT STATUS PARENT
fb6161ce85e wa01_rbd-01 USER Wed Jul 30 14:16:19 2025 expired at Wed Jul 30 14:16:19 2025
# 查看块设备列表
[root@ceph04:~]# rbd ls -l -p wa01
NAME SIZE PARENT FMT PROT LOCK
snap02_clone_rbd 10 GiB 2
· 从回收站恢复块设备
[root@ceph04:~]# rbd trash restore -p wa01 --image wa01_rbd-01 --image-id fb6161ce85e
# 查看回收站，什么都没有了
[root@ceph04:~]# rbd trash ls -l -p wa01
# 查看块设备列表，恢复了；
[root@ceph04:~]# rbd ls -l -p wa01
NAME SIZE PARENT FMT PROT LOCK
snap02_clone_rbd 10 GiB 2 excl
wa01_rbd-01 10 GiB 2
# 验证数据是否正常（就是查看详细信息）
[root@ceph04:~]# rbd info wa01/wa01_rbd-01
rbd image 'wa01_rbd-01':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: fb6161ce85e
block_name_prefix: rbd_data.fb6161ce85e
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Tue Jul 29 16:41:44 2025
access_timestamp: Wed Jul 30 12:08:03 2025
modify_timestamp: Tue Jul 29 16:41:44 2025
· 从回收站中删除rbd镜像
# 查看rbd镜像列表
[root@ceph04:~]# rbd ls -l -p wa01
NAME SIZE PARENT FMT PROT LOCK
snap02_clone_rbd 10 GiB 2 excl
wa01_rbd-01 10 GiB 2
wa01_rbd-01@snap01 10 GiB 2
wa01_rbd-01@snap02 10 GiB 2
wa01_rbd-01@snap03 10 GiB 2
wa01_rbd-01@snap04 10 GiB 2
wa01_rbd-01@snap05 10 GiB 2
# 先将块设备移动到回收站
[root@ceph04:~]# rbd trash move wa01/wa01_rbd-01
# 再次查看rbd镜像列表，发现，镜像的快照，伴随着镜像一起移动到了回收站；
[root@ceph04:~]# rbd ls -l -p wa01
NAME SIZE PARENT FMT PROT LOCK
snap02_clone_rbd 10 GiB 2 excl
# 查看回收站
[root@ceph04:~]# rbd trash ls -l -p wa01
ID NAME SOURCE DELETED_AT STATUS PARENT
fb6161ce85e wa01_rbd-01 USER Wed Jul 30 16:18:22 2025 expired at Wed Jul 30 16:18:22 2025
# 彻底删除回收站中的rbd镜像
[root@ceph04:~]# rbd trash remove wa01/fb6161ce85e
Removing image: 100% complete...done.
[root@ceph04:~]# rbd trash ls -l wa01
七、MDS组件CephFS文件系统存储
1，创建两个存储池
两个存储池分别对应：
- 元数据存储池（metadata）
- 数据存储池（data）
# 创建两个存储池
root@ceph01:/# ceph osd pool create cephfs_inode
pool 'cephfs_inode' created
root@ceph01:/# ceph osd pool create cephfs_data
pool 'cephfs_data' created
# 查看两个存储池
root@ceph01:/# ceph osd pool ls
...
cephfs_inode
cephfs_data
# 查看系统状态，会显示：两个存储池没有分类
root@ceph01:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_WARN
2 pool(s) do not have an application enabled
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 4m)
mgr: ceph02.mjbyul(active, since 2d), standbys: ceph01.vaajsm, ceph03.bocrhz
osd: 9 osds: 9 up (since 2d), 9 in (since 2d)
data:
pools: 4 pools, 97 pgs
objects: 27 objects, 12 MiB
usage: 889 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 97 active+clean
2，创建文件系统
# 创建一个文件系统，名称为：xinjizhwiafs,里面包含两个存储池；
# 表示：文件系统bakwitefs，他的元数据pool是cephfs_inode,他的数据pool是ceph_data
root@ceph01:/# ceph fs new bakwitefs cephfs_inode cephfs_data
Pool 'cephfs_data' (id '12') has pg autoscale mode 'on' but is not marked as bulk.
Consider setting the flag by running
# ceph osd pool set cephfs_data bulk true
new fs with metadata pool 11 and data pool 12
# 查看文件系统列表
root@ceph01:/# ceph fs ls
name: bakwitefs, metadata pool: cephfs_inode, data pools: [cephfs_data ]
# 查看存储池详细信息,发现存储池已经分类为：cephfs了；
root@ceph01:/# ceph osd pool ls detail
......
pool 11 'cephfs_inode' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1093 lfor 0/0/1079 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.81
pool 12 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1093 lfor 0/0/1087 flags hashpspool stripe_width 0 application cephfs read_balance_score 2.53
再次查看集群状态
root@ceph01:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_ERR
1 filesystem is offline
1 filesystem is online with fewer MDS than max_mds
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 45m)
mgr: ceph02.mjbyul(active, since 2d), standbys: ceph01.vaajsm, ceph03.bocrhz
mds: 0/0 daemons up
osd: 9 osds: 9 up (since 2d), 9 in (since 3d)
data:
volumes: 1/1 healthy
pools: 4 pools, 97 pgs
objects: 27 objects, 12 MiB
usage: 889 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 97 active+clean
3，启动MDS文件系统
root@ceph01:/# ceph orch apply mds bakwitefs
Scheduled mds.bakwitefs update...
再次查看集群状态:
- 发现：默认一主、一备的mds服务；
root@ceph01:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 51m)
mgr: ceph02.mjbyul(active, since 2d), standbys: ceph01.vaajsm, ceph03.bocrhz
mds: 1/1 daemons up, 1 standby # 显示MDS，一主，一备；
osd: 9 osds: 9 up (since 2d), 9 in (since 3d)
data:
volumes: 1/1 healthy
pools: 4 pools, 97 pgs
objects: 49 objects, 12 MiB
usage: 889 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 97 active+clean
io:
client: 1.2 KiB/s wr, 0 op/s rd, 3 op/s wr
4，查看文件系统详细信息
可以看到：默认一主一备；
- 主是ceph02
- 备是ceph03
root@ceph01:/# ceph fs status bakwitefs
bakwitefs - 0 clients
============
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active bakwitefs.ceph02.ledklv Reqs: 0 /s 10 13 12 0
POOL TYPE USED AVAIL
cephfs_inode metadata 96.0k 1731G
cephfs_data data 0 1731G
STANDBY MDS # 备用节点
bakwitefs.ceph03.zmzvgo
MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
5，调整MDS高可用节点数量
# 设置集群mds节点数量；
root@ceph01:/# ceph orch apply mds bakwitefs --placement="ceph01 ceph02 ceph03"
Scheduled mds.bakwitefs update...
# 再次查看文件系统状态，发现已经一主两备了；
root@ceph01:/# ceph fs status bakwitefs
bakwitefs - 0 clients
============
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active bakwitefs.ceph02.ledklv Reqs: 0 /s 10 13 12 0
POOL TYPE USED AVAIL
cephfs_inode metadata 96.0k 1731G
cephfs_data data 0 1731G
STANDBY MDS
bakwitefs.ceph01.lkokoh
bakwitefs.ceph03.nrwoyb
MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
# 查看集群状态
root@ceph01:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 67m)
mgr: ceph02.mjbyul(active, since 2d), standbys: ceph01.vaajsm, ceph03.bocrhz
mds: 1/1 daemons up, 2 standby
osd: 9 osds: 9 up (since 2d), 9 in (since 3d)
data:
volumes: 1/1 healthy
pools: 4 pools, 97 pgs
objects: 49 objects, 12 MiB
usage: 889 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 97 active+clean
6，客户端使用CephFS
注意：CephFS需要linux内核加载ceph内核模块
· 客户端内核加载ceph模块
1，临时加载内核ceph模块
[root@ceph04:~]# modinfo ceph
filename: /lib/modules/5.15.0-144-generic/kernel/fs/ceph/ceph.ko
license: GPL
description: Ceph filesystem for Linux
author: Patience Warnick <patience@newdream.net>
author: Yehuda Sadeh <yehuda@hq.newdream.net>
author: Sage Weil <sage@newdream.net>
alias: fs-ceph
srcversion: 2FCF4F127F7C5521F58D0F2
depends: libceph,fscache,netfs
retpoline: Y
intree: Y
name: ceph
vermagic: 5.15.0-144-generic SMP mod_unload modversions
sig_id: PKCS#7
signer: Build time autogenerated kernel key
sig_key: 0B:7C:16:96:38:5D:1E:B8:19:CA:95:FE:B0:03:61:6B:47:B1:53:1E
sig_hashalgo: sha512
signature: 59:8F:4C:85:1F:73:61:CD:AB:D7:DD:C3:FC:F4:CA:95:D2:9D:A7:84:
73:4D:28:4F:80:37:3C:3C:84:09:AB:20:0E:29:CD:3F:42:56:18:C3:
C1:8B:56:BD:8D:3A:ED:92:96:F4:8F:EF:3A:27:AA:FD:79:2C:C6:AC:
72:0C:FE:F9:CD:69:33:60:AD:2A:B1:8A:81:F5:C1:63:D3:A4:2C:B3:
A7:0E:45:62:1C:DC:B3:CB:2A:BE:C0:87:6F:5B:DB:50:86:8F:24:10:
72:CB:D1:76:97:35:7C:D9:6B:D6:29:38:70:76:2D:BC:61:D1:FC:96:
24:6D:0D:15:33:21:33:76:39:51:AC:43:92:29:F5:82:BD:3B:55:FA:
C1:DD:5C:7B:65:BF:5C:AC:FB:40:71:30:78:FA:26:BE:8A:5E:DB:1F:
02:8B:AE:F9:05:10:2F:1A:7A:16:32:97:F4:B3:8A:1F:48:CC:17:6C:
53:2E:4F:C1:9D:2D:0B:E6:F2:CB:81:73:76:1B:AF:85:0E:9C:D3:35:
C5:2D:08:6A:30:D0:7A:C0:E8:80:2F:9D:C9:F3:61:A9:DA:D6:2E:38:
EE:CC:89:14:32:52:14:3C:40:54:D4:3C:CB:20:7C:AF:FB:2D:F3:DF:
A2:6E:D7:52:B9:E4:6F:A0:3F:D1:C3:4F:FA:1A:3E:B4:2E:1B:B0:E1:
B9:0C:51:33:75:8E:5B:11:48:5C:27:DC:E2:B2:C6:1D:86:FF:44:74:
9B:8B:DD:08:C0:DF:A4:6A:89:5B:CF:57:5F:09:03:03:9B:E8:CC:23:
1A:F8:42:8E:56:CB:E8:0B:6B:41:DD:7B:57:A7:8A:D5:54:EF:75:7E:
C2:CE:F8:6D:5B:ED:36:1D:46:F0:46:2A:A2:08:7D:86:B5:FB:E7:08:
AC:9D:D3:05:F5:77:8C:C1:B3:C1:7D:64:22:B9:B0:DD:CB:25:67:4F:
58:E2:7C:B6:EF:50:54:11:79:B3:65:E0:84:E3:CF:9F:73:7E:35:00:
B3:27:6A:3B:B3:83:F9:B3:0B:B1:B8:49:52:31:C1:22:CC:BD:A1:4A:
D7:A5:CF:EE:CA:5F:58:61:96:DA:BA:5E:59:8C:69:41:94:12:50:C6:
0C:A9:56:C0:FF:64:E9:68:21:74:37:F8:98:B1:FF:D9:E1:37:88:53:
D7:60:44:98:C3:C0:2E:F8:75:37:AB:1F:4D:D4:48:41:46:F9:54:34:
AF:B0:DF:F0:D5:14:F7:3A:FD:9D:D9:AA:E9:53:02:9C:D6:26:41:8F:
1C:FC:52:8D:DC:ED:6A:44:19:91:8C:AF:BA:91:44:96:70:C8:05:FE:
93:07:5F:1A:90:09:D3:FE:72:A7:3D:7C
parm: disable_send_metrics:Enable sending perf metrics to ceph cluster (default: on)
2，永久加载内核ceph模块
#1，将iptables模块到内核的命令写入开机自动执行文件（让其永久加载到内核）
[root@m01 ~ 19:50:14]# vim /etc/rc.d/rc.local
......
modprobe ceph
#2，给开机自动执行文件执行权限
[root@m01 ~ 19:57:02]# chmod +x /etc/rc.d/rc.local
· 创建客户端用户
1，创建用户
root@ceph01:/# ceph auth add client.wa_cephfs mon "allow r" mds "allow rw" osd "allow rwx"
added key for client.wa_cephfs
root@ceph01:/# ceph auth get client.wa_cephfs
[client.wa_cephfs]
key = AQDyK4tolBiDCRAA1OSGRkgnPwzmZ7E5oUJsPQ==
caps mds = "allow rw"
caps mon = "allow r"
caps osd = "allow rwx"
2，导出用户为秘钥keyring（不用做）
这一步可以不做，因为客户端cephfs挂载，只需要用到秘钥即可
root@ceph01:/# ceph auth export client.wa_cephfs -o ceph.client.wa_cephfs.keyring
root@ceph01:/# cat ceph.client.wa_cephfs.keyring
[client.wa_cephfs]
key = AQDyK4tolBiDCRAA1OSGRkgnPwzmZ7E5oUJsPQ==
caps mds = "allow rw"
caps mon = "allow r"
caps osd = "allow rwx"
3，单独导出用户秘钥key
root@ceph01:/# ceph auth print-key client.wa_cephfs > wa_cephfs.key
root@ceph01:/# cat wa_cephfs.key
AQDyK4tolBiDCRAA1OSGRkgnPwzmZ7E5oUJsPQ==
4，将用户秘钥key文件发送给客户端（可以不做）
root@ceph01:/# scp wa_cephfs.key 10.0.0.144:/etc/ceph/
· 客户端挂载cephfs
1，客户端创建挂载目录
【语法】：【mount -t ceph 集群地址:/文件系统名称 挂载目录 -o name=用户名，secret=用户秘钥】
【注意】：[用户名] 前面的"client"必须去掉，否则挂载失败；
[root@ceph04:~]# mount -t ceph 10.0.0.141:6789,10.0.0.142:6789,10.0.0.143:6789:/ /data -o name=wa_cephfs,secret=AQDyK4tolBiDCRAA1OSGRkgnPwzmZ7E5oUJsPQ==,fs=bakwitefs
# df查看挂载信息；
[root@ceph04:~]# df -h
......
/dev/rbd0 9.8G 200K 9.3G 1% /mnt
10.0.0.141:6789,10.0.0.142:6789,10.0.0.143:6789:/ 1.7T 0 1.7T 0% /data
2，服务端查看文件系统状态
root@ceph01:/# ceph fs status bakwitefs
bakwitefs - 1 clients # 这里显示，一个客户端正在使用；
============
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active bakwitefs.ceph02.ledklv Reqs: 0 /s 10 13 12 1
POOL TYPE USED AVAIL
cephfs_inode metadata 96.0k 1731G
cephfs_data data 0 1731G
STANDBY MDS
bakwitefs.ceph01.lkokoh
bakwitefs.ceph03.nrwoyb
MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
3，客户端写入数据测试
[root@ceph04:~]# cp /etc/services /data/`date +%F_%T`.txt
[root@ceph04:~]# cp /etc/services /data/`date +%F_%T`.txt
[root@ceph04:~]# cp /etc/services /data/`date +%F_%T`.txt
[root@ceph04:~]# cp /etc/services /data/`date +%F_%T`.txt
[root@ceph04:~]# cp /etc/services /data/`date +%F_%T`.txt
[root@ceph04:~]# cp /etc/services /data/`date +%F_%T`.txt
[root@ceph04:~]# cp /etc/services /data/`date +%F_%T`.txt
[root@ceph04:~]# ls -l /data
total 173
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:54.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:55.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:56.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:57.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:58.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:59.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:00.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:01.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:02.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:03.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:04.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:05.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:06.txt
4，开机自动挂载
| 编辑字段 | 含义 |
|---|---|
| 10.0.0.201:6789,10.0.0.202:6789,10.0.0.203:6789:/ | 设备的远程文件系统【:/】表示挂载整个文件系统（根目录） |
| /data | 挂载点 |
| ceph | 表示文件系统类型 |
| name=wa_cephfs | 使用文件系统的用户名 |
| secret=AQDyK4tolBiDCRAA1OSGRkgnPwzmZ7E5oUJsPQ== | 用户验证秘钥 |
| noatime | 表示不更新文件的访问时间，可以提高性能 |
| _netdev | 表示是网络文件系统 |
| 0 | 表示不备份 |
| 2 | 表示检查文件系统的启动顺序；根是1，其他是2，0表示不检查； |
【注意】：[用户名] 前面的"client"必须去掉，否则挂载失败；
[root@ceph04:~]# vim /etc/fstab
......
10.0.0.141:6789,10.0.0.142:6789,10.0.0.143:6789:/ /fuse/data ceph name=wa_cephfs,secret=AQDAkJ1o8BlFCBAAW3i2Hfzk+rHJGsn3kxYJMg==,fs=bakwitefs,noatime,_netdev 0 2
# 重启查看
[root@ceph04:~]# df -h
......
10.0.0.141:6789,10.0.0.142:6789,10.0.0.143:6789:/ 1.7T 0 1.7T 0% /data
......
· 内核不支持ceph模块的挂载方式
说明：
- 如果客户端使用了低版本linux内核，不支持ceph模块
- 那么需要借助【ceph-fuse】进行挂载cephfs
- 但是，ceph-fuse挂载的性能，略低于内核的ceph模块；
1，客户端安装ceph-fuse
[root@ceph04:~]# apt -y install ceph-fuse
2，创建挂载目录
[root@ceph04:~]# mkdir -p /ceph-fuse/data
3，挂载CephFS
【注意】：用户keyring文件，必须存在/cet/ceph下面，否则无法挂载；
[root@ceph04:~]# ceph-fuse -n client.wa_cephfs -m 10.0.0.141:6789,10.0.0.142:6789,10.0.0.143:6789 --client_fs bakwitefs /ceph-fuse/data
2025-08-01T08:11:58.191+0800 7ff3a16433c0 -1 init, newargv = 0x55784aed16f0 newargc=13
2025-08-01T08:11:58.191+0800 7ff3a16433c0 -1 init, args.argv = 0x55784aed1840 args.argc=4
ceph-fuse[30683]: starting ceph client
ceph-fuse[30683]: starting fuse
4， 查看挂载点数据
[root@ceph04:~]# ll /ceph-fuse/data/
total 174
drwxr-xr-x 2 root root 166569 Jul 31 17:25 ./
drwxr-xr-x 3 root root 4096 Aug 1 08:08 ../
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:54.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:55.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:56.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:57.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:58.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:59.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:00.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:01.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:02.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:03.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:04.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:05.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:06.txt
5，开机自动挂载
[root@ceph04:~]# vim /etc/rc.d/rc.local
......
ceph-fuse -n client.wa_cephfs -m 10.0.0.141:6789,10.0.0.142:6789,10.0.0.143:6789 --client_fs bakwitefs /ceph-fuse/data
重启测试
[root@ceph04:~]# df -h
......
10.0.0.141:6789,10.0.0.142:6789,10.0.0.143:6789:/ 1.7T 0 1.7T 0% /ceph-fuse/data
......
八、RGW(rados gateway)对象存储
1，对象存储概念
| 名词解释 | 解释说明 |
|---|---|
| 对象（object） | 每个对象就是一个用户存储的文件信息；例如：【123.jpg】 |
| 桶（bucket） | 对象的逻辑地址：每个对象都会分属于一个桶； |
2，部署RGW组件
· 部署rgw
# 部署三个节点的rgw实现高可用。rgw名称：bakwitergw
root@ceph01:/# ceph orch apply rgw bakwitergw --placement="ceph01 ceph02 ceph03"
Scheduled rgw.bakwitergw update...
# 查看集群状态
root@ceph01:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 113m)
mgr: ceph02.mjbyul(active, since 3d), standbys: ceph01.vaajsm, ceph03.bocrhz
mds: 1/1 daemons up, 2 standby
osd: 9 osds: 9 up (since 3d), 9 in (since 4d)
rgw: 3 daemons active (3 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 8 pools, 225 pgs
objects: 256 objects, 12 MiB
usage: 943 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 225 active+clean
# 查看组件部署节点
root@ceph01:/# ceph orch ps | grep rgw
rgw.bakwitergw.ceph01.bysxxy ceph01 *:80 running (6m) 6m ago 6m 51.5M ....
rgw.bakwitergw.ceph02.whrooq ceph02 *:80 running (6m) 6m ago 6m 50.1M ....
rgw.bakwitergw.ceph03.dmgbnd ceph03 *:80 running (6m) 6m ago 6m 52.8M ....
· 查看存储池列表
| rgw存储池 | 作用 |
|---|---|
| .rgw.root | 【根配置存储池】：rgw集群的全局配置信息的存储池；多节点同步元数据时使用； |
| default.rgw.control | 【控制存储池】：rgw集群的 “控制消息” 的存储池；集群同步刷新内存、确保多节点一致性的消息广播； |
| default.rgw.meta | 【元数据存储池】：rgw客户端的用户凭证、桶信息、使用量等等的存储池； |
| default.rgw.log | 【日志存储池】：rgw的审计日志、统计日志的存储池；用于计费、审计和故障排查。 |
root@ceph01:/# ceph osd pool ls
......
.rgw.root
default.rgw.log
default.rgw.control
default.rgw.meta
· 查看rgw服务信息
root@ceph01:/# ceph orch ls rgw rgw.bakwitergw --export
service_type: rgw #服务类型
service_id: bakwitergw #服务id
service_name: rgw.bakwitergw #服务名称
placement:
hosts: #集群主机列表
- ceph01
- ceph02
- ceph03
· 查看页面
http://10.0.0.141/
http://10.0.0.142/
http://10.0.0.143/
· 创建用户
| 命令 | 解释说明 |
|---|---|
| radosgw-admin | Ceph集群中，用来管理Rados Gateway（RGW）的命令行工具 |
| user create | 创建一个新的RGW用户 |
| --uid "bakwite" | 创建用户的唯一标识符（User ID）。在这里，用户ID是"bakwite"。 |
| --display-name "心机之蛙" | 指定了用户的显示名称（Display Name），用于在管理界面或显示时使用。 |
root@ceph01:/# radosgw-admin user create --uid "bakwite" --display-name "心机之蛙"
{
"user_id": "bakwite",
"display_name": "心机之蛙",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "bakwite",
"access_key": "Q90UY7T21J8N8OAKBDGI",
"secret_key": "Hun00sjM1EacTFwDiH1kzneQD4CfDJ6sBXM84ArG",
"active": true,
"create_date": "2025-08-01T09:13:08.984812Z"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": [],
"account_id": "",
"path": "/",
"create_date": "2025-08-01T09:13:08.978467Z",
"tags": [],
"group_ids": []
}
# 查看rgw账号列表
root@ceph01:/# radosgw-admin user list
[
"dashboard",
"bakwite"
]
# 查看用户详细信息
root@ceph01:/# radosgw-admin user info --uid=xjzw
3，亚马逊(Amazon)S3操作RGW
· 客户端安装s3cmd工具
[root@ceph04:~]# apt -y install s3cmd
· 生成s3配置文件
[root@ceph04:~]# s3cmd --configure
......
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: Q90UY7T21J8N8OAKBDGI #输入用户的Access Key
Secret Key: Hun00sjM1EacTFwDiH1kzneQD4CfDJ6sBXM84ArG #输入用户的Secret Key
Default Region [US]: #时区设置，我们直接回车
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: 10.0.0.141 # 域名设置：没有就写rgw节点的ip
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: 10.0.0.141/%(bucket) #设置访问路径格式；
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: # 加密密码，我们不设置，直接回车
Path to GPG program [/usr/bin/gpg]: #加密软件命令GPT的路径，默认回车即可；
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: no #是否配置https访问？？ 我们输入【no】不配置
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: # 是否设置代理，我们不设置，直接回车
New settings:
Access Key: Q90UY7T21J8N8OAKBDGI
Secret Key: Hun00sjM1EacTFwDiH1kzneQD4CfDJ6sBXM84ArG
Default Region: US
S3 Endpoint: 10.0.0.141
DNS-style bucket+hostname:port template for accessing a bucket: 10.0.0.141/%(bucket)
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] Y #确认以上设置；
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y # 保存以上设置
Configuration saved to '/root/.s3cfg'
· 查看s3配置文件
[root@ceph04:~]# cat .s3cfg
[default]
access_key = Q90UY7T21J8N8OAKBDGI
access_token =
add_encoding_exts =
add_headers =
bucket_location = US
ca_certs_file =
cache_file =
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = cloudfront.amazonaws.com
connection_max_age = 5
connection_pooling = True
content_disposition =
content_type =
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = 10.0.0.141
host_bucket = 10.0.0.141/%(bucket)
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key =
limit = -1
limitrate = 0
list_md5 = False
log_target_prefix =
long_listing = False
max_delete = -1
mime_type =
multipart_chunk_size_mb = 15
multipart_copy_chunk_size_mb = 1024
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
public_url_use_https = False
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
restore_priority = Standard
secret_key = Hun00sjM1EacTFwDiH1kzneQD4CfDJ6sBXM84ArG
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
signurl_use_https = False
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
ssl_client_cert_file =
ssl_client_key_file =
stats = False
stop_on_error = False
storage_class =
throttle_max = 100
upload_id =
urlencoding_mode = normal
use_http_expect = False
use_https = False
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
website_error =
website_index = index.html
· s3类型的桶管理
1，创建桶
# 桶名称为：bakwite-bucket
[root@ceph04:~]# s3cmd mb s3://bakwite-bucket
Bucket 's3://bakwite-bucket/' created
# 创建另一个桶
[root@ceph04:~]# s3cmd mb s3://bakwite-bucket02
Bucket 's3://bakwite-bucket02/' created
2，查看桶列表
# s3客户端查看方式
[root@ceph04:~]# s3cmd ls
2025-08-01 09:44 s3://bakwite-bucket
2025-08-01 09:47 s3://bakwite-bucket02
# ceph集群radosgw-admin命令查看
root@ceph01:/# radosgw-admin buckets list
[
"bakwite-bucket"
]
3，删除桶
[root@ceph04:~]# s3cmd rb s3://bakwite-bucket02
Bucket 's3://bakwite-bucket02/' removed
[root@ceph01:~]# s3cmd ls
2025-08-01 09:44 s3://bakwite-bucket
· s3cmd操作桶中数据
1，准备数据
# 上传一个视频文件
[root@ceph04:~]# rz -E
[root@ceph04:~]# ls -l
-rw-r--r-- 1 root root 93629148 Jul 17 11:13 第01节【课程介绍】新手开公司要学习哪些知识【优选课程网www.youxuan68.com】.mp4
# 创造10个文件
[root@ceph04:~]# touch {1..10},txt
2，上传数据到桶
· 上传文件
# 上传视频文件到桶中
[root@ceph04:~]# s3cmd put 第01节【课程介绍】新手开公司要学习哪些知识【优选课程网www.youxuan68.com】.mp4 s3://bakwite-bucket
# 上传1.txt和2.txt到【桶中的目录】（随便写）
[root@ceph04:~]# s3cmd put 1.txt 2.txt s3://bakwite-bucket/txt/
· 同步目录（上传目录下所有）
# 准备一个目录，下面弄点文件（这里我们使用docker解压包）
[root@ceph04:~]# ls -l docker/
total 196944
-rwxr-xr-x 1 root root 39451064 Apr 5 2023 containerd
-rwxr-xr-x 1 root root 7548928 Apr 5 2023 containerd-shim
-rwxr-xr-x 1 root root 9760768 Apr 5 2023 containerd-shim-runc-v2
-rwxr-xr-x 1 root root 21090304 Apr 5 2023 ctr
-rwxr-xr-x 1 root root 47955672 Apr 5 2023 docker
-rwxr-xr-x 1 root root 58246432 Apr 5 2023 dockerd
-rwxr-xr-x 1 root root 765808 Apr 5 2023 docker-init
-rwxr-xr-x 1 root root 2628514 Apr 5 2023 docker-proxy
-rwxr-xr-x 1 root root 14214624 Apr 5 2023 runc
# sync同步目录下的所有。到桶中；
[root@ceph04:~]# s3cmd sync ~/docker/ s3://bakwite-bucket/txt/
# 查看桶中数据
[root@ceph01:~]# s3cmd ls s3://bakwite-bucket/txt/
2025-08-01 10:53 0 s3://bakwite-bucket/txt/2.txt
2025-08-01 11:02 39451064 s3://bakwite-bucket/txt/containerd
2025-08-01 11:02 7548928 s3://bakwite-bucket/txt/containerd-shim
2025-08-01 11:02 9760768 s3://bakwite-bucket/txt/containerd-shim-runc-v2
2025-08-01 11:02 21090304 s3://bakwite-bucket/txt/ctr
2025-08-01 11:02 47955672 s3://bakwite-bucket/txt/docker
2025-08-01 11:02 765808 s3://bakwite-bucket/txt/docker-init
2025-08-01 11:02 2628514 s3://bakwite-bucket/txt/docker-proxy
2025-08-01 11:02 58246432 s3://bakwite-bucket/txt/dockerd
2025-08-01 11:02 14214624 s3://bakwite-bucket/txt/runc
3，查看桶数据
# 查看桶中数据（发现一个目录和一个视频文件）
[root@ceph04:~]# s3cmd ls s3://bakwite-bucket/
DIR s3://bakwite-bucket/txt/
2025-08-01 10:54 93629148 s3://bakwite-bucket/第01节【课程介绍】新手开公司要学习哪些知识【优选课程网www.youxuan68.com】.mp4
# 查看桶中目录下的数据（发现，1.txt和2.txt）
[root@ceph04:~]# s3cmd ls s3://bakwite-bucket/txt/
2025-08-01 10:53 0 s3://bakwite-bucket/txt/1.txt
2025-08-01 10:53 0 s3://bakwite-bucket/txt/2.txt
4，下载数据
[root@ceph04:~]# s3cmd get s3://bakwite-bucket/第01节【课程介绍】新手开公司要学习哪些知识【优选课程网www.youxuan68.com】.mp4 /data/
download: 's3://bakwite-bucket/bakwite8.avi' -> '/data/bakwite8.avi' [1 of 1]
0 of 0 0% in 0s 0.00 B/s done
[root@ceph04:~]# ls -l /data/
total 0
-rw-r--r-- 1 root root 0 Aug 1 09:58 bakwite8.avi
5，删除桶中数据
# 删除同种数据；
[root@ceph04:~]# s3cmd rm s3://bakwite-bucket/txt/1.txt
delete: 's3://bakwite-bucket/txt/1.txt'
# 查看桶中数据，看是否删除；
[root@ceph04:~]# s3cmd ls s3://bakwite-bucket/txt/
2025-08-01 10:53 0 s3://bakwite-bucket/txt/2.txt
· 在线视频案例
1，查看桶配置信息
[root@ceph04:~]# s3cmd info s3://bakwite-bucket
s3://bakwite-bucket/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: none #这里显示没有任何规则
CORS: none
ACL: 心机之蛙: FULL_CONTROL
2，页面访问桶的数据
发现权限不足，无法访问
http://10.0.0.141/bakwite-bucket/第01节【课程介绍】新手开公司要学习哪些知识【优选课程网www.youxuan68.com】.mp4
3，编辑桶的规则文件
设置为允许页面查看数据信息
[root@ceph04:~]# vim policy.json
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["*"]},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::bakwite-bucket/*"
]
}]
}
4，修改/设置桶规则
[root@ceph04:~]# s3cmd setpolicy policy.json s3://bakwite-bucket
s3://bakwite-bucket/: Policy updated
# 设置后查看桶配置信息中的规则policy字段
[root@ceph01:~]# s3cmd info s3://bakwite-bucket
s3://bakwite-bucket/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["*"]},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::bakwite-bucket/*"
]
}]
}
CORS: none
ACL: 心机之蛙: FULL_CONTROL
5，再次页面查看桶数据
http://10.0.0.141/bakwite-bucket/第01节【课程介绍】新手开公司要学习哪些知识【优选课程网www.youxuan68.com】.mp4
6，删除桶规则
[root@ceph04:~]# s3cmd delpolicy s3://bakwite-bucket
s3://bakwite-bucket/: Policy deleted
[root@ceph01:~]# s3cmd info s3://bakwite-bucket
s3://bakwite-bucket/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: none
CORS: none
ACL: 心机之蛙: FULL_CONTROL
删除后，视频资源页面无法查看了
· s3类型桶的策略权限
1，参考链接
#https://docs.ceph.com/en/latest/radosgw/bucketpolicy/
2，授权策略文件格式解读
| 策略文件字段 | 下级字段 | 解释说明 |
|---|---|---|
| Version | 【2012-10-17】：表示当前 亚马逊AWS IAM 策略的标准版本； 支持复杂条件（如 IP 限制、多重条件判断等）。 |
|
| Statement | 【权限规则的核心逻辑块】：一个数组，可包含多条独立规则； | |
| Effect | 【权限的类型】：Allow表示允许，Deny表示拒绝； | |
| Principal | 【规则针对谁？】：用户？角色？账户？等等; | |
| Action | 允许/拒绝的 【具体操作动作】（API 操作） | |
| Resource | 【规则生效的 范围】：s3存储桶？对象？ |
[root@ceph01:~]# vim policy.json
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["*"]}, #【规则针对谁？】：用户？角色？账户？等等;
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::bakwite-bucket/*" # 【规则生效的 范围】：s3存储桶？对象？
]
}]
}
Action常见操作动作
# "Action": "s3:GetObject"
##################################################
s3:* # 所有操作
s3:Get* # 所有Get开头的操作
s3:PutObject # 上传文件到桶中
s3:PutObjectAcl # 设置对象的Acl，设置文件的访问权限
s3:GetObject # 下载/获取对象
s3:GetObjectAcl # 读取对象的Acl，查看文件的权限
s3:DeleteObject # 删除 对象
s3:CreateBucket # 创建 桶
s3:DeleteBucket # 删除 桶
s3:ListAllMyBuckets # 列出账户的所有桶
s3:ListBucket # 查看 桶 中的对象（文件）列表
s3:GetBucketPolicy # 读取 桶 的规则策略
s3:PutBucketPolicy # 设置 通 的规则策略
s3:PutBucketAcl # 设置 桶 的Acl，管理桶的权限
Principal常见针对的主体
# "Principal": {"AWS": ["*"]}
##################################################
"Principal": "*" # 表示公开访问，谁来都行；
"Principal": {"AWS": ["*"]} # 【同上】表示公开访问，谁来都行；
"Principal": {"AWS": "arn:aws:iam:::user/xnjizhiwa"} # 只针对bakwite这个ceph中的rgw用户的规则
4，python操作RGW
· 安装pip3工具
[root@ceph04:~]# apt -y install python3-pip
· 下载boto模块
[root@ceph04:~]# pip install boto==2.49.0 -i https://pypi.tuna.tsinghua.edu.cn/simple
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting boto==2.49.0
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl (1.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 5.9 MB/s eta 0:00:00
Installing collected packages: boto
Successfully installed boto-2.49.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
· 编写python代码
[root@ceph04:~]# vim rgw-bakwite.py
import boto
import boto.s3.connection
access_key = 'Q90UY7T21J8N8OAKBDGI'
secret_key = 'Hun00sjM1EacTFwDiH1kzneQD4CfDJ6sBXM84ArG'
# 链接RGW服务
conn = boto.connect_s3(
aws_access_key_id = access_key,
aws_secret_access_key = secret_key,
host = '10.0.0.141',
is_secure=False,
calling_format = boto.s3.connection.OrdinaryCallingFormat(),
)
# 创建【桶】
bucket = conn.create_bucket('bakwite-rgw-python')
# 查看【桶】列表
for bucket in conn.get_all_buckets():
print("{name}\t{created}".format(
name = bucket.name,
created = bucket.creation_date,
))
# 查看bucket内容
for key in bucket.list():
print("{name}\t{size}\t{modified}".format(
name = key.name,
size = key.size,
modified = key.last_modified,
))
# 创建一个对象文件
key = bucket.new_key("bakwite_python_test.txt")
# 对象文件中的内容
key.set_contents_from_string("hei,xinjihiwa~")
# 生成对象文件下载的链接
hello_key = bucket.get_key('bakwite_python_test.txt')
hello_url = hello_key.generate_url(0, query_auth=False, force_http=True)
print(hello_url)
· 执行python脚本
[root@ceph04:~]# python3 rgw-bakwite.py
bakwite-bucket 2025-08-01T09:44:21.413Z
bakwite-rgw-python 2025-08-04T06:29:16.857Z
http://10.0.0.141/bakwite-rgw-python/bakwite_python_test.txt
· s3cmd下载测试
[root@ceph01:~]# s3cmd get s3://bakwite-rgw-python/bakwite_python_test.txt
download: 's3://bakwite-rgw-python/bakwite_python_test.txt' -> './bakwite_python_test.txt' [1 of 1]
14 of 14 100% in 0s 254.93 B/s done
· 访问链接测试
1，curl访问链接
发现权限不足
[root@ceph04:~]# curl http://10.0.0.141/bakwite-rgw-python/bakwite_python_test.txt
<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message></Message><BucketName>bakwite-rgw-python</BucketName><RequestId>tx0000048078d20af67bd5f-00689054d9-35926-default</RequestId><HostId>35926-default-default</HostId></Error>
页面访问
2，设置桶规则权限
#1，编辑规则文件
[root@ceph01:~]# vim policy02.json
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["*"]},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::bakwite-rgw-python/*"
]
}]
}
#2，配置桶规则
[root@ceph01:~]# s3cmd setpolicy policy02.json s3://bakwite-rgw-python
s3://bakwite-rgw-python/: Policy updated
#3，查看桶规则权限
[root@ceph01:~]# s3cmd info s3://bakwite-rgw-python/bakwite_python_test.txt
s3://bakwite-rgw-python/bakwite_python_test.txt (object):
File size: 14
Last mod: Mon, 04 Aug 2025 06:41:10 GMT
MIME type: application/octet-stream
Storage: STANDARD
MD5 sum: 828ab6beed95d748601f44111b2771a2
SSE: none
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["*"]},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::bakwite-rgw-python/*"
]
}]
}
CORS: none
ACL: 心机之蛙: FULL_CONTROL
2，再次curl访问
这次访问成功
[root@ceph04:~]# curl http://10.0.0.141/bakwite-rgw-python/bakwite_python_test.txt
hei,xinjihiwa~
5，swift操作RGW
· 创建一个新RGW账号
root@ceph01:/# radosgw-admin user create --uid "xjzw" --display-name "蛙"
{
"user_id": "xjzw",
"display_name": "蛙",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "xjzw",
"access_key": "OO6QCYSC2KIWFD2ZHPJM",
"secret_key": "cd2hyhQxl2uIg5wOnrjdKbjCZrAG3f6mDnBb2rft",
"active": true,
"create_date": "2025-08-04T08:19:20.488158Z"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": [],
"account_id": "",
"path": "/",
"create_date": "2025-08-04T08:19:20.485302Z",
"tags": [],
"group_ids": []
}
# 查看账号列表
root@ceph01:/# radosgw-admin user list
[
"dashboard",
"bakwite",
"xjzw"
]
· 基于RGW账号创建swift子账号
| 字段说明 | 解释说明 |
|---|---|
| subuser create | 创建子账号 |
| --uid=xjzw --subuser=xjzw:swift | 指定主账号，指定子账号名称；【主账号名:子账号名】 |
| --access=full | 指定权限；full表示所有权限，包括读写； |
root@ceph01:/# radosgw-admin subuser create --uid=xjzw --subuser=xjzw:swift --access=full
......
"keys": [
{
"user": "xjzw",
"access_key": "OO6QCYSC2KIWFD2ZHPJM",
"secret_key": "cd2hyhQxl2uIg5wOnrjdKbjCZrAG3f6mDnBb2rft",
"active": true,
"create_date": "2025-08-04T08:19:20.488158Z"
}
],
"swift_keys": [
{
"user": "xjzw:swift",
"secret_key": "SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16",
"active": true,
"create_date": "2025-08-04T08:30:57.832313Z"
......
# 看看子账号信息
root@ceph01:/# radosgw-admin user info --uid=xjzw
......
"keys": [
{
"user": "xjzw",
"access_key": "OO6QCYSC2KIWFD2ZHPJM",
"secret_key": "cd2hyhQxl2uIg5wOnrjdKbjCZrAG3f6mDnBb2rft",
"active": true,
"create_date": "2025-08-04T08:19:20.488158Z"
}
],
"swift_keys": [
{
"user": "xjzw:swift",
"secret_key": "SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16", #这个秘钥
"active": true,
"create_date": "2025-08-04T08:30:57.832313Z"
}
],
.......
· 安装python组件
[root@ceph04:~]# apt -y install python-setuptools python3-pip
· 安装python的swift模块
[root@ceph04:~]# pip install python-swiftclient==4.6.0
Collecting python-swiftclient==4.6.0
Downloading python_swiftclient-4.6.0-py3-none-any.whl (88 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 88.8/88.8 KB 715.3 kB/s eta 0:00:00
Requirement already satisfied: requests>=2.4.0 in /usr/lib/python3/dist-packages (from python-swiftclient==4.6.0) (2.25.1)
Installing collected packages: python-swiftclient
Successfully installed python-swiftclient-4.6.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
· swift创建桶
[root@ceph04:~]# swift -A http://10.0.0.141/auth -U xjzw:swift -K SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16 post swift-bucket
· 查看swift桶列表
swift不叫存储桶，而是叫“容器”，但是二者是一个意思
# 查看桶列表
[root@ceph04:~]# swift -A http://10.0.0.141/auth -U xjzw:swift -K SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16 list
swift-bucket
# 查看桶列表长格式
[root@ceph04:~]# swift -A http://10.0.0.141/auth -U xjzw:swift -K SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16 list -l
0 0 2025-08-04 08:47:57 default-placement swift-bucket
0 0
使用s3cmd再查看一下，发现swift和s3两个不重复,二者并不兼容；
[root@ceph04:~]# s3cmd ls
2025-08-01 09:44 s3://bakwite-bucket
2025-08-04 06:29 s3://bakwite-rgw-python
· 上传文件到swift桶
[root@ceph04:~]# swift -A http://10.0.0.141/auth -U xjzw:swift -K SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16 upload swift-bucket /data/*
data/2025-07-31_17:24:54.txt
data/2025-07-31_17:25:03.txt
data/2025-07-31_17:24:55.txt
data/2025-07-31_17:25:00.txt
data/2025-07-31_17:24:57.txt
data/2025-07-31_17:25:01.txt
data/2025-07-31_17:25:02.txt
data/2025-07-31_17:24:56.txt
data/2025-07-31_17:24:59.txt
data/2025-07-31_17:24:58.txt
data/2025-07-31_17:25:06.txt
data/2025-07-31_17:25:04.txt
data/2025-07-31_17:25:05.txt
· 查看存储桶中文件
[root@ceph04:~]# swift -A http://10.0.0.141/auth -U xjzw:swift -K SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16 list swift-bucket
data/2025-07-31_17:24:54.txt
data/2025-07-31_17:24:55.txt
data/2025-07-31_17:24:56.txt
data/2025-07-31_17:24:57.txt
data/2025-07-31_17:24:58.txt
data/2025-07-31_17:24:59.txt
data/2025-07-31_17:25:00.txt
data/2025-07-31_17:25:01.txt
data/2025-07-31_17:25:02.txt
data/2025-07-31_17:25:03.txt
data/2025-07-31_17:25:04.txt
data/2025-07-31_17:25:05.txt
data/2025-07-31_17:25:06.txt
# 长格式显示
[root@ceph04:~]# swift -A http://10.0.0.141/auth -U xjzw:swift -K SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16 list swift-bucket -l
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:24:54.txt
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:24:55.txt
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:24:56.txt
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:24:57.txt
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:24:58.txt
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:24:59.txt
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:25:00.txt
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:25:01.txt
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:25:02.txt
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:25:03.txt
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:25:04.txt
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:25:05.txt
12813 2025-08-04 09:17:46 text/plain data/2025-07-31_17:25:06.txt
166569
· 下载存储桶中文件
1，下载整个目录
# 下载整个目录
[root@ceph04:~]# swift -A http://10.0.0.141/auth -U xjzw:swift -K SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16 download swift-bucket --prefix "data/"
[root@ceph04:~]# ll data/
total 216
drwxr-xr-x 2 root root 4096 Aug 4 17:27 ./
drwx------ 6 root root 4096 Aug 4 17:25 ../
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:54.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:55.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:56.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:57.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:58.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:59.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:00.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:01.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:02.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:03.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:04.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:05.txt
-rw-r--r-- 1 root root 12813 Jul 31 17:25 2025-07-31_17:25:06.txt
2，下载单个文件
[root@ceph04:~]# swift -A http://10.0.0.141/auth -U xjzw:swift -K SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16 download swift-bucket "data/2025-07-31_17:24:54.txt"
[root@ceph04:~]# ls -l data/
total 24
-rw-r--r-- 1 root root 12813 Jul 31 17:24 2025-07-31_17:24:54.txt
· 删除文件
1，删除单个文件
[root@ceph04:~]# swift -A http://10.0.0.141/auth -U xjzw:swift -K SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16 delete swift-bucket "data/2025-07-31_17:24:54.txt"
data/2025-07-31_17:24:54.txt
[root@ceph04:~]# swift -A http://10.0.0.141/auth -U xjzw:swift -K SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16 list swift-bucket
data/2025-07-31_17:24:55.txt
data/2025-07-31_17:24:56.txt
data/2025-07-31_17:24:57.txt
data/2025-07-31_17:24:58.txt
data/2025-07-31_17:24:59.txt
data/2025-07-31_17:25:00.txt
data/2025-07-31_17:25:01.txt
data/2025-07-31_17:25:02.txt
data/2025-07-31_17:25:03.txt
data/2025-07-31_17:25:04.txt
data/2025-07-31_17:25:05.txt
data/2025-07-31_17:25:06.txt
2，删除整个目录
[root@ceph04:~]# swift -A http://10.0.0.141/auth -U xjzw:swift -K SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16 delete swift-bucket --prefix "data/"
data/2025-07-31_17:24:58.txt
data/2025-07-31_17:24:56.txt
data/2025-07-31_17:24:57.txt
data/2025-07-31_17:25:04.txt
data/2025-07-31_17:25:02.txt
data/2025-07-31_17:25:03.txt
data/2025-07-31_17:25:01.txt
data/2025-07-31_17:24:59.txt
data/2025-07-31_17:24:55.txt
data/2025-07-31_17:25:05.txt
data/2025-07-31_17:25:00.txt
data/2025-07-31_17:25:06.txt
[root@ceph04:~]# swift -A http://10.0.0.141/auth -U xjzw:swift -K SCqB19eRG52pOdvbp54IhOsrmMXidYbNAO1vCF16 list swift-bucket
九、Ceph集群监控管理
1，查看集群组件
· 查看集群组件列表
root@ceph01:/# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 90s ago 6d count:1
ceph-exporter 3/3 6m ago 6d *
crash 3/3 6m ago 6d *
grafana ?:3000 1/1 90s ago 6d count:1
mds.bakwitefs 3/3 6m ago 21h ceph01;ceph02;ceph03
mgr 3/3 6m ago 2d ceph01;ceph02;ceph03
mon 3/3 6m ago 2d ceph01;ceph02;ceph03
node-exporter ?:9100 3/3 6m ago 6d *
osd 9 6m ago - <unmanaged>
prometheus ?:9095 1/1 90s ago 6d count:1
· 查看集群组件部署详情
# 查看一个节点所启动的组件进程
root@ceph01:/# ceph orch ps ceph01
# 查看所有节点启动的所有组件进程
root@ceph01:/# ceph orch ps
......
2，更新集群组件启动数量
# MON节点
root@ceph01:/# ceph orch apply mon --placement="ceph01 ceph02 ceph03"
# MGR节点(cephfs高可用节点)
root@ceph01:/# ceph orch apply mds --placement="ceph01 ceph02 ceph03"
# prometheus节点
root@ceph01:/# ceph orch apply prometheus --placement="ceph01 ceph02 ceph03"
# mgr节点
root@ceph01:/# ceph orch apply mgr --placement="ceph01 ceph02 ceph03"
# grafana组件
root@ceph01:/# ceph orch apply grafana --placement="ceph01 ceph02 ceph03"
# rgw（对象存储）组件
root@ceph01:/# ceph orch apply rgw bakwitergw --placement="ceph01 ceph02 ceph03"
3，查看集群组件状态信息
# 查看mon状态
root@ceph01:/# ceph mon stat
e5: 3 mons at {
ceph01=[v2:10.0.0.141:3300/0,v1:10.0.0.141:6789/0],
ceph02=[v2:10.0.0.142:3300/0,v1:10.0.0.142:6789/0],
ceph03=[v2:10.0.0.143:3300/0,v1:10.0.0.143:6789/0]
}
removed_ranks: {}
disallowed_leaders: {},
election epoch 168,
leader 0 ceph01,
quorum 0,1,2 ceph01,ceph02,ceph03
# 查看mds状态
root@ceph01:/# ceph mds stat
bakwitefs:1 {0=bakwitefs.ceph02.ledklv=up:active} 2 up:standby
# 查看mgr状态
root@ceph01:/# ceph mgr stat
{
"epoch": 35,
"available": true,
"active_name": "ceph02.mjbyul",
"num_standby": 2
}
4，重启集群节点中的组件
root@ceph01:/# ceph orch daemon restart node-exporter.ceph01
root@ceph01:/# ceph orch daemon restart prometheus.ceph01
root@ceph01:/# ceph orch daemon restart mon.ceph01
5，查看集群OSD设备列表
root@ceph01:/# ceph orch device ls
6，查看集群主机列表
root@ceph01:/# ceph orch host ls
HOST ADDR LABELS STATUS
ceph01 10.0.0.141 _admin
ceph02 10.0.0.142
ceph03 10.0.0.143
3 hosts in cluster
7，查看cephadm编排器状态
· 查看状态
# 查看集群状态
root@ceph01:/# ceph orch status
Backend: cephadm
Available: Yes
Paused: No
# 查看集群状态详细信息
root@ceph01:/# ceph orch status --detail
Backend: cephadm # 当前编排器的引擎
Available: Yes # 编排器cephadm服务是否可用
Paused: No # 编排器正常运行，会自动维护集群状态
Host Parallelism: 10 # cephadm在编排主机时的并发控制主机数量，主机并行操作数量
· 设置编排器并发处理数量
# 降低并行度（大型集群减少负载）
root@ceph01:/# ceph config set mgr mgr/cephadm/max_concurrent_hosts 5
# 增加并行度（加速操作）
root@ceph01:/# ceph config set mgr mgr/cephadm/max_concurrent_hosts 20
· 暂停与恢复编排器
root@ceph01:/# ceph orch pause # 进入维护模式
root@ceph01:/# ceph orch resume # 恢复正常操作
8，查看集群状态
root@ceph01:/# ceph -s
cluster:
id: 53b83466-693a-11f0-ae5c-3bb1aa4e0ea6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 5h)
mgr: ceph02.mjbyul(active, since 2d), standbys: ceph01.vaajsm, ceph03.bocrhz
mds: 1/1 daemons up, 2 standby
osd: 9 osds: 9 up (since 2d), 9 in (since 3d)
data:
volumes: 1/1 healthy
pools: 4 pools, 97 pgs
objects: 62 objects, 12 MiB
usage: 897 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 97 active+clean
9，集群监控组件
· prometheus监控
10.0.0.141:9095
· grafana监控展示
#注意：必须是https哦~
https://10.0.0.141:3000
· 设置登录信息
1，编辑grafana配置文件
# 找到grafana容器
[root@ceph01:~]# docker ps | grep grafana
4e9ea5bc8b9d quay.io/ceph/grafana:10.4.0 "/run.sh" 3 days ago Up 3 days ceph-53b83466-693a-11f0-ae5c-3bb1aa4e0ea6-grafana-ceph01
# 进入容器中
[root@ceph01:~]# docker exec -it 4e9ea5bc8b9d bash
# 编辑grafana配置文件
ceph01:/usr/share/grafana$ vi /etc/grafana/grafana.ini
# This file is generated by cephadm.
[users]
default_theme = light
[auth.anonymous]
enabled = true
org_name = 'Main Org.'
org_role = 'Viewer'
[server]
domain = 'bootstrap.storage.lab'
protocol = https
cert_file = /etc/grafana/certs/cert_file
cert_key = /etc/grafana/certs/cert_key
http_port = 3000
http_addr =
[snapshots]
external_enabled = false
[security]
disable_initial_admin_creation = false # 关闭禁用admin登录功能
cookie_secure = true
cookie_samesite = none
allow_embedding = true
# 重启grafana容器
[root@ceph01:~]# docker restart 4e9ea5bc8b9d
4e9ea5bc8b9d
2，登录grafana
admin
admin
Discussion
评论