AKAI TSUKI

System development or Technical something

I'd like to use CephFS on CentOS 7.

Before this page, please check these post.
Try to install Ceph in CentOS 7 referencing "STORAGE CLUSTER QUICK START" - AKAI TSUKI
Install Ceph in CentOS 7. - AKAI TSUKI


I'd like to use CephFS.
http://docs.ceph.com/docs/master/cephfs/createfs/

I create pools.
Rgarding to pg_num, I see below url.
http://docs.ceph.com/docs/master/rados/operations/placement-groups/

[cuser@ceph01 ~]$ sudo ceph osd pool create cephfs_data 128
pool 'cephfs_data' created
[cuser@ceph01 ~]$

[cuser@ceph01 ~]$ sudo ceph osd pool create cephfs_metadata 128
Error ERANGE:  pg_num 128 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
[cuser@ceph01 ~]$

Error occurred. so I lower pg_num value for cephfs_metadata.

[cuser@ceph01 ~]$ sudo ceph osd pool create cephfs_metadata 64
pool 'cephfs_metadata' created
[cuser@ceph01 ~]$
[cuser@ceph01 ~]$ sudo ceph fs new testfs cephfs_metadata cephfs_data
new fs with metadata pool 2 and data pool 1
[cuser@ceph01 ~]$

[cuser@ceph01 ~]$ sudo ceph fs ls
name: testfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
[cuser@ceph01 ~]$

I check status.

[cuser@ceph01 ~]$ sudo ceph osd stat
3 osds: 3 up, 3 in; epoch: e22
[cuser@ceph01 ~]$ sudo ceph osd status
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| id |  host  |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| 0  | ceph01 | 1027M | 14.9G |    0   |     0   |    0   |     0   | exists,up |
| 1  | ceph02 | 1027M | 14.9G |    0   |     0   |    0   |     0   | exists,up |
| 2  | ceph03 | 1027M | 14.9G |    0   |     0   |    0   |     0   | exists,up |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
[cuser@ceph01 ~]$

[cuser@ceph01 ~]$ sudo ceph osd versions
{
    "ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)": 3
}
[cuser@ceph01 ~]$
[cuser@ceph01 ~]$ sudo ceph osd pool ls detail
pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 22 flags hashpspool stripe_width 0 application cephfs
pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 22 flags hashpspool stripe_width 0 application cephfs

[cuser@ceph01 ~]$

[cuser@ceph01 ~]$ sudo ceph mds stat
testfs-1/1/1 up  {0=ceph01=up:active}
[cuser@ceph01 ~]$

[cuser@ceph01 ~]$ sudo cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
        key = *snip*
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"
[cuser@ceph01 ~]$

Clinet Side

I setup client to use CephFS.

[root@ceph05 ~]# chmod 600 admin.secret
[root@ceph05 ~]# ls -l
total 8
-rw-------  1 root root   41 Jul 15 20:10 admin.secret
-rw-------. 1 root root 1329 Jul 14 21:50 anaconda-ks.cfg
[root@ceph05 ~]# mkdir /mnt/mycephfs
[root@ceph05 ~]# ls -l /mnt/mycephfs/
total 0
[root@ceph05 ~]#
[root@ceph05 ~]# sudo mount -t ceph ceph01:/ /mnt/mycephfs -o name=admin,secretfile=/root/admin.secret
[root@ceph05 ~]#

[root@ceph05 ~]# df -hT
Filesystem              Type      Size  Used Avail Use% Mounted on
/dev/mapper/centos-root xfs        28G  1.5G   27G   6% /
devtmpfs                devtmpfs  1.9G     0  1.9G   0% /dev
tmpfs                   tmpfs     1.9G     0  1.9G   0% /dev/shm
tmpfs                   tmpfs     1.9G  8.5M  1.9G   1% /run
tmpfs                   tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1               xfs      1014M  171M  844M  17% /boot
tmpfs                   tmpfs     380M     0  380M   0% /run/user/0
172.16.10.111:/         ceph       15G     0   15G   0% /mnt/mycephfs
[root@ceph05 ~]#

I create a file on shared disk of CephFS.

[root@ceph05 ~]# ls -l /mnt/mycephfs/
total 0
[root@ceph05 ~]# vi /mnt/mycephfs/test.txt
[root@ceph05 ~]# cat /mnt/mycephfs/test.txt
message
[root@ceph05 ~]#

After I unmount, I can not see created file.

[root@ceph05 ~]# umount /mnt/mycephfs
[root@ceph05 ~]# df -hT
Filesystem              Type      Size  Used Avail Use% Mounted on
/dev/mapper/centos-root xfs        28G  1.5G   27G   6% /
devtmpfs                devtmpfs  1.9G     0  1.9G   0% /dev
tmpfs                   tmpfs     1.9G     0  1.9G   0% /dev/shm
tmpfs                   tmpfs     1.9G  8.5M  1.9G   1% /run
tmpfs                   tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1               xfs      1014M  171M  844M  17% /boot
tmpfs                   tmpfs     380M     0  380M   0% /run/user/0
[root@ceph05 ~]#
[root@ceph05 ~]# ls -l /mnt/mycephfs/
total 0
[root@ceph05 ~]#

I mount again.

[root@ceph05 ~]# sudo mount -t ceph ceph01:/ /mnt/mycephfs -o name=admin,secretfile=/root/admin.secret
[root@ceph05 ~]#
[root@ceph05 ~]# df -hT
Filesystem              Type      Size  Used Avail Use% Mounted on
/dev/mapper/centos-root xfs        28G  1.5G   27G   6% /
devtmpfs                devtmpfs  1.9G     0  1.9G   0% /dev
tmpfs                   tmpfs     1.9G     0  1.9G   0% /dev/shm
tmpfs                   tmpfs     1.9G  8.5M  1.9G   1% /run
tmpfs                   tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1               xfs      1014M  171M  844M  17% /boot
tmpfs                   tmpfs     380M     0  380M   0% /run/user/0
172.16.10.111:/         ceph       15G     0   15G   0% /mnt/mycephfs
[root@ceph05 ~]#
[root@ceph05 ~]# ls -l /mnt/mycephfs/
total 1
-rw-r--r-- 1 root root 8 Jul 15 20:14 test.txt
[root@ceph05 ~]# cat /mnt/mycephfs/test.txt
message
[root@ceph05 ~]#

and I confirm status.

[cuser@ceph01 ~]$ sudo ceph fs status
testfs - 1 clients
======
+------+--------+--------+---------------+-------+-------+
| Rank | State  |  MDS   |    Activity   |  dns  |  inos |
+------+--------+--------+---------------+-------+-------+
|  0   | active | ceph01 | Reqs:    0 /s |   13  |   14  |
+------+--------+--------+---------------+-------+-------+
+-----------------+----------+-------+-------+
|       Pool      |   type   |  used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 26.2k | 14.1G |
|   cephfs_data   |   data   |    8  | 14.1G |
+-----------------+----------+-------+-------+
+-------------+
| Standby MDS |
+-------------+
+-------------+
MDS version: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)
[cuser@ceph01 ~]$