kubernetes叢集中ceph叢集使用
塊裝置建立和使用
Ceph 能夠為 pod 提供裸的塊裝置卷，定義在Ceph資料備援級别的一個池中
1.storageclass 配置
- storageclass.yaml
該配置檔案為生産場景提供了3個副本，至少需要3個節點
資料會間歇性的在3個不同的k8s節點上複制，單節點故障不會造成資料的丢失或不可用
- storageclass-ec.yaml
以糾錯碼來替代鏡像達到資料持久性的功能，需要至少3個節點
Ceph 的糾錯碼比鏡像的效率更高，是以能夠提供高可用性而無需3份鏡像的代價
但會造成節點較高的編碼解碼計算，也就是會提高節點的cpu資源消耗
- storageclass-test.yaml:
測試場景，隻需要單個節點，隻有一份鏡像，有資料丢失的風險
使用 csi/rbd 目錄下的配置文間，csi是推薦的驅動，flex驅動不推薦使用
[[email protected] ~]# cd /tmp/rook/cluster/examples/kubernetes/ceph/csi/rbd/
[[email protected] rbd]# sed -i 's/failureDomain: host/failureDomain: osd/g' storageclass.yaml
[[email protected] rbd]# vim storageclass.yaml
[[email protected] rbd]# kubectl apply -f storageclass.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block created
[[email protected] rbd]# kubectl get sc -n rook-ceph
2.檢視配置結果
[[email protected] rbd]# kubectl -n rook-ceph get pod -l "app=rook-ceph-tools"
NAME READY STATUS RESTARTS AGE
rook-ceph-tools-7476c966b7-5f5kg 1/1 Running 0 46h
[[email protected] rbd]# NAME=$(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
[[email protected] rbd]# kubectl -n rook-ceph exec -it ${NAME} ceph status
[[email protected] rbd]# kubectl -n rook-ceph exec -it ${NAME} ceph osd status
[[email protected] rbd]# kubectl -n rook-ceph exec -it ${NAME} ceph osd df
3.kubernetes Dashboard 檢視結果
4.建立 Wordpress 進行測試
[[email protected] rbd]# cd /tmp/rook/cluster/examples/kubernetes
[[email protected] kubernetes]# sed -i 's|mysql:5.6|registry.cn-hangzhou.aliyuncs.com/vinc-auto/mysql:5.6|g' mysql.yaml
[[email protected] kubernetes]# sed -i 's|wordpress:4.6.1-apache|registry.cn-hangzhou.aliyuncs.com/vinc-auto/wordpress:4.6.1-apache|g' wordpress.yaml
[[email protected] kubernetes]# sed -i 's/LoadBalancer/NodePort/g' wordpress.yaml
[[email protected] kubernetes]# kubectl apply -f mysql.yaml
service/wordpress-mysql created
persistentvolumeclaim/mysql-pv-claim created
deployment.apps/wordpress-mysql created
[[email protected] kubernetes]# kubectl apply -f wordpress.yaml
service/wordpress created
persistentvolumeclaim/wp-pv-claim created
deployment.apps/wordpress created
1.檢視建立的資源
[[email protected] kubernetes]# kubectl get pvc -o wide
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
mysql-pv-claim Bound pvc-8a58a8cc-2620-47ba-8182-58e7b0ef998e 20Gi RWO rook-ceph-block 12m Filesystem
wp-pv-claim Bound pvc-50626a36-64f3-4fb8-a83f-e4dff26be1f4 20Gi RWO rook-ceph-block 12m Filesystem
[[email protected] kubernetes]# kubectl get deploy -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
wordpress 1/1 1 1 2m24s wordpress registry.cn-hangzhou.aliyuncs.com/vinc-auto/wordpress:4.6.1-apache app=wordpress,tier=frontend
wordpress-mysql 1/1 1 1 2m43s mysql registry.cn-hangzhou.aliyuncs.com/vinc-auto/mysql:5.6 app=wordpress,tier=mysql
[[email protected] kubernetes]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-xingdian 1/1 Running 1 2d9h 10.244.2.3 node-2 <none> <none>
nginx-xingdian-2 1/1 Running 0 2d8h 10.244.1.2 node-1 <none> <none>
wordpress-5587564997-lgp2w 1/1 Running 0 113s 10.244.2.26 node-2 <none> <none>
wordpress-mysql-55456f44b-kgnj6 1/1 Running 0 2m13s 10.244.2.25 node-2 <none> <none>
2.浏覽器通路 wordpress 進行部署
3.檢視Ceph叢集中的相關資料
[[email protected] ~]# kubectl -n rook-ceph get pod -l "app=rook-ceph-tools"
NAME READY STATUS RESTARTS AGE
rook-ceph-tools-7476c966b7-5f5kg 1/1 Running 0 47h
[[email protected] ~]# NAME=$(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
[[email protected] ~]# kubectl -n rook-ceph exec -it ${NAME} sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-4.2# ceph osd pool stats
pool replicapool id 1
client io 3.7 KiB/s rd, 195 KiB/s wr, 2 op/s rd, 2 op/s wr
sh-4.2# rbd ls -p replicapool
csi-vol-0377015d-4624-11eb-80eb-12916ae40d83
csi-vol-864d78df-4628-11eb-80eb-12916ae40d83
csi-vol-904c6c28-4628-11eb-80eb-12916ae40d83
sh-4.2# rbd info replicapool/'csi-vol-0377015d-4624-11eb-80eb-12916ae40d83'
rbd image 'csi-vol-0377015d-4624-11eb-80eb-12916ae40d83':
size 20 GiB in 5120 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 8a347eb76d55
block_name_prefix: rbd_data.8a347eb76d55
format: 2
features: layering
op_features:
flags:
create_timestamp: Tue Dec 22 11:20:30 2020
access_timestamp: Tue Dec 22 11:20:30 2020
modify_timestamp: Tue Dec 22 11:20:30 2020
4.删除測試環境
[[email protected] kubernetes]# cd /tmp/rook/cluster/examples/kubernetes
[[email protected] kubernetes]# kubectl delete -f wordpress.yaml
[[email protected] kubernetes]# kubectl delete -f mysql.yaml
[[email protected] kubernetes]# kubectl delete -n rook-ceph cephblockpools.ceph.rook.io replicapool
[[email protected] kubernetes]# kubectl delete storageclass rook-ceph-block