阿裡雲K8S服務支援CSI存儲卷

CSI插件簡介

CSI是Container Storage Interface的簡稱，為容器編排系統和存儲系統之間建立一套标準的存儲調用接口。

Kubernetes支援CSI接口标準，并在1.10版本進入beta階段。K8S除了支援CSI方式提供存儲接口外，還有In-Tree方式、Flexvolume方式提供的各種存儲接口。

In-Tree方式是嵌入在K8S源碼内部的存儲挂載實作，存在以下問題：

存儲插件需要一同随K8S釋出。
存儲插件的問題有可能會影響K8S部件正常運作。
存儲插件享有K8S部件同等的特權存在安全隐患。
存儲插件開發者必須遵循K8S社群的規則開發代碼。

FlexVolume機制通過調用一個可執行檔案方式去實作挂載，它能夠做到讓存儲提供方進行獨立開發維護。

部署可執行檔案時，需要host的root權限，依然存在安全隐患。
存儲插件在執行mount、attach操作時，往往需要到host去安裝第三方工具或加載依賴庫，這樣會使部署變得複雜。

CSI标準的出現解決了上述問題，後續K8S與存儲系統的內建标準将全面支援CSI。

阿裡雲存儲支援CSI Plugin

目前阿裡雲CSI存儲插件支援雲盤、OSS、NAS等存儲類型，并支援通過StorageClass建立動态存儲卷；

開源位址：

https://github.com/AliyunContainerService/csi-plugin

插件的部署拓撲如下：

使用示範

雲盤CSI Plugin部署

環境準備

通過阿裡雲容器服務建立K8S叢集（1.10版本及以上），或者自己通過阿裡雲ECS伺服器搭建K8S叢集。
配置Kubelet啟動參數：--enable-controller-attach-detach=true；

部署 CSI Attacher

# kubectl create -f attacher.yaml

CSI attacher使用0.2.0版本，attacher.yaml内容如下：

# This YAML file contains RBAC API objects,
# which are necessary to run external csi attacher for cinder.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-attacher

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-attacher-runner
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-attacher-role
subjects:
  - kind: ServiceAccount
    name: csi-attacher
    namespace: default
roleRef:
  kind: ClusterRole
  name: external-attacher-runner
  apiGroup: rbac.authorization.k8s.io
---

kind: Service
apiVersion: v1
metadata:
  name: csi-attacher
  labels:
    app: csi-attacher
spec:
  selector:
    app: csi-attacher
  ports:
    - name: dummy
      port: 12345

---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: csi-attacher
spec:
  serviceName: "csi-attacher"
  replicas: 1
  template:
    metadata:
      labels:
        app: csi-attacher
    spec:
      tolerations:
      - effect: NoSchedule
        operator: Exists
        key: node-role.kubernetes.io/master
      - effect: NoSchedule
        operator: Exists
        key: node.cloudprovider.kubernetes.io/uninitialized
      nodeSelector:
         node-role.kubernetes.io/master: ""
      serviceAccount: csi-attacher
      containers:
        - name: csi-attacher
          image: registry.cn-hangzhou.aliyuncs.com/plugins/csi-attacher:v0.2.0
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
          env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-diskplugin/csi.sock
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/csi-diskplugin
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-diskplugin
            type: DirectoryOrCreate

部署CSI Provsioner

# kubectl create -f provisioner.yaml

CSI provisioner使用0.2.0版本，内容如下：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-provisioner

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-provisioner-role
subjects:
  - kind: ServiceAccount
    name: csi-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: external-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

---
kind: Service
apiVersion: v1
metadata:
  name: csi-provisioner
  labels:
    app: csi-provisioner
spec:
  selector:
    app: csi-provisioner
  ports:
    - name: dummy
      port: 12345

---
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: csi-provisioner
spec:
  serviceName: "csi-provisioner"
  replicas: 1
  template:
    metadata:
      labels:
        app: csi-provisioner
    spec:
      tolerations:
      - effect: NoSchedule
        operator: Exists
        key: node-role.kubernetes.io/master
      - effect: NoSchedule
        operator: Exists
        key: node.cloudprovider.kubernetes.io/uninitialized
      nodeSelector:
         node-role.kubernetes.io/master: ""
      serviceAccount: csi-provisioner
      containers:
        - name: csi-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/plugins/csi-provisioner:v0.2.0
          args:
            - "--provisioner=csi-diskplugin"
            - "--csi-address=$(ADDRESS)"
            - "--v=5"
          env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-diskplugin/csi.sock
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/csi-diskplugin
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-diskplugin
            type: DirectoryOrCreate

部署Disk Plugin

Disk Plugin部署為DaemonSet應用，指令：

# kubectl create -f plugin.yaml

CSI driver-registrar使用0.2.0版本；

Socket位址設定為：/var/lib/kubelet/plugins/csi-diskplugin/csi.sock；

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-diskplugin

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-diskplugin
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "update"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-diskplugin
subjects:
  - kind: ServiceAccount
    name: csi-diskplugin
    namespace: default
roleRef:
  kind: ClusterRole
  name: csi-diskplugin
  apiGroup: rbac.authorization.k8s.io

---
# This YAML file contains driver-registrar & csi driver nodeplugin API objects,
# which are necessary to run csi nodeplugin for disk.

kind: DaemonSet
apiVersion: apps/v1beta2
metadata:
  name: csi-diskplugin
spec:
  selector:
    matchLabels:
      app: csi-diskplugin
  template:
    metadata:
      labels:
        app: csi-diskplugin
    spec:
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      serviceAccount: csi-diskplugin
      hostNetwork: true
      hostPID: true
      containers:
        - name: driver-registrar
          image: registry.cn-hangzhou.aliyuncs.com/plugins/driver-registrar:v0.2.0
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
          env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/csi-diskplugin/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/csi-diskplugin
        - name: csi-diskplugin
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
          image: registry.cn-hangzhou.aliyuncs.com/plugins/csi-diskplugin:v1.10-7424d08
          args :
            - "--nodeid=$(NODE_ID)"
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--v=5"
          env:
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: unix://var/lib/kubelet/plugins/csi-diskplugin/csi.sock
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: plugin-dir
              mountPath: /var/lib/kubelet/plugins/csi-diskplugin
            - name: pods-mount-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: "Bidirectional"
            - mountPath: /dev
              name: host-dev
            - mountPath: /sys
              name: host-sys
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /run/dbus
              name: host-run
              readOnly: true
            - mountPath: /var/log/alicloud
              name: host-log
      volumes:
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-diskplugin
            type: DirectoryOrCreate
        - name: pods-mount-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: Directory
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-diskplugin
            type: DirectoryOrCreate
        - name: host-dev
          hostPath:
            path: /dev
        - name: host-run
          hostPath:
            path: /run/dbus
        - name: host-sys
          hostPath:
            path: /sys
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: host-log
          hostPath:
            path: /var/log/alicloud/

部署檢查

# kubectl get pod

NAME                                 READY     STATUS    RESTARTS   AGE
csi-attacher-0                       1/1       Running   0          1d
csi-diskplugin-5w7wz                 2/2       Running   0          1d
csi-diskplugin-j99h9                 2/2       Running   0          1d
csi-diskplugin-ljg9s                 2/2       Running   0          1d
csi-diskplugin-nhbfp                 2/2       Running   0          1d
csi-provisioner-0                    1/1       Running   0          1d

使用CSI插件挂載雲盤

下面以建立動态雲盤資料卷為例，介紹如何使用CSI插件挂載；

建立雲盤StorageClass

定義一個SSD、ReadWrite、北京B區的StorageClass；

# kubectl create -f storageclass.yaml

storageclass.yaml内容如下：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-disk
provisioner: csi-diskplugin
parameters:
    zoneId: cn-beijing-b
    regionId: cn-beijing
    fsType: ext4
    type: cloud_ssd
    readOnly: "false"
reclaimPolicy: Delete

建立應用

建立Nginx服務并通過PVC挂載雲盤，在PVC配置csi-disk storageClassName；

# kubectl create -f nginx.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: disk-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 25Gi
  storageClassName: csi-disk
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment1
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        volumeMounts:
          - name: disk-pvc
            mountPath: "/data"
      volumes:
        - name: disk-pvc
          persistentVolumeClaim:
            claimName: disk-pvc

雲盤挂載驗證

檢視nginx容器：

# kubectl get pod | grep nginx
# nginx-deployment1-5879d9db88-wr9j9   1/1       Running   0          1d

檢視容器内挂載資訊：

# kubectl exec -ti nginx-deployment1-5879d9db88-wr9j9 df | grep data
/dev/vdd        25671908   45084  24299720   1% /data
# kubectl exec -ti nginx-deployment1-5879d9db88-wr9j9 touch /data/aliyun
# kubectl exec -ti nginx-deployment1-5879d9db88-wr9j9 ls /data
aliyun    lost+found

删除容器，驗證雲盤高可用：

# kubectl delete pod nginx-deployment1-5879d9db88-wr9j9
# kubectl get pod
NAME                                 READY     STATUS    RESTARTS   AGE
nginx-deployment1-5879d9db88-mjb5p   1/1       Running   0          14s
# kubectl exec -ti nginx-deployment1-5879d9db88-mjb5p ls /data
aliyun    lost+found

上述操作通過CSI插件成功挂載阿裡雲雲盤，并驗證了應用資料的高可用性；

阿裡雲K8S服務支援CSI存儲卷

CSI插件簡介

阿裡雲存儲支援CSI Plugin

使用示範

雲盤CSI Plugin部署

環境準備

部署 CSI Attacher

部署CSI Provsioner

部署Disk Plugin

部署檢查

使用CSI插件挂載雲盤

建立雲盤StorageClass

建立應用

雲盤挂載驗證

繼續閱讀

壓縮編碼M-JPEG、MPEG4、H.264

（Nginx）03_Nginx原理與優化一、Nginx原理二、master-workers機制三、面試題：

（Nginx）02_Nginx配置執行個體一、Nginx的應用：反向代理實作一：二、反向代理執行個體二：三、Nginx的配置執行個體三–負載均衡：四、Nginx的配置執行個體四–動靜分離：Nginx在實際開發中的應用：

nginx配置示例

轉詳解C#資料庫存取圖檔三大方式

BMP檔案結構及圖像每行位元組計算方法

磁盤結構及在Linux中的命名

niginx的location的location詳細規則和優先級關系以及測試案例2.1比對文法2.2優先級判斷2.3練習3.1處理邏輯如何執行3.2功能測試3.3流程解析

Puppet自動化Nginx+Mongrel負載均衡配置

Ubuntu修改nginx将多個域名分别配置到伺服器不同端口

修改or隐藏nginx版本号

Nginx安全改動：隐藏Nginx版本号背景解決serve_tokens指令參考

版本号隐藏

Nginx服務優化（版本隐藏、版本号僞裝、逾時管理、程序管理、日志分割）Nginx服務優化未完待續

nginx location中斜線的位置的重要性