資料存儲技術介紹
DSA(Direct Attacted Storage 直接附加存儲) 本地磁盤接口:IDE SATA SAS SCSI
NAS(NetWork Attacted Storage 網絡附加存儲) 網絡服務共享: 共享的是檔案夾,如 samba NFS
SAN(Storage Attacted Network 存儲附加網絡) 網絡服務共享: 共享的是裝置,如iscsi
ISCSI介紹
ISCSI(internet SCSI)是一個協定,通過TCP/IP網絡的方式将儲存設備共享到用戶端,仿真為本地裝置,讓本地計算機像管理直連存儲一樣,對共享的裝置存儲進行分區,格式化,挂載使用等.ISCSI之間通信時一種C/S架構
用戶端C:為發起者
服務端S:為目标,包含多個LUN(Logical Unit Number邏輯單元),真正提供服務的是LUN,一個LUN為一個裝置,可用同時共享多個裝置給用戶端
ISCSI原理
·SCSI層根據用戶端發出的請求建立SCSI CDB(指令描述塊）,并且傳給ISCSI層。同時接收來自ISCSI層的CDB,并向應用傳回資料。
·ISCSI對SCSICDB進行封裝，以便能夠在基于TCP/IP協定的網絡上進行傳輸，完成SCSI到TCP/IP的協定映射。這一層是ISCS協定的核心層。
·TCP/IP層對IP封包進行路由和轉發，并且提供端到端的透明可靠傳輸。
SAN/IPSAN
IPSAN是在SAN後産生的，
SAN預設指FCSAN,以光纖通道建構存儲網絡，
IPSAN則以IP網絡建構存儲網絡，較FCSAN,具有更經濟、自由擴充等特點。
ISCSI服務端部署
安裝啟動ISCSI服務
# 安裝ISCSI服務
yum install -y targetcli
systemctl enable target
systemctl start target
準備工作:準備共享硬碟裝置或磁盤檔案
[root@centos2 ~]# lsblk # 檢視裝置
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 10G 0 disk
└─sdb1 8:17 0 5G 0 part
sdc 8:32 0 10G 0 disk
sdd 8:48 0 20G 0 disk
sde 8:64 0 10G 0 disk
sdf 8:80 0 10G 0 disk
sr0 11:0 1 1G 0 rom
# 可用dd建立一個5G的檔案
dd if=/dev/zero of=/opt/sdb2 bs=1M count=5000
工作流:将裝置添加到backstores存儲庫--->設定全球唯一辨別名IQN ---> 設定TPG組,定義誰可用重哪個IP即端口通路IQN辨別對應的哪些裝置
[root@centos2 ~]# targetcli # targetcli 進行互動操作
Warning: Could not load preferences file /root/.targetcli/prefs.bin.
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> help
/> ls # 如下4類存儲可用 塊裝置,檔案,實體磁盤,劃分的磁盤
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- block .............................................. [Storage Objects: 0]
| o- fileio ............................................. [Storage Objects: 0]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 0]
o- loopback ..................................................... [Targets: 0]
/>
- 添加儲存設備到存儲庫
/> cd backstores #進入存儲庫
/backstores> block/ create block1 /dev/sde # 建立一個塊裝置,名稱block1,使用系統上/dev/sde裝置
Created block storage object block1 using /dev/sde.
/backstores> ls
o- backstores ............................................................ [...]
o- block ................................................ [Storage Objects: 1]
| o- block1 ...................... [/dev/sde (10.0GiB) write-thru deactivated] #剛建立的裝置
| o- alua ................................................. [ALUA Groups: 1]
| o- default_tg_pt_gp ..................... [ALUA state: Active/optimized]
o- fileio ............................................... [Storage Objects: 0]
o- pscsi ................................................ [Storage Objects: 0]
o- ramdisk .............................................. [Storage Objects: 0]
/backstores>
/backstores> block/ delete block1 #删除裝置
Deleted storage object block1.
/backstores>
/backstores> cd block #進入塊裝置
/backstores/block> create block002 /dev/sdb1 # 建立塊裝置block002
Created block storage object block002 using /dev/sdb1
/backstores/block>
/backstores/block> create block001 /dev/sde # 重新建立裝置block001
Created block storage object block001 using /dev/sde.
/backstores/block>
/> cd backstores/fileio
/backstores/fileio> create block003 /opt/sdb2 # 進入檔案裝置庫,并建立檔案裝置
Created fileio block003 with size 5242880000
/backstores/fileio> ls # 檢視建立的裝置
o- fileio ................................................. [Storage Objects: 1]
o- block003 ...................... [/opt/sdb2 (4.9GiB) write-back deactivated]
o- alua ................................................... [ALUA Groups: 1]
o- default_tg_pt_gp ....................... [ALUA state: Active/optimized]
/backstores/fileio> cd ..
/backstores> ls
o- backstores ............................................................ [...]
o- block ................................................ [Storage Objects: 2]
| o- block001 .................... [/dev/sde (10.0GiB) write-thru deactivated]
| | o- alua ................................................. [ALUA Groups: 1]
| | o- default_tg_pt_gp ..................... [ALUA state: Active/optimized]
| o- block002 .................... [/dev/sdb1 (5.0GiB) write-thru deactivated]
| o- alua ................................................. [ALUA Groups: 1]
| o- default_tg_pt_gp ..................... [ALUA state: Active/optimized]
o- fileio ............................................... [Storage Objects: 1]
| o- block003 .................... [/opt/sdb2 (4.9GiB) write-back deactivated]
| o- alua ................................................. [ALUA Groups: 1]
| o- default_tg_pt_gp ..................... [ALUA state: Active/optimized]
o- pscsi ................................................ [Storage Objects: 0]
o- ramdisk .............................................. [Storage Objects: 0]
/backstores>
- 設定IQN(建立全球唯一辨別裝置名),必須符合統一格式
# IQN格式: iqn.年-月.二級域名倒寫:共享名
/backstores> cd /iscsi
/iscsi> create iqn.2021-05.com.iscsitest:storage # 設定iqn
Created target iqn.2021-05.com.iscsitest:storage.
Created TPG 1. # 自動設定TPG組為 1
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260. #預設監聽的IP和端口号
/iscsi> ls
o- iscsi .......................................................... [Targets: 1]
o- iqn.2021-05.com.iscsitest:storage ............................... [TPGs: 1]
o- tpg1 ............................................. [no-gen-acls, no-auth] # TPG1組
o- acls ........................................................ [ACLs: 0] # 誰可通路
o- luns ........................................................ [LUNs: 0] # 可通路哪些裝置
o- portals .................................................. [Portals: 1]
o- 0.0.0.0:3260 ................................................... [OK] # 或者配置服務端自己的IP
/iscsi>
/iscsi> cd iqn.2021-05.com.iscsitest:storage/tpg1/acls/ # 進入通路權限設定
/iscsi/iqn.20.../tpg1/portals> delete 192.168.198.129 3260
Deleted network portal 192.168.198.132:3260
/iscsi/iqn.20.../tpg1/portals> create 192.168.198.129 3260
- 設定可以通路裝置的使用者為名
# 設定使用者名 iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/acls> create iqn.2021-05.com.iscsitest:cliente001
Created Node ACL for iqn.2021-05.com.iscsitest:cliente001
- 将裝置共享出來
/iscsi/iqn.20...age/tpg1/acls> ../luns/ create /backstores/block/block001 # 将block001裝置共享出來
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/acls> cd ../luns/
/iscsi/iqn.20...age/tpg1/luns> create /backstores/block/block002 # 将block002裝置共享出來
Created LUN 1.
Created LUN 1->1 mapping in node ACL iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/luns> create /backstores/fileio/block003 # 将block003檔案裝置共享出來
Created LUN 2.
Created LUN 2->2 mapping in node ACL iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/luns> cd ..
/iscsi/iqn.20...:storage/tpg1> ls
o- tpg1 ................................................. [no-gen-acls, no-auth]
o- acls ............................................................ [ACLs: 1]
| o- iqn.2021-05.com.iscsitest:cliente001 ................... [Mapped LUNs: 3]
| o- mapped_lun0 ................................ [lun0 block/block001 (rw)]
| o- mapped_lun1 ................................ [lun1 block/block002 (rw)]
| o- mapped_lun2 ............................... [lun2 fileio/block003 (rw)]
o- luns ............................................................ [LUNs: 3]
| o- lun0 ..................... [block/block001 (/dev/sde) (default_tg_pt_gp)]
| o- lun1 .................... [block/block002 (/dev/sdb1) (default_tg_pt_gp)]
| o- lun2 ................... [fileio/block003 (/opt/sdb2) (default_tg_pt_gp)]
o- portals ...................................................... [Portals: 1]
o- 0.0.0.0:3260 ....................................................... [OK]
/iscsi/iqn.20...:storage/tpg1>
- 相應的權限及通路設定
/iscsi/iqn.20...:storage/tpg1> portals/ delete 0.0.0.0 3260 # 也可以對通路的IP和端口進行删除和建立
Deleted network portal 0.0.0.0:3260
/iscsi/iqn.20...:storage/tpg1> portals/ create 0.0.0.0 3260
Using default IP port 3260
Binding to INADDR_ANY (0.0.0.0)
Created network portal 0.0.0.0:3260.
/iscsi/iqn.20...:storage/tpg1> exit # 儲存退出
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup/.
Configuration saved to /etc/target/saveconfig.json
[root@centos2 ~]#
[root@centos2 ~]# vi /etc/target/saveconfig.json # 剛才設定儲存的檔案
[root@centos2 ~]# systemctl restart target # 設定完後重新開機服務
ISCSI用戶端部署
工作流程:安裝用戶端 ---> 設定用戶端iqn裝置名稱 ---> 安裝用戶端服務并啟動 ---> 通路共享 ---> 分區,格式化,自動挂載
1. 安裝用戶端
[root@centos ~]# yum install -y iscsi-initiator-utils
2. 設定iscsi名稱
[root@centos ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2021-05.com.iscsitest:cliente001
3. 啟動iscsi用戶端服務
[root@centos ~]# systemctl start iscsi
[root@centos ~]# systemctl enable iscsi
[root@centos ~]# man iscsiadm
#iscsid 為iscsi服務的守護程序,會随iscsi自動起來,配置檔案/etc/iscsi/iscsid.conf
4. 發現共享裝置
[root@centos ~]# iscsiadm --mode discoverydb --type sendtargets --portal 192.168.198.129 --discover
192.168.198.129:3260,1 iqn.2021-05.com.iscsitest:storage
[root@centos ~]# ll /var/lib/iscsi/nodes #檢視
total 0
drw------- 3 root root 36 May 23 02:36 iqn.2021-05.com.iscsitest:storage
5. 連接配接共享裝置
[root@centos ~]# iscsiadm --mode node --targetname iqn.2021-05.com.iscsitest:storage --portal 192.168.198.129 --login
Logging in to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.129,3260] (multiple)
Login to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.129,3260] successful.
[root@centos ~]# lsblk # 檢視挂載成功
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk # sdc/sdd/sde為新添加
sdd 8:48 0 5G 0 disk
sde 8:64 0 4.9G 0 disk
sr0 11:0 1 1G 0 rom
6. 進行分區格式化挂載
[root@centos ~]# fdisk /dev/sdc
[root@centos ~]# vi /etc/fstab
/dev/sdc1 /mnt/block1 ext4 _netdev 0 0 # _netdev 網絡檔案,不要用預設
[root@centos ~]# mkfs.ext4 /dev/sdc1
[root@centos ~]# mount -a
[root@centos ~]#
[root@centos ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 899M 0 899M 0% /dev
tmpfs 910M 0 910M 0% /dev/shm
tmpfs 910M 9.5M 901M 2% /run
tmpfs 910M 0 910M 0% /sys/fs/cgroup
/dev/mapper/centos-root 17G 4.7G 13G 28% /
/dev/sda1 1014M 181M 834M 18% /boot
tmpfs 182M 0 182M 0% /run/user/0
192.168.198.130:/test 17G 4.7G 13G 28% /mnt/nfs
/dev/sdc1 9.7G 37M 9.2G 1% /mnt/block1 # 檢視已經挂載成功
用戶端停止ISCSI 服務
# 檢視服務名
[root@centos ~]# iscsiadm --mode node
192.168.198.129:3260,1 iqn.2021-05.com.iscsitest:storage
# 用戶端停止使用ISCSI共享
[root@centos ~]# iscsiadm --mode node --targetname iqn.2021-05.com.iscsitest:storage --portal 192.168.198.129:3260 --logout
# 徹底删除,如果login登入不了,需要删除裝置,重新發現并登入
[root@centos ~]# rm -rf /var/lib/iscsi/nodes/iqn.2021-05.com.iscsitest:storage/
[root@centos ~]# rm -rf /var/lib/iscsi/send_targets/192.168.198.129,3260/
# 以上相關指令可用簡寫
IPSAN多鍊路共享
解決單點故障
1. 準備工作
服務端和用戶端都配備兩塊網卡進行連接配接
添加網卡,配置IP位址
# 服務端,添加新網卡
[root@centos2 ~]# ip a
192.168.198.129/24
192.168.198.132/24
# 用戶端也添加新網卡
[root@centos ~]# ip a
192.168.198.133/24
192.168.198.130/24
# nmcli配置IP位址
nmcli con add con-name eth1 ifname ens36 type 802-3-ethernet ipv4.method manual ipv4.add 192.168.198.136/24
# 檢查以上網卡是否互相ping通
[root@centos2 ~]# ping 192.168.198.133
[root@centos2 ~]# ping 192.168.198.130
[root@centos ~]#192.168.198.132
[root@centos ~]#192.168.198.129
2. 服務端設定
參考:ISCSI服務端部署,配置成如下配置
[root@centos2 ~]# targetcli
/> ls
o- / ...................................................................... [...]
o- backstores ........................................................... [...]
| o- block ............................................... [Storage Objects: 1]
| | o- block001 ..................... [/dev/sdc (10.0GiB) write-thru activated]
| | o- alua ................................................ [ALUA Groups: 1]
| | o- default_tg_pt_gp .................... [ALUA state: Active/optimized]
| o- fileio .............................................. [Storage Objects: 0]
| o- pscsi ............................................... [Storage Objects: 0]
| o- ramdisk ............................................. [Storage Objects: 0]
o- iscsi ......................................................... [Targets: 1]
| o- iqn.2021-05.com.iscsitest:storage1 ............................. [TPGs: 1]
| o- tpg1 ............................................ [no-gen-acls, no-auth]
| o- acls ....................................................... [ACLs: 1]
| | o- iqn.2021-05.com.iscsitest:clientemultipath ........ [Mapped LUNs: 1]
| | o- mapped_lun0 ........................... [lun0 block/block001 (rw)]
| o- luns ....................................................... [LUNs: 1]
| | o- lun0 ................ [block/block001 (/dev/sdc) (default_tg_pt_gp)] # LUN 把裝置共享出來
| o- portals ................................................. [Portals: 2]
| o- 192.168.198.129:3260 .......................................... [OK] # 配置服務端的IP進行共享
| o- 192.168.198.132:3260 .......................................... [OK]
o- loopback ...................................................... [Targets: 0]
3. 用戶端設定
工作流程: 安裝用戶端并啟動 ---> 連接配接共享裝置從兩條線路 ---> 分區格式化 ---> 安裝多路徑軟體 ---> 實作多路徑負載均衡
用戶端配置參考:ISCSI用戶端部署
# 用戶端發現兩個裝置,并進行挂載
[root@centos ~]# iscsiadm --mode discoverydb --type sendtargets --portal 192.168.198.129 --discover
192.168.198.129:3260,1 iqn.2021-05.com.iscsitest:storage
192.168.198.132:3260,1 iqn.2021-05.com.iscsitest:storage
[root@centos ~]# iscsiadm --mode node --targetname iqn.2021-05.com.iscsitest:storage --portal 192.168.198.129 --login
Logging in to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.129,3260] (multiple)
Login to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.129,3260] successful.
[root@centos ~]#
[root@centos ~]# iscsiadm --mode node --targetname iqn.2021-05.com.iscsitest:storage --portal 192.168.198.132 --login
Logging in to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.132,3260] (multiple)
Login to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.132,3260] successful.
分區及格式化
[root@centos ~]# fdisk /dev/sdc
[root@centos ~]# mkfs.ext4 /dev/sdc1
[root@centos ~]# partprobe #更新分區标資訊
[root@centos ~]#
[root@centos ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk
└─sdc1 8:33 0 10G 0 part
sdd 8:48 0 10G 0 disk
└─sdd1 8:49 0 10G 0 part
sr0 11:0 1 1G 0 rom
安裝multipath多路徑軟體并啟動
# 安裝多路徑軟體
yum install -y device-mapper-multipath
# 拷貝配置檔案
cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/
# 啟動服務
systemctl enable multipathd; systemctl start multipathd
# 檢視
[root@centos ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk
└─mpatha 253:2 0 10G 0 mpath
└─mpatha1 253:3 0 10G 0 part
sdd 8:48 0 10G 0 disk
└─mpatha 253:2 0 10G 0 mpath
└─mpatha1 253:3 0 10G 0 part
sr0 11:0 1 1G 0 rom
配置multipath負載均衡模式
[root@centos ~]# multipath -ll # 多路徑block001裝置id 360014051f73df963d734b45aa3f450e9
mpatha (360014051f73df963d734b45aa3f450e9) dm-2 LIO-ORG ,block001
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active # 預設是提供AB模式,可修改配置檔案,變成負載均衡模式
| `- 21:0:0:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 22:0:0:0 sdd 8:48 active ready running
[root@centos ~]# vi /etc/multipath.conf
#defaults {
# user_friendly_names yes
# find_multipaths yes
#}
multipaths {
multipath {
wwid 360014051f73df963d734b45aa3f450e9 # 裝置編号
alias multipathtest # 别名
path_grouping_policy multibus # 多路徑組政策
path_selector "round-robin 0" # 負載均衡模式
failback manual
rr_weight priorities # 按優先級輪詢
no_path_retry 5 # 重試時間
}
multipath {
wwid 1DEC_____321816758474
alias red
}
}
[root@centos ~]# systemctl start multipathd
[root@centos ~]# systemctl start iscsi
[root@centos ~]# multipath -ll
multipathtest (360014051f73df963d734b45aa3f450e9) dm-2 LIO-ORG ,block001
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 21:0:0:0 sdc 8:32 active ready running # 再次檢視,多路徑已經變成rr負載均衡模式
`- 22:0:0:0 sdd 8:48 active ready running
# multipath -F #删除現有路徑,新的路徑就會被删除
# multipath -v2 #格式化路徑,格式化後又出現
[root@centos ~]# ll /dev/mapper
total 0
lrwxrwxrwx 1 root root 7 May 23 16:24 centos-root -> ../dm-0
lrwxrwxrwx 1 root root 7 May 23 16:24 centos-swap -> ../dm-1
crw------- 1 root root 10, 236 May 23 14:28 control
lrwxrwxrwx 1 root root 7 May 23 16:56 multipathtest -> ../dm-2
lrwxrwxrwx 1 root root 7 May 23 16:56 multipathtest1 -> ../dm-3
[root@centos ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk
└─multipathtest 253:2 0 10G 0 mpath # 這裡是剛剛配置檔案中設定的别名,也可以不設定别名
└─multipathtest1 253:3 0 10G 0 part
sdd 8:48 0 10G 0 disk
└─multipathtest 253:2 0 10G 0 mpath
└─multipathtest1 253:3 0 10G 0 part
sr0 11:0 1 1G 0 rom
# 挂載使用(也可以umount後重新格式化)
[root@centos ~]# mount /dev/mapper/multipathtest1 /mnt/multi/
multipath多鍊路容災測試
#在用戶端可進行讀寫操作
[root@centos ~]# touch /mnt/multi/file1
# 将服務端網卡down掉
[root@centos2 ~]# ifdown ens37
# 在用戶端檢視,掉線一條鍊路,另一條正常運作
[root@centos ~]# multipath -ll
multipathtest (360014051f73df963d734b45aa3f450e9) dm-2 LIO-ORG ,block001
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 21:0:0:0 sdc 8:32 active faulty running
`- 22:0:0:0 sdd 8:48 active ready running
# 将服務端網卡啟動
[root@centos2 ~]# ifup ens37
# 在用戶端檢視,兩條鍊路都正常運作
[root@centos ~]# multipath -ll
multipathtest (360014051f73df963d734b45aa3f450e9) dm-2 LIO-ORG ,block001
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 21:0:0:0 sdc 8:32 active ready running
`- 22:0:0:0 sdd 8:48 active ready running
![Docker - Dockerfile之ADD、COPY、WORKDIR、USER、EXPOSE指令詳解[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)