linux学习之搭建高可用分布式文件系统MFS

分布式文件系统mfs

vm1，vm2做高可用；vm3，vm4做存储结点，物理机做client

192.168.2.199 vm1.example.com

192.168.2.202 vm2.example.com

192.168.2.205 vm3.example.com

192.168.2.175 vm4.example.com

192.168.2.199 mfsmaster

vip 213

1、master配置启动

lftp i:~> get pub/docs/mfs/mfs-1.6.27-1.tar.gz

[[email protected] ~]# mv mfs-1.6.27-1.tar.gz mfs-1.6.27.tar.gz

[[email protected] ~]# yum install -y fuse-devel

[[email protected] ~]# rpmbuild -tb mfs-1.6.27.tar.gz

[[email protected] ~]# cd rpmbuild/RPMS/x86_64/

[[email protected] x86_64]# rpm -ivh mfs-cgi-1.6.27-2.x86_64.rpm mfs-cgiserv-1.6.27-2.x86_64.rpm mfs-master-1.6.27-2.x86_64.rpm

[[email protected] x86_64]# cd /etc/mfs/

[[email protected] mfs]# cp mfsmaster.cfg.dist mfsmaster.cfg

[[email protected] mfs]# cp mfsexports.cfg.dist mfsexports.cfg

[[email protected] mfs]# cp mfstopology.cfg.dist mfstopology.cfg

[[email protected] mfs]# cd /var/lib/mfs/

[[email protected] mfs]# cp metadata.mfs.empty metadata.mfs

[[email protected] mfs]# chown -R nobody .

[[email protected] mfs]# vim /etc/hosts

192.168.2.199 mfsmaster

[[email protected] mfs]# mfsmaster 启动mfsmaster

启动mfscgiserv

[[email protected] mfscgi]# mfsmaster

[[email protected] mfs]# cd /usr/share/mfscgi/

[[email protected] mfscgi]# chmod +x *.cgi

[[email protected] mfscgi]# mfscgiserv

物理机访问192.168.2.199:9425

2、配置存储结点

[[email protected] ~]# scp rpmbuild/RPMS/x86_64/mfs-chunkserver-1.6.27-2.x86_64.rpm vm3.example.com:

[[email protected] ~]# scp rpmbuild/RPMS/x86_64/mfs-chunkserver-1.6.27-2.x86_64.rpm vm4.example.com:

两个结点均做相同配置

[[email protected] ~]# rpm -ivh mfs-chunkserver-1.6.27-2.x86_64.rpm

[[email protected] ~]# mkdir /mnt/chunk1

[[email protected] ~]# mkdir /var/lib/mfs

[[email protected] ~]# chown nobody /mnt/chunk1/ /var/lib/mfs/

[[email protected] ~]# cd /etc/mfs/

[[email protected] mfs]# cp mfschunkserver.cfg.dist mfschunkserver.cfg

[[email protected] mfs]# cp mfshdd.cfg.dist mfshdd.cfg

[[email protected] mfs]# vim mfshdd.cfg

/mnt/chunk1

[[email protected] mfs]# vim /etc/hosts

192.168.2.199 mfsmaster

[[email protected] mfs]# mfschunkserver

然后刷新192.168.2.199:9425网页，查看存储结点服务器

3、client客户端配置

[[email protected] x86_64]# scp mfs-client-1.6.27-2.x86_64.rpm 192.168.2.168:

[[email protected] ~]# rpm -ivh mfs-client-1.6.27-2.x86_64.rpm

[[email protected] ~]# cd /etc/mfs/

[[email protected] mfs]# cp mfsmount.cfg.dist mfsmount.cfg

[[email protected] mfs]# vim mfsmount.cfg

/mnt/mfs

[[email protected] mfs]# vim /etc/hosts

192.168.2.199 mfsmaster

[[email protected] mfs]# mkdir /mnt/mfs

[[email protected] mfs]# mfsmount 就会挂载

测试

[[email protected] mfs]# mfssetgoal -r 2 dir2/ 设置文件夹dir2中的文件均保存2份

[[email protected] mfs]# mfsgetgoal dir1/

dir1/: 1

[[email protected] mfs]# mfsgetgoal dir2/

dir2/: 2

[[email protected] mfs]# cp /etc/passwd dir1/

[[email protected] mfs]# cp /etc/fstab dir2/

[[email protected] mfs]# mfsfileinfo dir1/passwd

dir1/passwd:

chunk 0: 0000000000000001_00000001 / (id:1 ver:1)

copy 1: 192.168.2.175:9422

[[email protected] mfs]# mfsfileinfo dir2/fstab

dir2/fstab:

chunk 0: 0000000000000003_00000001 / (id:3 ver:1)

copy 1: 192.168.2.175:9422

copy 2: 192.168.2.205:9422

[[email protected] ~]# mfschunkserver stop 停止存储结点的服务

[[email protected] mfs]# mfsfileinfo dir2/fstab

dir2/fstab:

chunk 0: 0000000000000003_00000001 / (id:3 ver:1)

copy 1: 192.168.2.175:9422

再次开启，又会看到两份，这样避免单点故障

[[email protected] mfs]# mfschunkserver stop再次关闭

[[email protected] mfs]# mfsfileinfo dir1/passwd

dir1/passwd:

chunk 0: 0000000000000001_00000001 / (id:1 ver:1)

no valid copies !!! 虽然能看到文件，但是是无效的

[[email protected] mfs]# mfsfileinfo dir2/fstab

dir2/fstab:

chunk 0: 0000000000000003_00000001 / (id:3 ver:1)

no valid copies !!!

误删文件回复

[[email protected] ~]# mkdir /mnt/meta

[[email protected] ~]# mfsmount -m /mnt/meta/ -H mfsmaster

[[email protected] ~]# cd /mnt/meta/trash/

[[email protected] trash]# mv 0000093F\|etc\|xdg\|autostart\|pulseaudio.desktop undel/

master恢复

[[email protected] ~]# mfsmetarestore -a

[[email protected] ~]# mfsmaster

4、制作master的HA

停止mfs

[[email protected] ~]# umount /mnt/mfs/

[[email protected] chunk1]# mfschunkserver stop

[[email protected] ~]# mfsmaster stop

制作启动脚本

[[email protected] init.d]# vim mfs

#!/bin/bash

# Init file for the MooseFS master service

# chkconfig: - 92 84

# description: MooseFS master

# processname: mfsmaster

# Source function library.

# Source networking configuration.

. /etc/init.d/functions

. /etc/sysconfig/network

# Source initialization configuration.

# Check that networking is up.

[ "${NETWORKING}" == "no" ] && exit 0

[ -x "/usr/sbin/mfsmaster" ] || exit 1

[ -r "/etc/mfs/mfsmaster.cfg" ] || exit 1

[ -r "/etc/mfs/mfsexports.cfg" ] || exit 1

RETVAL=0

prog="mfsmaster"

datadir="/var/lib/mfs"

mfsbin="/usr/sbin/mfsmaster"

mfsrestore="/usr/sbin/mfsmetarestore"

start () {

echo -n $"Starting $prog: "

$mfsbin start >/dev/null 2>&1

if [ $? -ne 0 ];then

$mfsrestore -a >/dev/null 2>&1 && $mfsbin start >/dev/null 2>&1

RETVAL=$?

echo

return $RETVAL

}

stop () {

echo -n $"Stopping $prog: "

$mfsbin -s >/dev/null 2>&1 || killall -9 $prog #>/dev/null 2>&1

RETVAL=$?

echo

return $RETVAL

}

restart () {

stop

start

}

reload () {

echo -n $"reload $prog: "

$mfsbin reload >/dev/null 2>&1

RETVAL=$?

echo

return $RETVAL

}

restore () {

echo -n $"restore $prog: "

$mfsrestore -a >/dev/null 2>&1

RETVAL=$?

echo

return $RETVAL

}

case "$1" in

start)

start

;;

stop)

stop

;;

restart)

restart

;;

reload)

reload

;;

restore)

restore

;;

status)

status $prog

RETVAL=$?

;;

echo $"Usage: $0 {start|stop|restart|reload|restore|status}"

RETVAL=1

esac

exit $RETVAL

[[email protected] init.d]# chmod +x mfs

[[email protected] init.d]# /etc/init.d/mfs start 测试

[[email protected] ~]# ps -axu | grep mfsmaster

[[email protected] init.d]# /etc/init.d/mfs stop

[[email protected] init.d]# scp mfs vm2.example.com:/etc/init.d/

[[email protected] x86_64]# scp mfs-master-1.6.27-2.x86_64.rpm mfs-cgi-1.6.27-2.x86_64.rpm mfs-cgiserv-1.6.27-2.x86_64.rpm vm2.example.com:

[[email protected] ~]# rpm -ivh mfs-master-1.6.27-2.x86_64.rpm mfs-cgi-1.6.27-2.x86_64.rpm mfs-cgiserv-1.6.27-2.x86_64.rpm

[[email protected] mfs]# cp mfsmaster.cfg.dist mfsmaster.cfg

[[email protected] mfs]# cp mfsexports.cfg.dist mfsexports.cfg

[[email protected] mfs]# cp mfstopology.cfg.dist mfstopology.cfg

[[email protected] mfs]# cd /var/lib/mfs/

[[email protected] mfs]# cp metadata.mfs.empty metadata.mfs

[[email protected] mfs]# chown -R nobody .

[[email protected] mfs]# cd /usr/share/mfscgi/

[[email protected] mfscgi]# chmod +x *.cgi

[[email protected] ~]# vim /etc/hosts 修改所有结点，mfsmaster解析为虚拟ip

192.168.2.213 mfsmaster

pacemaker还原

[[email protected] ~]# /etc/init.d/corosync start 两个结点先启动

crm(live)resource# stop vip

crm(live)configure# delete vip

crm(live)configure# delete webdata

crm(live)configure# delete website

crm(live)configure# show

node vm1.example.com

node vm2.example.com

primitive vmfence stonith:fence_xvm \

params pcmk_host_map="vm1.example.com:vm1;vm2.example.com:vm2" \

op monitor interval="60s" \

meta target-role="Started"

property $id="cib-bootstrap-options" \

dc-version="1.1.10-14.el6-368c726" \

cluster-infrastructure="classic openais (with plugin)" \

expected-quorum-votes="2" \

stonith-enabled="true" \

no-quorum-policy="ignore"

crm(live)configure# commit

[[email protected] ~]# /etc/init.d/corosync stop 关闭

5、安装DRBD，存放master调用文件

lftp i:~> get pub/docs/drbd/rhel6/drbd-8.4.3.tar.gz

[[email protected] ~]# tar zxf drbd-8.4.3.tar.gz

[[email protected] ~]# cd drbd-8.4.3

[[email protected] drbd-8.4.3]# yum install -y flex kernel-devel

[[email protected] drbd-8.4.3]# ./configure --enable-spec --with-km

[[email protected] drbd-8.4.3]# cp ../drbd-8.4.3.tar.gz /root/rpmbuild/SOURCES/

[[email protected] drbd-8.4.3]# rpmbuild -bb drbd.spec

[[email protected] drbd-8.4.3]# rpmbuild -bb drbd-km.spec

[[email protected] ~]# cd rpmbuild/RPMS/x86_64/

[[email protected] x86_64]# rpm -ivh drbd-*

[[email protected] x86_64]# scp drbd-* vm2.example.com:

[[email protected] ~]# rpm -ivh drbd-*

然后vm1和vm2添加2G大小的虚拟磁盘

[[email protected] ~]# vim /etc/drbd.d/mfsdata.res

resource mfsdata {

meta-disk internal;

device /dev/drbd1;

syncer {

verify-alg sha1;

}

on vm1.example.com {

disk /dev/vdb1;

address 192.168.2.199:7789;

}

on vm2.example.com {

disk /dev/vdb1;

address 192.168.2.202:7789;

}

[[email protected] ~]# scp /etc/drbd.d/mfsdata.res vm2.example.com:/etc/drbd.d/

分出/dev/vdb1，两个结点都做。如下：

[[email protected] ~]# fdisk -cu /dev/vdb

[[email protected] ~]# drbdadm create-md mfsdata

[[email protected] ~]# /etc/init.d/drbd start

设置结点1为主结点，格式化

[[email protected] ~]# drbdsetup primary /dev/drbd1 --force

[[email protected] ~]# mkfs.ext4 /dev/drbd1

给磁盘中写入mfs的文件

[[email protected] ~]# mount /dev/drbd1 /mnt/

[[email protected] ~]# cd /var/lib/mfs/

[[email protected] mfs]# mv * /mnt/

[[email protected] mfs]# cd /mnt/

[[email protected] mnt]# chown nobody .

[[email protected] ~]# umount /mnt/

[[email protected] ~]# drbdadm secondary mfsdata

另一个结点查看

[[email protected] ~]# drbdadm primary mfsdata

[[email protected] ~]# mount /dev/drbd1 /var/lib/mfs/

[[email protected] ~]# cd /var/lib/mfs/

[[email protected] mfs]# ls

changelog.2.mfs changelog.6.mfs metadata.mfs metadata.mfs.empty stats.mfs

changelog.3.mfs lost+found metadata.mfs.back.1 sessions.mfs

[[email protected] ~]# umount /var/lib/mfs/

关闭客户端iscsi

[[email protected] ~]# iscsiadm -m node -u

[[email protected] ~]# iscsiadm -m node -o delete

[[email protected] ~]# /etc/init.d/iscsi stop

[[email protected] ~]# chkconfig iscsi off

[[email protected] ~]# chkconfig iscsid off

6、corosync加入资源

crm(live)configure# primitive MFSDATA ocf:linbit:drbd params drbd_resource=mfsdata 资源drbd用于mfsmaster

crm(live)configure# primitive MFSfs ocf:heartbeat:Filesystem params device=/dev/drbd1 directory=/var/lib/mfs fstype=ext4 文件系统资源

crm(live)configure# ms mfsdataclone MFSDATA meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true 定义主备

crm(live)configure# primitive mfsmaster lsb:mfs op monitor interval=30s 定义mfsmaster资源

crm(live)configure# group mfsgrp vip MFSfs mfsmaster

crm(live)configure# colocation mfs-with-drbd inf: mfsgrp mfsdataclone:Master

crm(live)configure# order mfs-after-drbd inf: mfsdataclone:promote mfsgrp:start

crm(live)configure# commit

监控如下：

Online: [ vm1.example.com vm2.example.com ]

vmfence (stonith:fence_xvm): Started vm1.example.com

Master/Slave Set: mfsdataclone [MFSDATA]

Masters: [ vm1.example.com ]

Slaves: [ vm2.example.com ]

Resource Group: mfsgrp

vip (ocf::heartbeat:IPaddr2): Started vm1.example.com

MFSfs (ocf::heartbeat:Filesystem): Started vm1.example.com

mfsmaster (lsb:mfs): Started vm1.example.com

配置文件如下：

node vm1.example.com

node vm2.example.com

primitive MFSDATA ocf:linbit:drbd \

params drbd_resource="mfsdata"

primitive MFSfs ocf:heartbeat:Filesystem \

params device="/dev/drbd1" directory="/var/lib/mfs" fstype="ext4"

primitive mfsmaster lsb:mfs \

op monitor interval="30s"

primitive vip ocf:heartbeat:IPaddr2 \

params ip="192.168.2.213" cidr_netmask="32" \

op monitor interval="30s"

primitive vmfence stonith:fence_xvm \

params pcmk_host_map="vm1.example.com:vm1;vm2.example.com:vm2" \

op monitor interval="60s" \

meta target-role="Started"

group mfsgrp vip MFSfs mfsmaster

ms mfsdataclone MFSDATA \

meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"

colocation mfs-with-drbd inf: mfsgrp mfsdataclone:Master

order mfs-after-drbd inf: mfsdataclone:promote mfsgrp:start

property $id="cib-bootstrap-options" \

dc-version="1.1.10-14.el6-368c726" \

cluster-infrastructure="classic openais (with plugin)" \

expected-quorum-votes="2" \

stonith-enabled="true" \

no-quorum-policy="ignore"

测试高可用

[[email protected] ~]# /etc/init.d/corosync start

[[email protected] ~]# mfschunkserver

[[email protected] ~]# mfsmount

关闭vm1的corosync，vm2接管所有资源，vm1状态OFFLINE: [ vm1.example.com ]，Masters: [ vm2.example.com ] Stopped: [ vm1.example.com ]，再开启vm1的corosync，关闭再开启vm2的corosync，使资源启动在vm1上；

在客户端[[email protected] dir2]# dd if=/dev/zero of=bigfile bs=1M count=300 过程中关闭vm1的mfs，fence会将其重启，资源服务移到另一结点（这里出现了vm3的崩溃，可能是快照满了，若客户端有双份文件的话，会丢失一份）；若关闭corosync资源服务也会移到另一结点，都不会影响客户端的文件。

之后使用cat查看文件来测试，客户端一直cat fstab，master关闭corosync测试，会有短暂切换延迟。

fence测试，关闭vm1的eth0，vm1进入重启，过程中vm2需要启动drbd，vm1开启之后，开启corosync，drbd是开机自启的，也可以把corosync开机自启。

总结：之后在自己电脑上做，遇到一些问题，导致重装master，使用rpm -e mfs-master卸载，重装然后格式化drbd，修改权限

这里corosync和drbd在一块，drbd做的是mfs的master的存储文件/var/lib/mfs，master相当于调度

vm3和vm4还是存储结点[[email protected] ~]# cd /mnt/chunk1/

[[email protected] ~]# ls /mnt/chunk1/

00 0D 1A 27 34 41 4E 5B 68 75 82 8F 9C A9 B6 C3 D0 DD EA F7

最后需要做的就是开机启动drbd，corosync！

7、使用heartbeat+mfsmaster高可用

这里是在自己原来做过heartbeat的电脑做的，主机名有变化

先关闭所有，注意顺序，客户端先卸载

[[email protected] ~]# umount /mnt/mfs/

[[email protected] ~]# mfschunkserver stop

[[email protected] ~]# /etc/init.d/corosync stop

[[email protected] ~]# vim /etc/ha.d/haresources

[[email protected] ~]# vim /etc/ha.d/haresources 修改

vm1.example.com IPaddr::192.168.2.213/24/eth0:0 drbddisk::mfsdata Filesystem::/dev/drbd1::/var/lib/mfs::ext4 mfs

[[email protected] ~]# /etc/init.d/drbd start 开启drbd，比较重要

[[email protected] ~]# /etc/init.d/drbd start 直到看到两个都是secondary

[[email protected] ~]# /etc/init.d/heartbeat start 开启heartbeat，注意查看日志

[[email protected] ~]# /etc/init.d/heartbeat start

[[email protected] ~]# mfschunkserver 开启存储节点

[[email protected] ~]# mfschunkserver

[[email protected] ~]# mfsmount 挂载，df查看

[[email protected] ~]# cat /mnt/mfs/1/fstab 查看文件内容

高可用测试

[[email protected] ~]# /etc/init.d/heartbeat stop 资源跳到另一个结点，不影响查看文件，再次开启会跳回来

[[email protected] ~]# /etc/init.d/mfs start 关闭MFS查看文件会卡住，不带有服务资源得检测。

linux学习之搭建高可用分布式文件系统MFS

1、master配置启动

2、配置存储结点

3、client客户端配置

4、制作master的HA

5、安装DRBD，存放master调用文件

6、corosync加入资源

7、使用heartbeat+mfsmaster高可用

继续阅读

Apache (You don't have permission to access / on this server.）

debian9升级4.9.0内核到4.19.2内核过程

centOS7 配置 vsftpd 虚拟用户及权限Vsftpd配置虚拟用户及权限

linux-svn卸载与安装

vsftp虚拟多用户多权限一键部署脚本

Ubuntu14.04 LTS下安装mongodb

Nginx服务优化（1）——隐藏版本号、修改用户与组、网页缓存时间、日志切割、连接超时一、隐藏版本号二、修改用户与组三、配置Nginx网页缓存时间四、实现Nginx日志分割五、配置Nginx实现连接超时六、补充关于时间日期的命令

httpd服务的部署、启动、配置和简单优化一、部署二、启动三、配置文件

配置网页内容访问

手动安装Intel network I217-LM网卡的Linux驱动

禁止ubuntu系统弹出报错界面

Ubuntu Linux下Apache的配置文件

samba服务器的功能

【Linux】UDP广播报文接收速率问题

Linux设备模型（中）之上层容器

PowerPC平台 Linux移植三