15、 Heartbeat+DRBD+MySQL高可用架構方案與實施過程細節
參考自:http://oldboy.blog.51cto.com/2561410/1240412
heartbeat和keepalived應用場景及差別 | 很多網友說為什麼不使用keepalived而使用長期不更新的heartbeat,下面說一下它們之間的應用場景及差別: 1、對于web,db,負載均衡(lvs,haproxy,nginx)等,heartbeat和keepalived都可以實作 2、lvs最好和keepalived結合,因為keepalived最初就是為lvs産生的,(heartbeat沒有對RS的健康檢查功能,heartbeat可以通過ldircetord來進行健康檢查的功能) 3、mysql雙主多從,NFS/MFS存儲,他們的特點是需要資料同步,這樣的業務最好使用heartbeat,因為heartbeat有自帶的drbd腳本 總結: 無資料同步的應用程式高可用可選擇keepalived, 有資料同步的應用程式高可用可選擇heartbeat(DRBD) 。 |
1、 安裝部署準備 | (1)架構拓撲 ![]() 架構說明: 一主多從最常用的架構,多個從庫可以使用lvs來提供讀的負載均衡。 解決一主單點的問題,當主庫當機後,可以實作主庫當機後備節點自動接管,所有的從庫會自動和新的主庫進行同步,實作了mysql主庫的熱備方案 (2)系統環境: ![]() ![]() ![]() |
2、heatbeat安裝部署 | (1)、配置伺服器間心跳連接配接路由 主節點 [ro[email protected] ~]# route add -host 172.16.4.3 dev eth2<==到對端的心跳路由 [[email protected] ~]# route add -host 172.168.4.3 dev eth3<==到對端的DRBD資料路由 備節點 [[email protected] ~]# route add -host 172.16.4.2 dev eth2 [[email protected] ~]# route add -host 172.168.4.2 dev eth3 (2)、安裝heartbeat [[email protected] ~]# yum install heartbeat -y [[email protected] ~]# yum install heartbeat -y 提示:需要執行兩遍安裝heartbeat操作 (3)、配置heartbeat 主備節點兩端的配置檔案(ha.cfauthkeysharesources)完全相同 1)、ha.cf [[email protected] ~]# vim /etc/ha.d/ha.cf #log configure debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local1 #options configure keepalive 2 deadtime 30 warntime 10 initdead 120 #bcast eth2 mcast eth2 225.0.0.7 694 1 0 #node configure auto_failback on node master1 <==主節點主機名 node master2 <==備節點主機名 crm no 2)、配置authkeys [[email protected] ~]# vim /etc/ha.d/authkeys auth 1 1 sha1 47e9336850f1db6fa58bc470bc9b7810eb397f04 3)、配置haresources [[email protected] ~]# vim /etc/ha.d/haresources master1 IPaddr::192.168.4.1/16/eth1 #master1 IPaddr::192.168.4.1/16/eth1 drbddisk::data Filesystem::/dev/drbd1::/data::ext3 mysqld 說明: drbddisk::data <==啟動drbd data資源,相當于執行/etc/ha.d/resource.d/drbddisk data stop/start操作 Filesystem::/dev/drbd1::/data::ext3 <==drbd分區挂載到/data目錄,相當于執行/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext3 stop/start <==相當于系統中執行mount /dev/drbd1 /data mysql <==啟動mysql服務腳本,相當于/etc/init.d/mysql stop/start (4)、啟動heartbeat [[email protected] ~]# /etc/init.d/heartbeat start [[email protected] ~]# chkconfig heartbeat off 說明:關閉開機自啟動,當伺服器重新開機時,需要人工去啟動 (5)、測試heartbeat 正常狀态 [[email protected] ~]# ip addr|grep eth1 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 inet 192.168.4.2/16 brd 192.168.255.255 scope global eth1 inet 192.168.4.1/16 brd 192.168.255.255 scope global secondary eth1:0 [[email protected] ~]# ip addr|grep eth1 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 inet 192.168.4.3/16 brd 192.168.255.255 scope global eth1 說明:master1節點擁有vip位址,master2節點沒有 模拟主節點當機後的狀态 [[email protected] ~]# /etc/init.d/heartbeat stop [[email protected] ~]# ip addr|grep eth1 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 inet 192.168.4.3/16 brd 192.168.255.255 scope global eth1 inet 192.168.4.1/16 brd 192.168.255.255 scope global secondary eth1:0 說明:master1當機後,vip位址漂移到master2節點上,master2成為主節點 模拟主節點故障恢複後的狀态 [[email protected] ~]# /etc/init.d/heartbeat start [[email protected] ~]# ip addr|grep eth1 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 inet 192.168.4.2/16 brd 192.168.255.255 scope global eth1 inet 192.168.4.1/16 brd 192.168.255.255 scope global secondary eth1:0 說明:master1搶占vip資源 |
3、DRBD 安裝部署 | (1)、新添加硬碟 [ro[email protected] ~]# fdisk /dev/sdb #sdb磁盤分兩個分區sdb1和sdb2 [[email protected] ~]# partprobe [[email protected] ~]# mkfs.ext3 /dev/sdb1 #sdb2分區為meta data分區,不需要格式化操作 [[email protected] ~]# tune2fs -c -1 /dev/sdb1 #設定最大挂載數為-1 (2)、安裝DRBD [[email protected] ~]# yum install kmod-drbd83 drbd83 -y [[email protected] ~]# modprobe drbd #注意:不要設定echo 'modprobe drbd' >>/etc/rc.loca開機自動加載drbd子產品,如果drbd服務是開機自啟動的,會先啟動drbd服務在加載drbd的順序,導緻drbd啟動不了出現的問題 (3)、配置DRBD 主備節點兩端配置檔案完全一緻 [[email protected] ~]# cat /etc/drbd.conf global { # minor-count 64; # dialog-refresh 5; # 5 seconds # disable-ip-verification; usage-count no; } common { protocol C; disk { on-io-error detach; #size 454G; no-disk-flushes; no-md-flushes; } net { sndbuf-size 512k; # timeout 60; # 6 seconds (unit = 0.1 seconds) # connect-int 10; # 10 seconds (unit = 1 second) # ping-int 10; # 10 seconds (unit = 1 second) # ping-timeout 5; # 500 ms (unit = 0.1 seconds) max-buffers 8000; unplug-watermark 1024; max-epoch-size 8000; # ko-count 4; # allow-two-primaries; cram-hmac-alg "sha1"; shared-secret "hdhwXes23sYEhart8t"; after-sb-0pri disconnect; after-sb-1pri disconnect; after-sb-2pri disconnect; rr-conflict disconnect; # data-integrity-alg "md5"; # no-tcp-cork; } syncer { rate 120M; al-extents 517; } } resource data { on master1 { device /dev/drbd1; disk /dev/sdb1; address 192.168.4.2:7788; meta-disk /dev/sdb2 [0]; } on master2 { device /dev/drbd1; disk /dev/sdb1; address 192.168.4.3:7788; meta-disk /dev/sdb2 [0]; } } (4)、初始化meta分區 [[email protected] ~]# drbdadm create-md data Writing meta data... initializing activity log NOT initialized bitmap New drbd meta data block successfully created. (5)、初始化裝置同步(覆寫備節點,保持資料一緻) [r[email protected] ~]# drbdadm -- --overwrite-data-of-peer primary data (6)、啟動drbd [[email protected] ~]# drbdadm up all [[email protected] ~]# chkconfig drbd off (7)、挂載drbd分區到data資料目錄 [[email protected] ~]# drbdadm primary all [[email protected] ~]# mount /dev/drbd1 /data #說明:/data目錄為資料庫的資料目錄 (8)、測試DRBD 正常狀态 [[email protected] ~]# cat /proc/drbd version: 8.3.13 (api:88/proto:86-96) GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by [email protected], 2012-05-07 11:56:36 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:497984 nr:0 dw:1 dr:498116 al:1 bm:31 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 [[email protected] ~]# cat /proc/drbd version: 8.3.13 (api:88/proto:86-96) GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by [email protected], 2012-05-07 11:56:36 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:0 nr:497984 dw:497984 dr:0 al:0 bm:30 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 #說明:master1為主節點,master2為備節點 模拟master1當機 [[email protected] ~]# umount /dev/drbd1 [[email protected] ~]# drbdadm down all [[email protected] ~]# cat /proc/drbd version: 8.3.13 (api:88/proto:86-96) GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by [email protected], 2012-05-07 11:56:36 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:497985 dw:497985 dr:0 al:0 bm:30 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 [[email protected] ~]# drbdadm primary all [[email protected] ~]# mount /dev/drbd1 /data [[email protected] ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 19G 5.1G 13G 29% / /dev/sda1 190M 18M 163M 10% /boot tmpfs 60M 0 60M 0% /dev/shm /dev/drbd1 471M 11M 437M 3% /data #說明:master1當機後,master2可以更新為主節點,可挂載drbd分區繼續使用 |
4、MySQL 安裝部署 | 注意:三台資料庫都安裝mysql服務,master2隻安裝到makeinstall即可,mysqld服務不要設定為開機自啟動 (1)、解決perl編譯問題 echo 'export LC_ALL=C'>> /etc/profile source /etc/profile (2)、安裝CAMKE cd /home/xu/tools wget http://www.cmake.org/files/v2.8/cmake-2.8.4.tar.gz tar zxf cmake-2.8.4.tar.gz cd cmake-2.8.4 ./configure make & make install (3)、建立使用者 groupadd mysql useradd -g mysql mysql (4)、編譯安裝mysql wget http://mysql.ntu.edu.tw/Downloads/MySQL-5.5/mysql-5.5.27.tar.gz tar zxf mysql-5.5.27.tar.gz cd mysql-5.5.27 cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \ -DMYSQL_UNIX_ADDR=/tmp/mysql.sock \ -DDEFAULT_CHARSET=utf8 \ -DDEFAULT_COLLATION=utf8_general_ci \ -DWITH_EXTRA_CHARSETS=complex \ -DWITH_READLINE=1 \ -DENABLED_LOCAL_INFILE=1 make -j 4 make install (5)、設定mysql環境變量 [[email protected] ~]# echo 'PATH=$PATH:/usr/local/mysql/bin' >>/etc/profile [[email protected] ~]# source /etc/profile (6)、初始化資料庫 [[email protected] ~]# mount /dev/drbd1 /data #說明:資料庫存放資料的目錄是drbd分區 [[email protected] ~]# cd /usr/local/mysql/ [[email protected] ~]# ./scripts/mysql_install_db --datadir=/data/ --user=mysql (7)、啟動資料庫 [[email protected] ~]# vim /etc/init.d/mysqld datadir=/data #說明:修改mysql啟動腳本,指定資料庫的目錄為/data [[email protected] ~]# /etc/init.d/mysqld start [[email protected] ~]# chkconfig mysqld off (8)、測試資料庫 [[email protected] ~]# mysql -uroot -e "show databases;" +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | +--------------------+ |
5、故障切換測試 | (1)、架構正常狀态 master1主節點正常狀态 [[email protected] ~]# ip addr|grep eth1 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 inet 192.168.4.2/16 brd 192.168.255.255 scope global eth1 inet 192.168.4.1/16 brd 192.168.255.255 scope global secondary eth1:0 [[email protected] ~]# cat /proc/drbd version: 8.3.13 (api:88/proto:86-96) GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by [email protected], 2012-05-07 11:56:36 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:39558 nr:12 dw:39570 dr:151 al:16 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 [[email protected] ~]# mysql -uroot -e "create database coral;" [[email protected] ~]# mysql -uroot -e "show databases like 'coral';" +------------------+ | Database (coral) | +------------------+ | coral | +------------------+ #說明:master1為主節點,擁有VIP位址,為drbd的主節點 master2備節點正常狀态 [[email protected] ~]# ip addr|grep eth1 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 inet 192.168.4.3/16 brd 192.168.255.255 scope global eth1 [[email protected] ~]# cat /proc/drbd version: 8.3.13 (api:88/proto:86-96) GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by [email protected], 2012-05-07 11:56:36 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:0 nr:48 dw:48 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 #說明:master2備節點沒有VIP位址,為drbd備節點 (2)、模拟master1當機故障狀态 [[email protected] ~]# /etc/init.d/heartbeat stop <==模拟master1故障當機 [[email protected] ~]# tailf /var/log/ha-log <==檢視備節點接管日志 heartbeat[13209]: 2013/01/23_04:09:36 info: Received shutdown notice from 'master1'. heartbeat[13209]: 2013/01/23_04:09:36 info: Resources being acquired from master1. heartbeat[15293]: 2013/01/23_04:09:36 info: acquire local HA resources (standby). heartbeat[15294]: 2013/01/23_04:09:37 info: No local resources [/usr/share/heartbeat/ResourceManager listkeys master2] to acquire. heartbeat[15293]: 2013/01/23_04:09:37 info: local HA resource acquisition completed (standby). heartbeat[13209]: 2013/01/23_04:09:37 info: Standby resource acquisition done [foreign]. harc[15319]: 2013/01/23_04:09:37 info: Running /etc/ha.d/rc.d/status status mach_down[15335]: 2013/01/23_04:09:37 info: Taking over resource group IPaddr::192.168.4.1/16/eth1 ResourceManager[15361]: 2013/01/23_04:09:37 info: Acquiring resource group: master1 IPaddr::192.168.4.1/16/eth1 drbddisk::data Filesystem::/dev/drbd1::/data::ext3 mysqld IPaddr[15388]: 2013/01/23_04:09:37 INFO: Resource is stopped ResourceManager[15361]: 2013/01/23_04:09:37 info: Running /etc/ha.d/resource.d/IPaddr 192.168.4.1/16/eth1 start IPaddr[15486]: 2013/01/23_04:09:38 INFO: Using calculated netmask for 192.168.4.1: 255.255.0.0 IPaddr[15486]: 2013/01/23_04:09:38 INFO: eval ifconfig eth1:0 192.168.4.1 netmask 255.255.0.0 broadcast 192.168.255.255 IPaddr[15457]: 2013/01/23_04:09:38 INFO: Success ResourceManager[15361]: 2013/01/23_04:09:38 info: Running /etc/ha.d/resource.d/drbddisk data start Filesystem[15636]: 2013/01/23_04:09:39 INFO: Resource is stopped ResourceManager[15361]: 2013/01/23_04:09:39 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext3 start Filesystem[15717]: 2013/01/23_04:09:39 INFO: Running start for /dev/drbd1 on /data Filesystem[15706]: 2013/01/23_04:09:39 INFO: Success ResourceManager[15361]: 2013/01/23_04:09:40 info: Running /etc/init.d/mysqld start mach_down[15335]: 2013/01/23_04:09:44 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired mach_down[15335]: 2013/01/23_04:09:44 info: mach_down takeover complete for node master1. heartbeat[13209]: 2013/01/23_04:09:44 info: mach_down takeover complete. heartbeat[13209]: 2013/01/23_04:10:09 WARN: node master1: is dead heartbeat[13209]: 2013/01/23_04:10:09 info: Dead node master1 gave up resources. heartbeat[13209]: 2013/01/23_04:10:09 info: Link master1:eth2 dead. #說明:當備節點無法檢測到主節點的心跳時,自動接管資源,啟動VIP位址、drbd服務,自動挂載drbd,啟動mysqld服務,備節點接管後,資料依然存在,檢測啟動的服務如下: [[email protected] ~]# ip addr|grep eth1 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 inet 192.168.4.3/16 brd 192.168.255.255 scope global eth1 inet 192.168.4.1/16 brd 192.168.255.255 scope global secondary eth1:0 [[email protected] ~]# cat /proc/drbd version: 8.3.13 (api:88/proto:86-96) GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by [email protected], 2012-05-07 11:56:36 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:3 nr:95 dw:98 dr:10 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 [[email protected] ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 19G 4.7G 14G 26% / /dev/sda1 190M 18M 163M 10% /boot tmpfs 60M 0 60M 0% /dev/shm /dev/drbd1 471M 40M 408M 9% /data [[email protected] ~]# mysql -uroot -e "show databases like 'coral';" +------------------+ | Database (coral) | +------------------+ | coral | +------------------+ (3)、模拟master1當機恢複狀态 啟動的順序是:先啟動VIP--啟動drbd資源--挂載drbd分區--啟動mysqld服務,日志如下: [[email protected] ~]# /etc/init.d/heartbeat start [[email protected] ~]# tailf /var/log/ha-log heartbeat[27970]: 2013/01/09_17:34:14 info: Version 2 support: no heartbeat[27970]: 2013/01/09_17:34:14 WARN: Logging daemon is disabled --enabling logging daemon is recommended heartbeat[27970]: 2013/01/09_17:34:14 info: ************************** heartbeat[27970]: 2013/01/09_17:34:14 info: Configuration validated. Starting heartbeat 2.1.3 heartbeat[27971]: 2013/01/09_17:34:14 info: heartbeat: version 2.1.3 heartbeat[27971]: 2013/01/09_17:34:14 info: Heartbeat generation: 1351554533 heartbeat[27971]: 2013/01/09_17:34:14 info: glib: UDP multicast heartbeat started for group 225.0.0.7 port 694 interface eth2 (ttl=1 loop=0) heartbeat[27971]: 2013/01/09_17:34:14 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[27971]: 2013/01/09_17:34:14 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[27971]: 2013/01/09_17:34:14 info: G_main_add_SignalHandler: Added signal handler for signal 17 heartbeat[27971]: 2013/01/09_17:34:14 info: Local status now set to: 'up' heartbeat[27971]: 2013/01/09_17:34:16 info: Link master2:eth2 up. heartbeat[27971]: 2013/01/09_17:34:16 info: Status update for node master2: status active harc[27978]: 2013/01/09_17:34:16 info: Running /etc/ha.d/rc.d/status status heartbeat[27971]: 2013/01/09_17:34:17 info: Comm_now_up(): updating status to active heartbeat[27971]: 2013/01/09_17:34:17 info: Local status now set to: 'active' heartbeat[27971]: 2013/01/09_17:34:17 info: remote resource transition completed. heartbeat[27971]: 2013/01/09_17:34:17 info: remote resource transition completed. heartbeat[27971]: 2013/01/09_17:34:17 info: Local Resource acquisition completed. (none) heartbeat[27971]: 2013/01/09_17:34:18 info: master2 wants to go standby [foreign] heartbeat[27971]: 2013/01/09_17:34:20 info: standby: acquire [foreign] resources from master2 heartbeat[27997]: 2013/01/09_17:34:20 info: acquire local HA resources (standby). ResourceManager[28010]: 2013/01/09_17:34:20 info: Acquiring resource group: master1 IPaddr::192.168.4.1/16/eth1 drbddisk::data Filesystem::/dev/drbd1::/data::ext3 mysqld IPaddr[28037]: 2013/01/09_17:34:21 INFO: Resource is stopped ResourceManager[28010]: 2013/01/09_17:34:21 info: Running /etc/ha.d/resource.d/IPaddr 192.168.4.1/16/eth1 start IPaddr[28135]: 2013/01/09_17:34:21 INFO: Using calculated netmask for 192.168.4.1: 255.255.0.0 IPaddr[28135]: 2013/01/09_17:34:21 INFO: eval ifconfig eth1:0 192.168.4.1 netmask 255.255.0.0 broadcast 192.168.255.255 IPaddr[28106]: 2013/01/09_17:34:21 INFO: Success ResourceManager[28010]: 2013/01/09_17:34:21 info: Running /etc/ha.d/resource.d/drbddisk data start Filesystem[28286]: 2013/01/09_17:34:21 INFO: Resource is stopped ResourceManager[28010]: 2013/01/09_17:34:21 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext3 start Filesystem[28367]: 2013/01/09_17:34:21 INFO: Running start for /dev/drbd1 on /data Filesystem[28356]: 2013/01/09_17:34:21 INFO: Success ResourceManager[28010]: 2013/01/09_17:34:22 info: Running /etc/init.d/mysqld start heartbeat[27997]: 2013/01/09_17:34:25 info: local HA resource acquisition completed (standby). heartbeat[27971]: 2013/01/09_17:34:25 info: Standby resource acquisition done [foreign]. heartbeat[27971]: 2013/01/09_17:34:25 info: Initial resource acquisition complete (auto_failback) heartbeat[27971]: 2013/01/09_17:34:25 info: remote resource transition completed. 備節點釋放資源順序:停止mysqld服務--解除安裝drbd1分區--設定drbd為備節點--關閉VIP位址,日志如下: [[email protected] ~]# tailf /var/log/ha-log heartbeat[13209]: 2013/01/23_04:26:53 info: Heartbeat restart on node master1 heartbeat[13209]: 2013/01/23_04:26:53 info: Link master1:eth2 up. heartbeat[13209]: 2013/01/23_04:26:53 info: Status update for node master1: status init heartbeat[13209]: 2013/01/23_04:26:53 info: Status update for node master1: status up harc[16151]: 2013/01/23_04:26:53 info: Running /etc/ha.d/rc.d/status status harc[16167]: 2013/01/23_04:26:53 info: Running /etc/ha.d/rc.d/status status heartbeat[13209]: 2013/01/23_04:26:53 info: all clients are now paused heartbeat[13209]: 2013/01/23_04:26:55 info: Status update for node master1: status active harc[16183]: 2013/01/23_04:26:55 info: Running /etc/ha.d/rc.d/status status heartbeat[13209]: 2013/01/23_04:26:55 info: all clients are now resumed heartbeat[13209]: 2013/01/23_04:26:55 info: remote resource transition completed. heartbeat[13209]: 2013/01/23_04:26:55 info: master2 wants to go standby [foreign] heartbeat[13209]: 2013/01/23_04:26:55 info: standby: master1 can take our foreign resources heartbeat[16199]: 2013/01/23_04:26:55 info: give up foreign HA resources (standby). ResourceManager[16212]: 2013/01/23_04:26:55 info: Releasing resource group: master1 IPaddr::192.168.4.1/16/eth1 drbddisk::data Filesystem::/dev/drbd1::/data::ext3 mysqld ResourceManager[16212]: 2013/01/23_04:26:55 info: Running /etc/init.d/mysqld stop ResourceManager[16212]: 2013/01/23_04:26:57 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext3 stop Filesystem[16297]: 2013/01/23_04:26:57 INFO: Running stop for /dev/drbd1 on /data Filesystem[16297]: 2013/01/23_04:26:57 INFO: Trying to unmount /data Filesystem[16297]: 2013/01/23_04:26:57 INFO: unmounted /data successfully Filesystem[16286]: 2013/01/23_04:26:57 INFO: Success ResourceManager[16212]: 2013/01/23_04:26:57 info: Running /etc/ha.d/resource.d/drbddisk data stop ResourceManager[16212]: 2013/01/23_04:26:57 info: Running /etc/ha.d/resource.d/IPaddr 192.168.4.1/16/eth1 stop IPaddr[16445]: 2013/01/23_04:26:58 INFO: ifconfig eth1:0 down IPaddr[16416]: 2013/01/23_04:26:58 INFO: Success heartbeat[16199]: 2013/01/23_04:26:58 info: foreign HA resource release completed (standby). heartbeat[13209]: 2013/01/23_04:26:58 info: Local standby process completed [foreign]. heartbeat[13209]: 2013/01/23_04:27:02 WARN: 1 lost packet(s) for [master1] [15:17] heartbeat[13209]: 2013/01/23_04:27:02 info: remote resource transition completed. heartbeat[13209]: 2013/01/23_04:27:02 info: No pkts missing from master1! heartbeat[13209]: 2013/01/23_04:27:02 info: Other node completed standby takeover of foreign resources. |
6、從庫同VIP同步 | (1)、master配置 1)、設定server-id值并開啟Binlog參數 [[email protected] ~]# vim /etc/my.cnf log-bin=/usr/local/mysql/mysql-bin server-id = 3 [[email protected] ~]# /etc/init.d/mysqld restart #注意:隻有master1有重新開機操作,master2無需重新開機操作,因為備節點的mysql是未啟動狀态,備節點隻有heartbeat才能啟動mysql 2)、授權并建立同步賬戶rep [[email protected] ~]# mysql -uroot -p mysql> GRANT REPLICATION SLAVE ON *.* TO 'rep'@'192.168.4.%' IDENTIFIED BY 'rep'; (2)、slave配置 1)、設定server-id值并關閉binlog設定 [[email protected] ~]# vim /etc/my.cnf #log-bin=mysql-bin server-id = 4 [[email protected] ~]# /etc/init.d/mysqld restart #說明:從庫無需開啟binlog日志功能,除非有需求做級聯複制架構或對mysql增量備份操作才開啟 2)、配置同步參數 [[email protected] ~]# mysql -uroot CHANGE MASTER TO MASTER_HOST='192.168.4.1', MASTER_PORT=3306, MASTER_USER='rep', MASTER_PASSWORD='rep', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=0; 3)、檢查是否主從同步 [[email protected] ~]# mysql -uroot mysql> show slave status\G ... Slave_IO_Running: Yes Slave_SQL_Running: Yes ... (3)、模拟高可用當機切換是否影響從庫同步 1)、主從正常狀态 [[email protected] ~]# mysql -uroot mysql> create database coral1; Query OK, 1 row affected (0.02 sec) [[email protected] ~]# mysql -uroot -e "show slave status\G"|egrep "Slave_IO_Running|Slave_SQL_Running" Slave_IO_Running: Yes Slave_SQL_Running: Yes [[email protected] ~]# mysql -uroot -e "show databases like 'coral%';" +-------------------+ | Database (coral%) | +-------------------+ | coral1 | +-------------------+ 2)、模拟高可用主節點當機 [roo[email protected] ~]# /etc/init.d/heartbeat stop #說明:模拟主節點當機 [[email protected] ~]# ip addr|grep eth1 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 inet 192.168.4.3/16 brd 192.168.255.255 scope global eth1 inet 192.168.4.1/16 brd 192.168.255.255 scope global secondary eth1:0 [[email protected] ~]# mysql -uroot mysql> create database coral2; Query OK, 1 row affected (0.08 sec) #說明:VIP位址已經漂移到master2上面 [[email protected] ~]# mysql -uroot -e "show slave status\G"|egrep "Slave_IO_Running|Slave_SQL_Running" Slave_IO_Running: Yes Slave_SQL_Running: Yes [[email protected] ~]# mysql -uroot -e "show databases like 'coral%'" +-------------------+ | Database (coral%) | +-------------------+ | coral1 | | coral2 | +-------------------+ #注意:高可用主備節點切換過程中,會有一段時間從庫才能連接配接上,大于在60秒内 #說明:此時主從同步是正常的 3)、模拟高可用主節點當機恢複 [[email protected] ~]# /etc/init.d/heartbeat start [[email protected] ~]# mysql -uroot mysql> create database coral3; [[email protected] ~]# mysql -uroot -e "show slave status\G"|egrep "Slave_IO_Running|Slave_SQL_Running" Slave_IO_Running: Yes Slave_SQL_Running: Yes [[email protected] ~]# mysql -uroot -e "show databases like 'coral%'" +-------------------+ | Database (coral%) | +-------------------+ | coral1 | | coral2 | | coral3 | +-------------------+ #說明:高可用主節點故障恢複後也不影響主從庫的同步 |
7、高可用腦裂問題及解決方案 | (1)、導緻裂腦發生的原因 1、高可用伺服器之間心跳鍊路故障,導緻無法互相檢查心跳 2、高可用伺服器上開啟了防火牆,阻擋了心跳檢測 3、高可用伺服器上網卡位址等資訊配置不正常,導緻發送心跳失敗 4、其他服務配置不當等原因,如心跳方式不同,心跳廣播沖突,軟體BUG等 (2)、防止裂腦一些方案 1、加備援線路 2、檢測到裂腦時,強行關閉心跳檢測(遠端關閉主節點,控制電源的電路fence) 3、做好腦裂的監控報警 4、報警後,備節點在接管時設定比較長的時間去接管,給運維人員足夠的時間去處理(人為處理) 5、啟動磁盤鎖,正在服務的一方鎖住磁盤,裂腦發生時,讓對方完全搶不走"共享磁盤資源" 磁盤鎖存在的問題: 使用鎖磁盤會有死鎖的問題,如果占用共享磁盤的一方不主動"解鎖"另一方就永遠得不到共享磁盤,假如伺服器節點突然當機或崩潰,就不可能執行解鎖指令,備節點也就無法接管資源和服務了,有人在HA中設計了智能鎖,正在提供服務的一方隻在發現心跳全部斷開時才會啟用磁盤鎖,平時就不上鎖 |
轉載于:https://blog.51cto.com/20has/1954094