1.方案簡(jiǎn)介
本方案采用Heartbeat雙機熱備軟件來(lái)保證數據庫的高穩定性和連續性，數據的一致性由DRBD這個(gè)工具來(lái)保證。默認情況下只有一臺mysql在工作，當主mysql服務(wù)器出現問(wèn)題后，系統將自動(dòng)切換到備機上繼續提供服務(wù)，當主數據庫修復完畢，又將服務(wù)切回繼續由主mysql提供服務(wù)。
2.方案優(yōu)缺點(diǎn)
優(yōu)點(diǎn)：安全性高、穩定性高、可用性高，出現故障自動(dòng)切換。
缺點(diǎn)：只有一臺服務(wù)器提供服務(wù)，成本相對較高，不方便擴展，可能會(huì )發(fā)生腦裂。
3.軟件介紹
Heartbeat介紹
官方站點(diǎn)：http://linux-ha.org/wiki/Main_Page
heartbeat可以資源(VIP地址及程序服務(wù))從一臺有故障的服務(wù)器快速的轉移到另一臺正常的服務(wù)器提供服務(wù)，heartbeat和keepalived相似，heartbeat可以實(shí)現failover功能，但不能實(shí)現對后端的健康檢查
DRBD介紹
官方站點(diǎn)：http://www.drbd.org/
DRBD(DistributedReplicatedBlockDevice)是一個(gè)基于塊設備級別在遠程服務(wù)器直接同步和鏡像數據的軟件，用軟件實(shí)現的、無(wú)共享的、服務(wù)器之間鏡像塊設備內容的存儲復制解決方案。它可以實(shí)現在網(wǎng)絡(luò )中兩臺服務(wù)器之間基于塊設備級別的實(shí)時(shí)鏡像或同步復制(兩臺服務(wù)器都寫(xiě)入成功)/異步復制(本地服務(wù)器寫(xiě)入成功)，相當于網(wǎng)絡(luò )的RAID1，由于是基于塊設備(磁盤(pán)，LVM邏輯卷)，在文件系統的底層，所以數據復制要比cp命令更快。DRBD已經(jīng)被MySQL官方寫(xiě)入文檔手冊作為推薦的高可用的方案之一
4.方案拓撲
5.方案適用場(chǎng)景：
適用于數據庫訪(fǎng)問(wèn)量不太大，短期內訪(fǎng)問(wèn)量增長(cháng)不會(huì )太快，對數據庫可用性要求非常高的場(chǎng)景。
6.測試環(huán)境介紹（如下所示，均已關(guān)閉防火墻及selinux，生產(chǎn)環(huán)境自行開(kāi)放端口）
主機名 ip 系統 DRBD磁盤(pán) heartbeat版本db-server-01 192.168.0.10 centos6.2 64bit /dev/sda5 3.0.4db-server-02 192.168.0.20 centos6.2 64bit /dev/sda5 3.0.4
7.軟件安裝以及環(huán)境配置
（1）安裝drbd依賴(lài)組件（兩臺機器，安裝以后重啟系統，因為會(huì )升級內核版本，不重啟會(huì )對不上內核版本，有知道不用重啟的童鞋請給我留言^_^）：
yum install -y kernel kernel-devel kernel-headers flex
（2）下載軟件安裝（兩臺機器操作一樣）
wget http://oss.linbit.com/drbd/8.4/drbd-8.4.2.tar.gz
tar xf drbd-8.4.2.tar.gz cd drbd-8.4.2./configure --prefix=/usr/local/drbd --with-kmmake KDIR=/usr/src/kernels/2.6.32-431.11.2.el6.x86_64/ #很多童鞋無(wú)法加載drbd模塊，多半是正在運行的內核版本和新安裝的不相符make installmkdir -p /usr/local/drbd/var/run/drbdcp /usr/local/drbd/etc/rc.d/init.d/drbd /etc/rc.d/init.dchmod 755 /etc/init.d/drbdcd drbdmake cleanmake KDIR=/usr/src/kernels/2.6.32-431.11.2.el6.x86_64/cp drbd.ko /lib/modules/`uname -r`/kernel/lib/modprobe drbd
檢查是否加載了drbd模塊
[root@192.168.0.10 ~]# lsmod | grep drbddrbd 314246 0 libcrc32c 1246 1 drbd[root@192.168.0.10 ~]#
（3）DRBD配置（配置之前需要先使用fdisk對 /dev/sda進(jìn)行分區）
[root@192.168.0.10 ~]# df -HTFilesystem Type Size Used Avail Use% Mounted on/dev/sda2 ext4 19G 2.6G 16G 15% /tmpfs tmpfs 121M 0 121M 0% /dev/shm/dev/sda1 ext4 204M 52M 141M 27% /boot/dev/sda5 ext4 34G 185M 32G 1% /data[root@192.168.0.10 ~]#
我這里兩臺機器之前都已經(jīng)分區了，由于是自己筆記本上的虛擬機，所以懶得加磁盤(pán)了，我直接把 /data/卸載，然后格式化/dev/sda5，我兩臺機器都這樣操作，如果你有空的磁盤(pán)，照樣需要進(jìn)行分區，比如可以將一個(gè)1T的盤(pán)分一個(gè)區就行了。
[root@192.168.0.10 ~]# umount /data/ [root@192.168.0.10 ~]# mkfs.ext4 /dev/sda5mke2fs 1.41.12 (17-May-2010)Filesystem label=OS type: LinuxBlock size=4096 (log=2)Fragment size=4096 (log=2)Stride=0 blocks, Stripe width=0 blocks2048000 inodes, 8185344 blocks409267 blocks (5.00%) reserved for the super userFirst data block=0Maximum filesystem blocks=4294967296250 block groups32768 blocks per group, 32768 fragments per group8192 inodes per groupSuperblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624Writing inode tables: done Creating journal (32768 blocks): doneWriting superblocks and filesystem accounting information: doneThis filesystem will be automatically checked every 28 mounts or180 days, whichever comes first. Use tune2fs -c or -i to override.[root@192.168.0.10 ~]#
[root@192.168.0.10 ~]# fdisk -lDisk /dev/sda: 53.7 GB, 53687091200 bytes255 heads, 63 sectors/track, 6527 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x000eb0ff Device Boot Start End Blocks Id System/dev/sda1 * 1 26 204800 83 LinuxPartition 1 does not end on cylinder boundary./dev/sda2 26 2321 18432000 83 Linux/dev/sda3 2321 2451 1048576 82 Linux swap / Solaris/dev/sda4 2451 6528 32742400 5 Extended/dev/sda5 2451 6528 32741376 83 Linux[root@192.168.0.10 ~]#
我這里還要在/etc/fstab里面注釋一項：
#UUID=33958004-e8a7-4135-844f-707a5537e86a /data ext4 defaults 1 2
否則重啟機器的時(shí)候提示無(wú)法掛載，會(huì )無(wú)法啟動(dòng)的。
修改/etc/hosts文件，兩臺服務(wù)器操作一樣。
192.168.0.10 db-server-01192.168.0.20 db-server-02
drbd配置只需要修改/usr/local/drbd/etc/drbd.d/global_common.conf配置文件即可，修改后如下（兩臺服務(wù)器配置一樣）：
[root@192.168.0.10 ~]# cat /usr/local/drbd/etc/drbd.d/global_common.confglobal { usage-count yes; }common { syncer { rate 30M; } } #同步速率，視帶寬而定 resource r0 { #創(chuàng )建一個(gè)資源，名字叫"r0" protocol C; #選擇的是drbd的C 協(xié)議（數據同步協(xié)議，C為收到數據并寫(xiě)入后返回，確認成功） startup { } disk { on-io-error detach; } net { } on db-server-01 { #設定一個(gè)節點(diǎn)，分別以各自的主機名命名 device /dev/drbd0; #設定資源設備/dev/drbd0 指向實(shí)際的物理分區 /dev/sda5 disk /dev/sda5; address 192.168.0.10:7888; #設定監聽(tīng)地址以及端口 meta-disk internal; } on db-server-02 { device /dev/drbd0; disk /dev/sda5; address 192.168.0.20:7888; meta-disk internal; #internal表示是在同一個(gè)局域網(wǎng)內 }}[root@192.168.0.10 ~]#
（4）DRBD的管理與維護：
創(chuàng )建DRBD資源
配置好drbd以后，就需要使用命令創(chuàng )建配置的drbd資源，使用如下命令（兩臺服務(wù)器操作一樣）：
[root@192.168.0.10 ~]# dd if=/dev/zero of=/dev/sda5 bs=1M count=100 #不這樣做的話(huà)，在創(chuàng )建資源的時(shí)候報錯100+0 records in100+0 records out104857600 bytes (105 MB) copied, 3.34339 s, 31.4 MB/s[root@192.168.0.10 ~]#
[root@192.168.0.10 ~]# drbdadm create-md r0 Writing meta data...initializing activity logNOT initializing bitmapNew drbd meta data block successfully created.success[root@192.168.0.10 ~]#
（5）DRBD的啟動(dòng)與狀態(tài)查看（分別在兩臺服務(wù)器啟動(dòng)）
[root@192.168.0.10 ~]# /etc/init.d/drbd start Starting DRBD resources: [ create res: r0 prepare disk: r0 adjust disk: r0 adjust net: r0].....[root@192.168.0.10 ~]#
[root@192.168.0.20 ~]# /etc/init.d/drbd startStarting DRBD resources: [ create res: r0 prepare disk: r0 adjust disk: r0 adjust net: r0].[root@192.168.0.20 ~]#
查看drbd的狀態(tài)：
[root@192.168.0.10 ~]# /etc/init.d/drbd statusdrbd driver loaded OK; device status:version: 8.4.2 (api:1/proto:86-101)GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@db-server-01, 2014-04-18 21:15:57m:res cs ro ds p mounted fstype0:r0 Connected Secondary/Secondary Inconsistent/Inconsistent C[root@192.168.0.10 ~]#
可以看見(jiàn)都還沒(méi)有主節點(diǎn)。設置當前節點(diǎn)（192.168.0.10）為主節點(diǎn)，并進(jìn)行格式化和掛載 。
drbdadm -- --overwrite-data-of-peer primary allmkfs.ext4 /dev/drbd0mkdir /datamount /dev/drbd0 /data/
在另外一臺服務(wù)器創(chuàng )建掛載目錄，也創(chuàng )建/data
[root@192.168.0.20 ~]# mkdir /data
查看一下drbd的狀態(tài)（可以看見(jiàn)還在同步）：
[root@192.168.0.10 ~]# /etc/init.d/drbd statusdrbd driver loaded OK; device status:version: 8.4.2 (api:1/proto:86-101)GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@db-server-01, 2014-04-18 21:15:57m:res cs ro ds p mounted fstype... sync'ed: 13.7% (27596/31972)M0:r0 SyncSource Primary/Secondary UpToDate/Inconsistent C /data ext4[root@192.168.0.10 ~]#
（6）mysql安裝，我這里為了簡(jiǎn)單直接安裝編譯好的二進(jìn)制軟件包（兩臺服務(wù)器都需要安裝，操作一樣，只是第二臺mysql不需要初始化數據）
注意：兩臺服務(wù)器上的mysql用戶(hù)的uid和gid要一樣。不然切換后會(huì )導致mysql數據目錄的屬主不正確而啟動(dòng)失敗。
[root@192.168.0.10 ~]# wget http://cdn.mysql.com/Downloads/MySQL-5.5/mysql-5.5.37-linux2.6-x86_64.tar.gz
[root@192.168.0.10 ~]# tar xf mysql-5.5.37-linux2.6-x86_64.tar.gz -C /usr/local/[root@192.168.0.10 ~]# cd /usr/local/[root@192.168.0.10 local]# ln -s mysql-5.5.37-linux2.6-x86_64/ mysql[root@192.168.0.10 local]# groupadd mysql[root@192.168.0.10 local]# useradd -r -g mysql mysql[root@192.168.0.10 local]# cd mysql[root@192.168.0.10 mysql]# chown -R mysql .[root@192.168.0.10 mysql]# chgrp -R mysql .[root@192.168.0.10 mysql]# mkdir /data/mysql[root@192.168.0.10 mysql]# chown -R mysql.mysql /data/mysql/[root@192.168.0.10 mysql]# /usr/local/mysql/scripts/mysql_install_db --user=mysql --datadir=/data/mysql/ --basedir=/usr/local/mysql
[root@192.168.0.10 mysql]# chown -R root .[root@192.168.0.10 mysql]# cp support-files/my-medium.cnf /etc/my.cnf[root@192.168.0.10 mysql]# cp support-files/mysql.server /etc/init.d/mysqld[root@192.168.0.10 mysql]# chmod 755 /etc/init.d/mysqld
[root@192.168.0.10 mysql]# egrep 'datadir|basedir' /etc/my.cnf #兩臺服務(wù)器上的mysql配置文件都加入這里的配置 datadir=/data/mysqlbasedir=/usr/local/mysql [root@192.168.0.10 mysql]#
（7）手動(dòng)切換drbd的主從?？戳硗庖慌_服務(wù)器是否有數據（自動(dòng)切換需要使用heartbeat，后面介紹）：
[root@192.168.0.10 ~]# ll /data/total 20drwx------ 2 root root 16384 Apr 18 22:16 lost+founddrwxr-xr-x 5 mysql mysql 4096 Apr 18 23:01 mysql[root@192.168.0.10 ~]#
[root@192.168.0.20 ~]# ll /data/total 0[root@192.168.0.20 ~]#
[root@192.168.0.10 ~]# /etc/init.d/drbd statusdrbd driver loaded OK; device status:version: 8.4.2 (api:1/proto:86-101)GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@db-server-01, 2014-04-18 21:15:57m:res cs ro ds p mounted fstype0:r0 Connected Primary/Secondary UpToDate/UpToDate C /data ext4[root@192.168.0.10 ~]#
可以看見(jiàn)當前服務(wù)器是主，也就是數據在這臺服務(wù)器上，另外一臺服務(wù)器是沒(méi)有數據的。下面進(jìn)行手動(dòng)切換
主切換成從，需要先卸載文件系統，再執行降級為從的命令：
[root@192.168.0.10 ~]# umount /data/[root@192.168.0.10 ~]# drbdadm secondary all
從切換成主，要先執行升級成主的命令然后掛在文件系統：
[root@192.168.0.20 ~]# drbdadm primary all[root@192.168.0.20 ~]# mount /dev/drbd0 /data/[root@192.168.0.20 ~]# ll /data/total 20drwx------ 2 root root 16384 Apr 18 22:16 lost+founddrwxr-xr-x 5 mysql mysql 4096 Apr 18 23:01 mysql[root@192.168.0.20 ~]# /etc/init.d/drbd statusdrbd driver loaded OK; device status:version: 8.4.2 (api:1/proto:86-101)GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@db-server-02, 2014-04-18 21:22:55m:res cs ro ds p mounted fstype0:r0 Connected Primary/Secondary UpToDate/UpToDate C /data ext4[root@192.168.0.20 ~]#
可以看見(jiàn)已經(jīng)成功切換成主，并且mysql初始化數據也存在了。
DRBD腦裂后的處理
當DRBD出現腦裂后，會(huì )導致drbd兩邊的磁盤(pán)數據不一致，在確定要作為從的節點(diǎn)上切換成secondary，并放棄該資源的數據：
drbdadm secondary r0drbdadm -- --discard-my-data connect r0
在要作為primary的節點(diǎn)重新連接secondary（如果這個(gè)節點(diǎn)當前的連接狀態(tài)為WFConnection的話(huà)，可以省略），使用如下命令連接：
drbdadm connect r0
（8）Heartbeat安裝（兩臺服務(wù)器）
需要添加epel源，centos默認自己沒(méi)有該軟件包，當然你可以自己源碼編譯。
rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
yum install heartbeat -y
創(chuàng )建DRBD腳本文件drbddisk:(兩臺服務(wù)器)
注意：
此處是一個(gè)大坑，因為默認yum安裝Heartbeat，不會(huì )在/etc/ha.d/resource.d/創(chuàng )建drbddisk腳本，估計是版本太新了吧。記得前兩年都不會(huì )這樣的。囧。而且也無(wú)法在安裝后從本地其他路徑找到該文件。此處也是因為啟動(dòng)Heartbeat后無(wú)法PING通虛IP，最后通過(guò)查看/var/log/ha-log日志，找到一行ERROR: Cannot locate resource script drbddisk，然后進(jìn)而到/etc/ha.d/resource.d/路徑下發(fā)現竟然沒(méi)有drbddisk腳本，最后在google上找到該代碼，創(chuàng )建該腳本，終于測試通過(guò)：
[root@192.168.0.20 ~]# chmod 755 /etc/ha.d/resource.d/drbddisk [root@192.168.0.20 ~]# cat /etc/ha.d/resource.d/drbddisk #!/bin/bash## This script is inteded to be used as resource script by heartbeat## Copright 2003-2008 LINBIT Information Technologies# Philipp Reisner, Lars Ellenberg####DEFAULTFILE="/etc/default/drbd"DRBDADM="/sbin/drbdadm"if [ -f $DEFAULTFILE ]; then . $DEFAULTFILEfiif [ "$#" -eq 2 ]; then RES="$1" CMD="$2"else RES="all" CMD="$1"fi## EXIT CODES# since this is a "legacy heartbeat R1 resource agent" script,# exit codes actually do not matter that much as long as we conform to# http://wiki.linux-ha.org/HeartbeatResourceAgent# but it does not hurt to conform to lsb init-script exit codes,# where we can.# http://refspecs.linux-foundation.org/LSB_3.1.0/#LSB-Core-generic/LSB-Core-generic/iniscrptact.html####drbd_set_role_from_proc_drbd(){local outif ! test -e /proc/drbd; thenROLE="Unconfigured"returnfidev=$( $DRBDADM sh-dev $RES )minor=${dev#/dev/drbd}if [[ $minor = *[!0-9]* ]] ; then# sh-minor is only supported since drbd 8.3.1minor=$( $DRBDADM sh-minor $RES )fiif [[ -z $minor ]] || [[ $minor = *[!0-9]* ]] ; thenROLE=Unknownreturnfiif out=$(sed -ne "/^ *$minor: cs:/ { s/:/ /g; p; q; }" /proc/drbd); thenset -- $outROLE=${5%/**}: ${ROLE:=Unconfigured} # if it does not show upelseROLE=Unknownfi}case "$CMD" in start)# try several times, in case heartbeat deadtime# was smaller than drbd ping timetry=6while true; do$DRBDADM primary $RES && breaklet "--try" || exit 1 # LSB generic errorsleep 1done;; stop)# heartbeat (haresources mode) will retry failed stop# for a number of times in addition to this internal retry.try=3while true; do$DRBDADM secondary $RES && break# We used to lie here, and pretend success for anything != 11,# to avoid the reboot on failed stop recovery for "simple# config errors" and such. But that is incorrect.# Don't lie to your cluster manager.# And don't do config errors...let --try || exit 1 # LSB generic errorsleep 1done;; status)if [ "$RES" = "all" ]; then echo "A resource name is required for status inquiries." exit 10fiST=$( $DRBDADM role $RES )ROLE=${ST%/**}case $ROLE inPrimary|Secondary|Unconfigured)# expected;;*)# unexpected. whatever...# If we are unsure about the state of a resource, we need to# report it as possibly running, so heartbeat can, after failed# stop, do a recovery by reboot.# drbdsetup may fail for obscure reasons, e.g. if /var/lock/ is# suddenly readonly. So we retry by parsing /proc/drbd.drbd_set_role_from_proc_drbdesaccase $ROLE inPrimary)echo "running (Primary)"exit 0 # LSB status "service is OK";;Secondary|Unconfigured)echo "stopped ($ROLE)"exit 3 # LSB status "service is not running";;*)# NOTE the "running" in below message.# this is a "heartbeat" resource script,# the exit code is _ignored_.echo "cannot determine status, may be running ($ROLE)"exit 4 # LSB status "service status is unknown";;esac;; *)echo "Usage: drbddisk [resource] {start|stop|status}"exit 1;;esacexit 0[root@192.168.0.20 ~]#
（9）heartbeat配置
Hearbeat的配置主要包括三個(gè)配置文件，authkeys，ha.cf和haresources的配置，下面就分別來(lái)看看：
Authkerys的配置（兩臺服務(wù)器配置一樣）
這個(gè)文件用來(lái)配置密碼認證方式，支持3種認證方式，crc，md5和sha1，從左到右安全性越來(lái)越高，消耗的資源也越多。因此如果heartbeat運行在安全的網(wǎng)路之上，比如私網(wǎng)，那么可以將驗證方式設置成crc，master和backup的authkeys配置一樣。我的authkeys文件配置如下：
[root@192.168.0.10 ~]# cat /etc/ha.d/authkeys auth 11 crc[root@192.168.0.10 ~]# chmod 600 /etc/ha.d/authkeys
注意：該文件權限必須是600
ha.cf的配置（兩臺機器稍微有點(diǎn)區別），Primary（192.168.0.10）如下：
[root@192.168.0.10 ~]# cat /etc/ha.d/ha.cf logfile /var/log/ha-log #定義Heartbeat的日志名字及位置 logfacility local0 keepalive 2 #設定心跳(監測)時(shí)間為2秒 deadtime 15 #設定死亡時(shí)間為15秒 ucast eth1 192.168.0.20#采用單播的方式，IP地址指定為對方IP auto_failback off #當Primary機器發(fā)生故障切換到Secondary機器后Primary恢復后是否進(jìn)行切回操作 （最好是我們有需求手動(dòng)進(jìn)行切換）node db-server-01node db-server-02[root@192.168.0.10 ~]#
Secondary（192.168.0.20）如下：
[root@192.168.0.20 ~]# cat /etc/ha.d/ha.cf logfile /var/log/ha-log #定義Heartbeat的日志名字及位置 logfacility local0 keepalive 2 #設定心跳(監測)時(shí)間為2秒 deadtime 15 #設定死亡時(shí)間為15秒 ucast eth1 192.168.0.10#采用單播的方式，IP地址指定為對方IP auto_failback off#當Primary機器發(fā)生故障切換到Secondary機器后Primary恢復后是否進(jìn)行切回操作（一般我們可以看需求，否則不用自動(dòng)切換） node db-server-01node db-server-02[root@192.168.0.20 ~]#
haresources的配置（兩臺機器配置一樣）：
[root@192.168.0.10 ~]# cat /etc/ha.d/haresources db-server-01 IPaddr::192.168.0.88/24/eth1 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext4 mysqld [root@192.168.0.10 ~]#
注：該文件內IPaddr,Filesystem等腳本存放路徑在/etc/ha.d/resource.d/下,也可在該目錄下存放服務(wù)啟動(dòng)腳本（例如：mysqld）,將相同腳本名稱(chēng)添到/etc/ha.d/haresources內容中，從而跟隨heartbeat啟動(dòng)而啟動(dòng)該腳本。
IPaddr::192.168.0.88/24/eth1：用IPaddr腳本配置浮動(dòng)VIP
drbddisk::r0：用drbddisk腳本實(shí)現DRBD主從節點(diǎn)資源組的掛載和卸載
Filesystem::/dev/drbd0::/data::ext4：用Filesystem腳本實(shí)現磁盤(pán)掛載和卸載
（10）heartbeat的管理
配置好heartbeat之后，需要將mysql從自啟動(dòng)服務(wù)器中去掉，因為主heartbeat啟動(dòng)的時(shí)候會(huì )掛載drdb文件系統以及啟動(dòng)mysql，切換的時(shí)候會(huì )將主上的mysql停止并卸載文件系統，從上會(huì )掛載文件系統，并啟動(dòng)mysql。因此需要做如下操作（兩臺服務(wù)器）：
[root@192.168.0.10 ~]# chkconfig mysqld off[root@192.168.0.10 ~]# chkconfig heartbeat off[root@192.168.0.10 ~]# chkconfig drbd off
[root@192.168.0.10 ~]# cat /etc/rc.local #!/bin/sh## This script will be executed *after* all the other init scripts.# You can put your own initialization stuff in here if you don't# want to do the full Sys V style init stuff.touch /var/lock/subsys/localmodprobe drbd #必須先加載模塊，這也是因為將啟動(dòng)命令放在這里的原因/etc/init.d/drbd start/etc/init.d/heartbeat start[root@192.168.0.10 ~]#
到這里heartbeat+drbd+mysql高可用環(huán)境就搭建結束了。接下來(lái)進(jìn)行測試。
高可用測試
（1）在第一臺服務(wù)器上面啟動(dòng)mysql服務(wù)。（192.168.0.10）
[root@192.168.0.10 ~]# /etc/init.d/mysqld startStarting MySQL.The server quit without updating PID file (/[FAILED]ql/db-server-01.pid).[root@192.168.0.10 ~]# ll /data/total 0[root@192.168.0.10 ~]#
怎么回事？/data/下面為空。這里是因為我們在前面已經(jīng)把這個(gè)節點(diǎn)變?yōu)镾econdary
[root@192.168.0.10 ~]# /etc/init.d/drbd statusdrbd driver loaded OK; device status:version: 8.4.2 (api:1/proto:86-101)GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@db-server-01, 2014-04-18 21:15:57m:res cs ro ds p mounted fstype0:r0 Connected Secondary/Primary UpToDate/UpToDate C[root@192.168.0.10 ~]#
我們現在需要手動(dòng)切換回來(lái)。才能啟動(dòng)mysql
[root@192.168.0.20 ~]# umount /data/[root@192.168.0.20 ~]# drbdadm secondary all[root@192.168.0.20 ~]#
[root@192.168.0.10 ~]# drbdadm primary all[root@192.168.0.10 ~]# mount /dev/drbd0 /data/[root@192.168.0.10 ~]# ll /data/total 20drwx------ 2 root root 16384 Apr 18 22:16 lost+founddrwxr-xr-x 5 mysql mysql 4096 Apr 18 23:01 mysql[root@192.168.0.10 ~]# /etc/init.d/drbd statusdrbd driver loaded OK; device status:version: 8.4.2 (api:1/proto:86-101)GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@db-server-01, 2014-04-18 21:15:57m:res cs ro ds p mounted fstype0:r0 Connected Primary/Secondary UpToDate/UpToDate C /data ext4[root@192.168.0.10 ~]#
可以看見(jiàn)已經(jīng)切換回來(lái)了，我們現在可以啟動(dòng)mysql了。
[root@192.168.0.10 ~]# /etc/init.d/mysqld start Starting MySQL....... [ OK ][root@192.168.0.10 ~]# mysqlWelcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 1Server version: 5.5.37-log MySQL Community Server (GPL)Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql>
（2）在兩臺服務(wù)器上面啟動(dòng)heartbeat
[root@192.168.0.10 ~]# /etc/init.d/heartbeat startStarting High-Availability services: INFO: Resource is stoppedDone.[root@192.168.0.10 ~]#
[root@192.168.0.20 ~]# /etc/init.d/heartbeat startStarting High-Availability services: INFO: Resource is stoppedDone.[root@192.168.0.20 ~]#
[root@192.168.0.10 ~]# ip addr | grep eth13: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 192.168.0.10/24 brd 192.168.0.255 scope global eth1 inet 192.168.0.88/24 brd 192.168.0.255 scope global secondary eth1[root@192.168.0.10 ~]#
可以看見(jiàn)虛擬ip192.168.0.88已經(jīng)存在了。說(shuō)明成功了。我們看看heartbeat的日志就能發(fā)現。
[root@192.168.0.10 ~]# tail -n 20 /var/log/ha-log harc(default)[5598]: 2014/04/19_00:25:21 info: Running /etc/ha.d//rc.d/status statusApr 19 00:25:22 db-server-01 heartbeat: [5591]: info: Comm_now_up(): updating status to activeApr 19 00:25:22 db-server-01 heartbeat: [5591]: info: Local status now set to: 'active'Apr 19 00:25:22 db-server-01 heartbeat: [5591]: info: Status update for node db-server-02: status activeharc(default)[5618]: 2014/04/19_00:25:22 info: Running /etc/ha.d//rc.d/status statusApr 19 00:25:33 db-server-01 heartbeat: [5591]: info: remote resource transition completed.Apr 19 00:25:33 db-server-01 heartbeat: [5591]: info: remote resource transition completed.Apr 19 00:25:33 db-server-01 heartbeat: [5591]: info: Initial resource acquisition complete (T_RESOURCES(us))/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.0.88)[5671]: 2014/04/19_00:25:33 INFO: Resource is stoppedApr 19 00:25:33 db-server-01 heartbeat: [5635]: info: Local Resource acquisition completed.harc(default)[5752]: 2014/04/19_00:25:33 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-respip-request-resp(default)[5752]: 2014/04/19_00:25:33 received ip-request-resp IPaddr::192.168.0.88/24/eth1 OK yesResourceManager(default)[5775]: 2014/04/19_00:25:33 info: Acquiring resource group: db-server-01 IPaddr::192.168.0.88/24/eth1 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext4 mysqld/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.0.88)[5803]: 2014/04/19_00:25:33 INFO: Resource is stoppedResourceManager(default)[5775]: 2014/04/19_00:25:33 info: Running /etc/ha.d/resource.d/IPaddr 192.168.0.88/24/eth1 startIPaddr(IPaddr_192.168.0.88)[5926]: 2014/04/19_00:25:34 INFO: Adding inet address 192.168.0.88/24 with broadcast address 192.168.0.255 to device eth1IPaddr(IPaddr_192.168.0.88)[5926]: 2014/04/19_00:25:34 INFO: Bringing device eth1 upIPaddr(IPaddr_192.168.0.88)[5926]: 2014/04/19_00:25:34 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-192.168.0.88 eth1 192.168.0.88 auto not_used not_used/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.0.88)[5900]: 2014/04/19_00:25:34 INFO: Success/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0)[6030]: 2014/04/19_00:25:34 INFO: Running OK[root@192.168.0.10 ~]#
激動(dòng)的時(shí)刻到了，我們測試一下自動(dòng)切換。我們先看看兩臺服務(wù)器的狀態(tài)：
[root@192.168.0.10 ~]# df -HTFilesystem Type Size Used Avail Use% Mounted on/dev/sda2 ext4 19G 3.5G 15G 20% /tmpfs tmpfs 121M 0 121M 0% /dev/shm/dev/sda1 ext4 204M 52M 141M 27% /boot/dev/drbd0 ext4 33G 216M 32G 1% /data[root@192.168.0.10 ~]#
[root@192.168.0.20 ~]# df -HTFilesystem Type Size Used Avail Use% Mounted on/dev/sda2 ext4 19G 4.9G 13G 28% /tmpfs tmpfs 121M 0 121M 0% /dev/shm/dev/sda1 ext4 204M 52M 141M 27% /boot[root@192.168.0.20 ~]#
可以看見(jiàn)掛載在第一臺服務(wù)器。
測試方法：
1.停掉master上的mysqld，看看是否切換(因為heartheat不檢查服務(wù)的可用性，因此需要通過(guò)而外的腳本來(lái)實(shí)現)。
2.停掉master的heartheat看看是否能正常切換。
3.停掉master的網(wǎng)絡(luò )或者直接將master系統shutdown，看看能否正常切換。
4.啟動(dòng)master的heartbeat看看是否能正常切換回來(lái)。
5.重新啟動(dòng)master看看能否切換過(guò)程是否OK。
注意：這里說(shuō)的切換是不是已經(jīng)將mysql停掉、是否卸載了文件系統等等。
我就停止master（192.168.0.10）上的heartbeat來(lái)測試是否會(huì )自動(dòng)切換，這里除了第一條無(wú)法實(shí)現，其他的都可以切換：
[root@192.168.0.10 ~]# /etc/init.d/heartbeat stopStopping High-Availability services: Done.
[root@192.168.0.10 ~]# df -HTFilesystem Type Size Used Avail Use% Mounted on/dev/sda2 ext4 19G 3.5G 15G 20% /tmpfs tmpfs 121M 0 121M 0% /dev/shm/dev/sda1 ext4 204M 52M 141M 27% /boot[root@192.168.0.10 ~]# /etc/init.d/drbd statusdrbd driver loaded OK; device status:version: 8.4.2 (api:1/proto:86-101)GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@db-server-01, 2014-04-18 21:15:57m:res cs ro ds p mounted fstype0:r0 Connected Secondary/Primary UpToDate/UpToDate C[root@192.168.0.10 ~]#
可以看見(jiàn)已經(jīng)切換了，我們看另外一臺機器的情況：
[root@192.168.0.20 ~]# df -HTFilesystem Type Size Used Avail Use% Mounted on/dev/sda2 ext4 19G 4.9G 13G 28% /tmpfs tmpfs 121M 0 121M 0% /dev/shm/dev/sda1 ext4 204M 52M 141M 27% /boot/dev/drbd0 ext4 33G 216M 32G 1% /data[root@192.168.0.20 ~]# netstat -nltp | grep 3306 | grep -v greptcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 5542/mysqld [root@192.168.0.20 ~]#
可以發(fā)現已經(jīng)切換過(guò)來(lái)，mysql也自動(dòng)啟動(dòng)了。之前是沒(méi)有啟動(dòng)的。
[root@192.168.0.20 ~]# ip addr | grep eth13: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 192.168.0.20/24 brd 192.168.0.255 scope global eth1 inet 192.168.0.88/24 brd 192.168.0.255 scope global secondary eth1[root@192.168.0.20 ~]# mysqlWelcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 1Server version: 5.5.37-log MySQL Community Server (GPL)Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql>
可以看見(jiàn)，一切正常呢。如果我們查看日志，就可以看見(jiàn)到底發(fā)生了什么。
[root@192.168.0.20 ~]# tail -n 10 /var/log/ha-log ResourceManager(default)[4768]: 2014/04/19_00:36:42 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /data ext4 startFilesystem(Filesystem_/dev/drbd0)[5131]: 2014/04/19_00:36:42 INFO: Running start for /dev/drbd0 on /data/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0)[5122]: 2014/04/19_00:36:42 INFO: SuccessResourceManager(default)[4768]: 2014/04/19_00:36:43 info: Running /etc/init.d/mysqld startmach_down(default)[4741]: 2014/04/19_00:36:46 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredmach_down(default)[4741]: 2014/04/19_00:36:46 info: mach_down takeover complete for node db-server-01.Apr 19 00:36:46 db-server-02 heartbeat: [4637]: info: mach_down takeover complete.Apr 19 00:36:58 db-server-02 heartbeat: [4637]: WARN: node db-server-01: is deadApr 19 00:36:58 db-server-02 heartbeat: [4637]: info: Dead node db-server-01 gave up resources.Apr 19 00:36:58 db-server-02 heartbeat: [4637]: info: Link db-server-01:eth1 dead.[root@192.168.0.20 ~]#
對于mysqld服務(wù)掛掉的情況無(wú)法實(shí)現自動(dòng)切換，所以需要一個(gè)腳本來(lái)幫助我們完成，我這里有個(gè)簡(jiǎn)單的腳本，能實(shí)現當mysqld服務(wù)不可用時(shí)進(jìn)行自動(dòng)切換，當進(jìn)行切換時(shí)發(fā)送郵件等。該腳本放在主服務(wù)器執行，也就是運行mysqld服務(wù)的服務(wù)器上執行。
[root@192.168.0.20 ~]# cat mysqlmon.sh #!/bin/bashtrap 'echo PROGRAM INTERRUPTED; exit 1' INTusername=rootpassword=123456n=0log='/var/log/mysqlmon.log'while truedo if /usr/local/mysql/bin/mysql -u${username} -p${password} -e "use test" >&/dev/null then echo `date +"%Y-%m-%d %H:%M:%S"` mysqld is alive! >> ${log} n=0 else echo "`date +"%Y-%m-%d %H:%M:%S"` mysqld cannot be connected!" >> ${log} n=$[n + 1] if [ $n -eq 3 ] then /etc/init.d/heartbeat stop echo "`date +"%Y-%m-%d %H:%M:%S"` mysqld switched to backup!" >> ${log} echo "`date +"%Y-%m-%d %H:%M:%S"` mysqld switched to backup" | mutt -s "mysqld switched to backup" saltstack@163.com break fi fi sleep 10done[root@192.168.0.20 ~]#
掛在后臺執行：
[root@192.168.0.10 ~]# nohup mysqlmon.sh &
停止mysqld服務(wù)，看是否進(jìn)行切換以及發(fā)送郵件：
[root@192.168.0.10 ~]# /etc/init.d/mysqld stopShutting down MySQL. [ OK ][root@192.168.0.10 ~]#
[root@192.168.0.20 ~]# df -HTFilesystem Type Size Used Avail Use% Mounted on/dev/sda2 ext4 19G 4.9G 13G 28% /tmpfs tmpfs 121M 0 121M 0% /dev/shm/dev/sda1 ext4 204M 52M 141M 27% /boot/dev/drbd0 ext4 33G 216M 32G 1% /data[root@192.168.0.20 ~]# netstat -nltp | grep 3306tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 13771/mysqld [root@192.168.0.20 ~]#
總結：
搭建還不算復雜，但是也踩了不少坑，比如yum安裝的heartbeat沒(méi)有drbddisk腳本。該方案的優(yōu)點(diǎn)是安全性高、穩定性高、可用性高，出現故障自動(dòng)切換，但是缺點(diǎn)也很明顯，只有一臺服務(wù)器提供服務(wù)，成本相對較高。不方便擴展?？赡軙?huì )發(fā)生腦裂。當mysql服務(wù)掛掉或者不可用的情況下不能進(jìn)行自動(dòng)切換，需要通過(guò)crm模式實(shí)現或者額外的腳本實(shí)現(比如shell腳本監測到master的mysql不可用就將主上的heartbeat停掉，這樣就會(huì )切換到backup中去)。監控也特別重要，可以使用nagios或者zabbix監控。
參考資料：
聯(lián)系客服
