{"id":4,"date":"2012-08-30T08:54:23","date_gmt":"2012-08-30T08:54:23","guid":{"rendered":"https:\/\/www.nicktailor.com\/?p=4"},"modified":"2022-10-21T11:14:11","modified_gmt":"2022-10-21T11:14:11","slug":"using-keepalived-for-mysql-failover","status":"publish","type":"post","link":"https:\/\/nicktailor.com\/tech-blog\/using-keepalived-for-mysql-failover\/","title":{"rendered":"Using Keepalived for Mysql Failover with Master to Master Replication"},"content":{"rendered":"<p>Hello,<\/p>\n<p>I&#8217;m sure many people are wondering how to do this. So I decided to write this up on how I setup the architecture. This also for people who cant use mysql multimaster, because the application they are using doesn&#8217;t &#8220;offically&#8221; support mysql multimaster. If it does, you should look at that avenue before attempting this type of setup \ud83d\ude42<\/p>\n<p>Its pretty awesome, hope it helps you.<\/p>\n<p>I did not do VRRP failover, which is what some people on the net say to do. This would mean that your VIP would be bound to one server and require an ARP for it to failover to the secondary in a failover scenerio, for mysql this is verrrrrry bad especially in a production high traffic site. In my setup the VIP is bound to both DB&#8217;s and will NOT require an ARP for failover.<\/p>\n<p>At the time I wrote this VMAC support was not included in keepalived and it&#8217;s just starting to come out in keepalived, however still in its infancy and I don&#8217;t recommend trying it unless you&#8217;ve tested it thoughorly. Also I did this setup using Virtual Machines in a Production Environment for a University and it worked flawlessly.<\/p>\n<p>Architecture Setup will consist of<\/p>\n<p>&#8211;<em>Two LVS Pair Doing Direct Route<\/em><br \/>\n&#8211;<em>Two MYSQL DB servers doing MASTER TO MASTER replication in fail over scenerio with replication synch and failure protection<\/em><br \/>\n&#8211;<em>Keepalived will be using a custom misc_chk script which does a TCP, PORT, MYSQL SERVICE RUNNING, and TABLE WRITE check. Should this fail 3 times, it fails to the secondary DB.<\/em><br \/>\n<em> -Setting mysql backups as well, you will be able to do backups and restores without breaking replication or stopping mysql.<\/em><\/p>\n<p><strong>DIRECT ROUTE<\/strong><\/p>\n<p><em>&#8211;&nbsp;The virtual IP address is shared by real servers and the load balancer. The load balancer has an interface configured with the virtual IP address too, which is used to accept request packets, and it directly route the packets to the chosen servers. All the real servers have their&nbsp;<strong>non-arp alias<\/strong>&nbsp;interface configured with the virtual IP address or redirect packets destined for the virtual IP address to a local socket, so that the real servers can process the packets locally. The load balancer and the real servers must have one of their interfaces physically linked by a HUB\/Switch. The architecture of virtual server via direct routing is illustrated as follows:<\/em><\/p>\n<p><img decoding=\"async\" src=\"http:\/\/www.linuxvirtualserver.org\/Joseph.Mack\/linuxexpo99\/VS-DR.png\" alt=\"\"\/><\/p>\n<p>&nbsp;<\/p>\n<p>1. Provision yourself some servers using Debian or Ubuntu just for the keepalived stuff. I find that redhat has issues with compiling and its a bit slow. Your DB&#8217;s can be whatever distro you want.<\/p>\n<p>Note. Ensure that you have two nics setup on your VM&#8217;s and that they are on the same vlan&#8217;s. This setup is single armed. If you are Doing NAT you will need two VLANS.<\/p>\n<p><strong>2.<span style=\"text-decoration: underline;\"> COMPILE YOUR KEEPALIVED AND SETUP<\/span><\/strong><\/p>\n<p>If you decided to use debian you will need the following packages to compile from source. I would recommend compiling from source, so you don\u2019t have to worry about updates from the OS repo affecting your environments by accident. Ensure that you have the Kernel Headers installed, as you will need them.<\/p>\n<p><em>apt-get install gcc libssl-dev libpopt-dev libnl-dev ipvsadm<\/em><\/p>\n<p><em>.\/configure &#8211;with-kernel-dir=\/usr\/src\/linux-headers-2.6.35-23-server\/<\/em><\/p>\n<p><em>make &amp;&amp; make install<\/em><\/p>\n<p>Once this installed you will need to make some modification the startup scripts<\/p>\n<p><em>mv \/etc\/init.d\/keepalived \/etc\/init.d\/keepalived.bak<\/em><br \/>\n<em> cd \/etc\/init.d\/<\/em><br \/>\n<em> ln -s \/usr\/local\/etc\/rc.d\/init.d\/keepalived keepalived<\/em><br \/>\n<em> mv \/usr\/sbin\/keepalived \/usr\/sbin\/keepalived.original<\/em><br \/>\n<em> cd \/usr\/sbin<\/em><br \/>\n<em> ln -s \/usr\/local\/sbin\/keepalived keepalived<\/em><\/p>\n<p>Update the<em> \/etc\/init.d\/keepalived<\/em> script<\/p>\n<p>updated<em> \/etc\/init.d\/keepalived<\/em><br \/>\n=================================<br \/>\n#!\/bin\/sh<br \/>\n#<br \/>\n# Startup script for the Keepalived daemon<br \/>\n#<br \/>\n# processname: keepalived<br \/>\n# pidfile: \/var\/run\/keepalived.pid<br \/>\n#config: \/usr\/local\/etc\/keepalived<br \/>\n# chkconfig: &#8211; 21 79<br \/>\n# description: Start and stop Keepalived<br \/>\nPATH=\/sbin:\/bin:\/usr\/sbin:\/usr\/bin<br \/>\nDAEMON=\/usr\/local\/sbin\/keepalived<br \/>\nNAME=keepalived<br \/>\nDESC=keepalived<br \/>\nCONFIG=\/etc\/keepalived\/keepalived.conf<\/p>\n<p># Source function library<br \/>\n#. \/etc\/rc.d\/init.d\/functions<br \/>\n. \/lib\/lsb\/init-functions<\/p>\n<p># Source configuration file (we set KEEPALIVED_OPTIONS there)<\/p>\n<p>#. \/etc\/sysconfig\/keepalived<br \/>\n. \/usr\/local\/etc\/sysconfig\/<br \/>\n====================================================<\/p>\n<p>Next you will need to edit the file below on both lvs servers<\/p>\n<p>\/etc\/sysctl.conf on both LVS1 and LVS2 save the file and run <em>&#8216;sysctl -p<\/em>&#8216; to load it to the running config<br \/>\nnet.ipv4.ip_forward = 1<br \/>\n==========================================<\/p>\n<p>3. Configure your nics on LVS1 and LVS2<br \/>\n\/etc\/network\/interfaces (The ip&#8217;s listed below are example only please dont use them hoping it will work)<\/p>\n<p># This file describes the network interfaces available on your system<br \/>\n# and how to activate them. For more information, see interfaces(5).<br \/>\n# The loopback network interface<br \/>\nauto lo<br \/>\niface lo inet loopback<\/p>\n<p>#This line adds a static route for the entire subnet so traffic can reach the real servers #properly, this line is not&nbsp;necessary, however if the packets are dropping at the destination, #this line helps correct that, it usually better&nbsp;to have this in place.<br \/>\n<em>up ip route add 172.16.0.1\/26 dev eth0;:<\/em><\/p>\n<p># The primary network interface<br \/>\nauto eth0 eth1<br \/>\niface eth0 inet static<br \/>\naddress 172.16.0.1<br \/>\nnetmask 255.255.255.192<br \/>\nnetwork 172.16.0.1<br \/>\nbroadcast 172.16.0.1<br \/>\ngateway 172.16.0.1<\/p>\n<p># dns-* options are implemented by the resolvconf package, if installed<br \/>\ndns-nameservers 172.16.0.1 172.16.0.1<br \/>\ndns-search sa.it.nicktailor.com<\/p>\n<p>iface eth1 inet static<br \/>\n<em>#This interface is on the same vlan as the primary eth0 interface on purpose. This i#nterface is soley used by keepalived as the sync interface \/ hearbeat for keepalived #daemon to sync to the second director lvs2.<\/em><\/p>\n<p>address 172.16.98.2<br \/>\n<em>#This ip should be an non routable address you define, just make sure that lvs2 is on the #same subnet as this non routeble address, so if this is lvs1 then lvs2 will be 172.16.98.3 #kind of thing.<br \/>\n<\/em>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; netmask 255.255.255.0<br \/>\n=============================================================<\/p>\n<p>4. Now what I have done is setup a reload script that will update LVS2 upon reload so I dont have to keep updating both LVS server everytime I have to do a graceful reload, not sure why keepalive has not implemented this feature yet. Developer, don&#8217;t think like systems engineers and vice versa.<\/p>\n<ul>\n<li>. setup ssh keys on both server so that root can ssh to lvs2 via ssh key<\/li>\n<li>Login into destination server<\/li>\n<li>cd \/root\/.ssh<\/li>\n<li>run &#8216;<em>ssh-keygen -t rsa&#8217;<\/em><\/li>\n<li><em><span style=\"text-align: justify;\">run &#8216;cat id_rsa.pub &gt; authorized_keys &amp;&amp; chmod 600 authorized_keys&#8217;<\/span><\/em><\/li>\n<li>Login into to lvs1<\/li>\n<li>cd \/root\/.ssh<\/li>\n<li>copy both id_rsa fils into this directory from lvs2<\/li>\n<li><em>run &#8216;cat id_rsa.pub &gt; authorized_keys &amp;&amp; chmod 600 authorized_keys&#8217;<\/em><\/li>\n<\/ul>\n<p>You should be able to ssh now &#8216;<span style=\"text-decoration: underline;\">ssh root@hostname<\/span>&#8216; and it should login. If it didn&#8217;t go google why \ud83d\ude1b<\/p>\n<p>5. A) Setup keepalived and automating the reload process<\/p>\n<p>Login into LVS1<br \/>\ncd \/etc\/keepalived\/<br \/>\nmkdir conf.d\/<br \/>\nmkdir backups\/<\/p>\n<p>Create a keepalived.conf in \/etc\/keepalived\/ and copy past the following in it, you will need to update the varibles according to your own setup<\/p>\n<p>sample keepalived.conf<br \/>\nhttps:\/\/www.nicktailor.com\/files\/keepalived.conf.lvs1<br \/>\nhttps:\/\/www.nicktailor.com\/files\/keepalived.conf.lvs2<br \/>\n##WARNING:<br \/>\n#If this configuration is copied over to the SLAVE LVS as is stuff will EXPLODE<br \/>\n### There are elements within that need to stay unique on each server.<br \/>\n### Specifically:<br \/>\n### -lvs id<br \/>\n### -state<br \/>\n### -priority<br \/>\n#########################################################################<\/p>\n<p><em>global_defs {<\/em><br \/>\n<em> notification_email {<\/em><br \/>\n<em> nick.tailor@nicktailor.com<\/em><br \/>\n<em> }<\/em><br \/>\n<em> notification_email_from root@lvs1<\/em><br \/>\n<em> smtp_server 127.0.0.1<\/em><br \/>\n<em> smtp_connect_timeout 30<\/em><br \/>\n<em> lvs_id lvs1<\/em><br \/>\n<em> }<\/em><\/p>\n<p><em>vrrp_instance external_linux {<\/em><br \/>\n<em> state MASTER<\/em><br \/>\n<em> smtp_alert<\/em><br \/>\n<em> interface eth0<\/em><br \/>\n<em> lvs_sync_daemon_interface eth0<\/em><br \/>\n<em> virtual_router_id 42<\/em><br \/>\n<em> priority 150<\/em><br \/>\n<em> advert_int 1<\/em><br \/>\n<em> preempt<\/em><br \/>\n<em> authentication {<\/em><br \/>\n<em> auth_type PASS<\/em><br \/>\n<em> auth_pass nicktailor<\/em><br \/>\n<em> }<\/em><br \/>\n<em> virtual_ipaddress {<\/em><br \/>\n<em> #VIP&#8217;s are listed and or Gateway ip is listed here if using NAT for DB boxes<\/em><br \/>\n<em> 172.16.10.1<\/em><br \/>\n<em> }<\/em><br \/>\n<em> virtual_server 172.16.10.1 3306 {<\/em><br \/>\n<em> delay_loop 10<\/em><br \/>\n<em> lb_algo rr<\/em><br \/>\n<em> lb_kind DR<\/em><br \/>\n<em> protocol TCP<\/em><br \/>\n<em> # persistence_timeout 30 (this is not needed for db with replication setup,<\/em><br \/>\n<em> # only enabled if you absolutely have to)<\/em><\/p>\n<p><em>#When removing servers from the pool comment starting below this line<\/em><br \/>\n<em> real_server 142.103.18.2 3306 {<\/em><br \/>\n<em> MISC_CHECK {<\/em><br \/>\n<em> misc_path &#8220;\/root\/keepalived\/dbcheck.sh 142.103.18.2&#8221;<\/em><br \/>\n<em> misc_timeout 30<br \/>\n<\/em><em>}<\/em><\/p>\n<p><em> }<\/em><br \/>\n<em> #make sure to comment above bracket when removing above server from pool<\/em><br \/>\n<em> real_server 142.103.18.3 3306 {<\/em><br \/>\n<em> MISC_CHECK {<\/em><br \/>\n<em> misc_path &#8220;\/root\/keepalived\/dbcheck.sh 142.103.18.2&#8221;<\/em><br \/>\n<em> misc_timeout 30<\/em><br \/>\n<em> }<\/em><br \/>\n<em> }<\/em><br \/>\n<em> #make sure to comment above last bracket when removing above server from pool, do not comment out below bracket<\/em><br \/>\n<em> }<\/em><br \/>\n================================================================<\/p>\n<p>I have broken the keepalived.conf file into three sections for the purpose of automating &amp; config updates to LVS2<\/p>\n<p><strong><em>Create<\/em><\/strong><br \/>\n<em>\/etc\/keepalived\/head<\/em> &lt;- Contains the Global LVS directives for the directors(director means LVS1 or LVS2) there are links below for example files.<\/p>\n<p>==================<br \/>\nWarning: Directives within the GLOBAL section MUST differ on each server in a failover setup!!!<\/p>\n<p><strong>Global-Directives<\/strong><br \/>\n<strong> * Unique Directives:<\/strong><\/p>\n<ul>\n<li>o lvs_id &#8211; { name of the LVS, usually lvs1-domain)<\/li>\n<li>o state &#8211; { MASTER or BACKUP depends on what is role of this server }<\/li>\n<li>o priority &#8211; { This value must be higher on master for example 100 on MASTER and 50 on BACKUP }<\/li>\n<\/ul>\n<p><strong> * Shared Directives<\/strong><\/p>\n<ul>\n<li>o virtual_router_id &#8211; { this should be differnet for each LVS configuration, tcpdump -n -p 112, will show you which ones are in use }<\/li>\n<li>o vrrp_instance &#8211; { name of instance, usually domain, or domain_eth0 }<\/li>\n<\/ul>\n<p>Example \/etc\/keepalived\/head (whatever you put in your keepalived.conf for the globaldefs put in your head file copy past it here is a sample ones.<\/p>\n<p>http:\/\/www.nikktailor.com\/files\/headfilelvs1<br \/>\nhttps:\/\/www.nicktailor.com\/files\/headfilelvs2<br \/>\n====================<br \/>\n<em>global_defs {<\/em><br \/>\n<em> notification_email {<\/em><br \/>\n<em> nick.tailor@nicktailor.com<\/em><br \/>\n<em> }<\/em><br \/>\n<em> notification_email_from root@lvs1.nicktailor.com<\/em><br \/>\n<em> smtp_server 127.0.0.1<\/em><br \/>\n<em> smtp_connect_timeout 30<\/em><br \/>\n<em> lvs_id lvs1<\/em><br \/>\n<em> }<\/em><br \/>\n<em> vrrp_instance external_linux {<\/em><br \/>\n<em> state BACKUP<\/em><br \/>\n<em> smtp_alert<\/em><br \/>\n<em> interface eth0<\/em><br \/>\n<em> lvs_sync_daemon_interface eth1<\/em><br \/>\n<em> virtual_router_id 42<\/em><br \/>\n<em> priority 50<\/em><br \/>\n<em> advert_int 1<\/em><br \/>\n<em> preempt<\/em><br \/>\n<em> authentication {<\/em><br \/>\n<em> auth_type PASS<\/em><br \/>\n<em> auth_pass nicktailor<\/em><br \/>\n<em> }<\/em><br \/>\n===================<\/p>\n<p>\/etc\/keepalived\/virtual_ips.conf &lt;- Contains the VIP ip&#8217;s, whatever you have in your keepalived.conf create this file and copy paste it.<\/p>\n<p>https:\/\/www.nicktailor.com\/files\/virtual_ips.conf<br \/>\n================================<br \/>\n<em>virtual_ipaddress {<\/em><\/p>\n<p><em>#VIP&#8217;s are listed and or Gateway ip is listed here if using NAT for DB boxes<\/em><br \/>\n<em> 172.16.10.1 #Development VIP<\/em><br \/>\n<em> 172.16.10.2 #Production VIP<\/em><br \/>\n<em> }<\/em><br \/>\n================================<\/p>\n<p>inside the conf.d\/ directory you will have the real server config. Make sure you <strong>never have duplicates<\/strong> as the reload script will mess up your keepalived.conf file if do and you can bring down your VIPS<\/p>\n<p>https:\/\/www.nicktailor.com\/files\/testmschk.conf (Example)<br \/>\n\/etc\/keepalived\/conf.d\/testmschkscript.conf<br \/>\n==================================================<br \/>\n<em>#reload config with &#8216;\/etc\/init.d\/keepalive_reload&#8217; this will update lvs2 with changes automatically<\/em><\/p>\n<p><em> #DO NOT RELOAD CONFIG FROM LVS2<\/em><br \/>\n<em> #To see pool use &#8216;watch ipvsadm -L -n&#8217;<\/em><br \/>\n<em> #&#8217;ip addr&#8217; will show you which lvs the vip is bound to<\/em><\/p>\n<p><em>virtual_server 172.16.10.1 3306 {<\/em><br \/>\n<em> delay_loop 10<\/em><br \/>\n<em> lb_algo rr<\/em><br \/>\n<em> lb_kind DR<\/em><br \/>\n<em> protocol TCP<\/em><br \/>\n<em> # persistence_timeout 30 (this is not needed for db with replication setup, only enabled if you absolutely have to)<\/em><\/p>\n<p><em>#this is the failover server if it fails over to this traffic is forced there are no checks done on this server by keepalived<\/em><br \/>\n<em> sorry_server 142.103.18.2 3306<\/em><\/p>\n<p><em>#When removing servers from the pool comment starting below this line<\/em><br \/>\n<em> real_server 142.103.18.1 3306 {<\/em><br \/>\n<em> MISC_CHECK {<\/em><br \/>\n<em> misc_path &#8220;\/root\/keepalived\/dbcheck.sh 142.103.18.1&#8221;<\/em><br \/>\n<em> misc_timeout 30<\/em><\/p>\n<p><em>}<\/em><br \/>\n<em> }<\/em><\/p>\n<p><em>#make sure to comment above last bracket when removing above server from pool, do not comment out below bracket<\/em><br \/>\n<em> }<\/em><br \/>\n================================================================<\/p>\n<p><strong>B) Automation Reload Script<\/strong><br \/>\ncreate the following file \/etc\/init.d\/keepalivedreload and give it executable permissions.<br \/>\ncreate \/etc\/keepalived\/backups directory as well.<\/p>\n<p>\/etc\/init.d\/keepalivedreload (LVS1)<br \/>\nhttps:\/\/www.nicktailor.com\/files\/keepalivedreloadlvs1<\/p>\n<p>a) this will copy the virtual ips file and conf.d directory to lvs2 and reload the config lvs2 and make a backup of the config prior to update.<br \/>\n===============================<br \/>\n#!\/bin\/bash<\/p>\n<p>KADIR=\/etc\/keepalived<br \/>\nBACKUPFILE=&#8221;$KADIR\/backups\/`date +&#8217;%Y%m%d-%H%M&#8217;`.conf&#8221;<br \/>\nCONFFILE=$KADIR\/keepalived.conf<\/p>\n<p>echo Making a backup of the conf file as $BACKUPFILE.gz<br \/>\ncp $KADIR\/keepalived.conf $BACKUPFILE<br \/>\ngzip $BACKUPFILE<\/p>\n<p>echo &#8220;Adding gateways and virtual IPs&#8230;&#8221;<\/p>\n<p>cat $KADIR\/head $KADIR\/virtual_ips.conf &gt; $CONFFILE<br \/>\necho &#8220;}&#8221; &gt;&gt; $CONFFILE<br \/>\n#echo &#8220;}&#8221; &gt;&gt; $CONFFILE<br \/>\necho &#8220;##################################################&#8221; &gt;&gt; $CONFFILE<\/p>\n<p>echo &#8220;Adding virtual servers&#8230;&#8221;<\/p>\n<p>for i in $KADIR\/conf.d\/*<br \/>\ndo<br \/>\ncat $i &gt;&gt; $CONFFILE<br \/>\ndone<\/p>\n<p>echo &#8220;Copying gateways, virtual IPs, and virtual servers to lvs2nicktailor.com&#8230;&#8221;<br \/>\nscp $KADIR\/virtual_ips.conf root@lvs2nicktailor.com:\/etc\/keepalived\/<\/p>\n<p>for i in $KADIR\/conf.d\/*<br \/>\ndo<br \/>\nscp $i root@lvs2nicktailor.com:\/etc\/keepalived\/conf.d\/<br \/>\ndone<\/p>\n<p>echo &#8220;Reloading keepalived config&#8230;&#8221;<br \/>\n\/etc\/init.d\/keepalived reload<\/p>\n<p>echo &#8220;Issuing reload on lvs2nicktailor.com&#8230;&#8221;<br \/>\necho &#8220;==============================&#8221;<br \/>\nssh root@lvs2nicktailor.com \/etc\/init.d\/keepalived_reload<br \/>\necho &#8220;Done.&#8221;<br \/>\n============================================<\/p>\n<p>Now I have made the second one so that you can&#8217;t make reloads updates to the primary lvs from the secondary, just because of human error. You&#8217;re free to do this however you please. I found this way to be beneficial.<\/p>\n<p>&nbsp;<\/p>\n<p>(LVS2) \/etc\/init.d\/keepalivedreload this script will just make a backup and reload on lvs2 and not update back to lvs1.<\/p>\n<p>https:\/\/www.nicktailor.com\/files\/keepalivedreloadlvs2<br \/>\n====================================<br \/>\n#!\/bin\/bash<\/p>\n<p>KADIR=\/etc\/keepalived<br \/>\nBACKUPFILE=&#8221;$KADIR\/backups\/`date +&#8217;%Y%m%d-%H%M&#8217;`.conf&#8221;<br \/>\nCONFFILE=$KADIR\/keepalived.conf<\/p>\n<p>echo Making a backup of the conf file as $BACKUPFILE.gz<br \/>\ncp $KADIR\/keepalived.conf $BACKUPFILE<br \/>\ngzip $BACKUPFILE<\/p>\n<p>echo &#8220;Adding gateways and virtual IPs&#8230;&#8221;<\/p>\n<p>cat $KADIR\/head $KADIR\/virtual_ips.conf &gt; $CONFFILE<br \/>\necho &#8220;}&#8221; &gt;&gt; $CONFFILE<br \/>\n#echo &#8220;}&#8221; &gt;&gt; $CONFFILE<br \/>\necho &#8220;##################################################&#8221; &gt;&gt; $CONFFILE<\/p>\n<p>echo &#8220;Adding virtual servers&#8230;&#8221;<\/p>\n<p>for i in $KADIR\/conf.d\/*<br \/>\ndo<br \/>\ncat $i &gt;&gt; $CONFFILE<br \/>\ndone<\/p>\n<p>#echo &#8220;Copying gateways, virtual IPs, and virtual servers to lvsdevl22.webi.it.ubc.ca&#8230;&#8221;<br \/>\n#scp $KADIR\/virtual_ips.conf<br \/>\n#root@lvs2nicktailor.com:\/etc\/keepalived\/<\/p>\n<p>#for i in $KADIR\/conf.d\/*<br \/>\n#do<br \/>\n#scp $i root@lvs2nicktailor.com:\/etc\/keepalived\/conf.d\/<br \/>\n#done<\/p>\n<p>echo &#8220;Reloading keepalived config&#8230;&#8221;<br \/>\n\/etc\/init.d\/keepalived reload<\/p>\n<p>#echo &#8220;Issuing reload on lvs2nicktailor.com&#8230;&#8221;<br \/>\n#echo &#8220;==============================&#8221;<br \/>\n#ssh root@lvs2nicktailor.com \/root\/bin\/keepalive_reload<br \/>\n#ssh root@lvs2nicktailor.com \/etc\/init.d\/keepalived reload<br \/>\necho &#8220;Done.&#8221;<\/p>\n<p>6. Test the reload by running it and seeing if the upates take effect from lvs1 to lvs2. You should see the keepalived.conf and \/etc\/keepalived\/conf.d directory all matching.<\/p>\n<p><span style=\"text-decoration: underline;\"><strong>MYSQL DATABASE SERVER DEPLOYMENT AND REPLICATION SETUP<\/strong><\/span><\/p>\n<p>1. Provision your mysql DB servers instal mysql<br \/>\n2. Ensure that your DB servers are on the same vlan as the LVS<br \/>\n3. Configure the nics<br \/>\n4. configure the loopback interface so the VIP is bound to it, since we are doing Direct ROUTE<\/p>\n<p>IE. redhat<\/p>\n<p>\/etc\/sysconfig\/network-scripts\/ifcfg-lo:0<br \/>\n=================<br \/>\nDEVICE=lo:0<br \/>\nIPADDR=142.103.18.1<br \/>\nNETMASK=255.255.255.255<br \/>\n# If you&#8217;re having problems with gated making 127.0.0.0\/8 a martian,<br \/>\n# you can change this to something else (255.255.255.255, for example)<br \/>\nONBOOT=yes<br \/>\nNAME=loopback<\/p>\n<p>5. Update \/etc\/sysctl.conf<\/p>\n<p><em>net.ipv4.conf.default.arp_ignore=1<\/em><br \/>\n<em> net.ipv4.conf.default.arp_announce=2<\/em><br \/>\n<em> net.ipv4.conf.all.arp_ignore=1<\/em><br \/>\n<em> net.ipv4.conf.all.arp_announce=2<\/em><\/p>\n<p>and run &#8216;<em>sysctl -p<\/em>&#8216;<\/p>\n<p>Note: The reason for this is it will make sure the server doesn\u2019t loop back and start ARP requests for the VIP.<br \/>\nMYSQL REPLICATION SETUP &#8211; i was too lazy to write it all out so I found his great link to explain how to set it up<br \/>\nhttp:\/\/www.howtoforge.com\/mysql_master_master_replication<br \/>\n=============================================<br \/>\nMaster 1\/Slave 2 ip: 192.168.16.4<\/p>\n<p>Master 2\/Slave 1 ip : 192.168.16.5<\/p>\n<p>&nbsp;<\/p>\n<p>Step 2:<br \/>\nOn Master 1, make changes in my.cnf:<\/p>\n<p>[mysqld]<br \/>\ndatadir=\/var\/lib\/mysql<br \/>\nsocket=\/var\/lib\/mysql\/mysql.sock<br \/>\nold_passwords=1<\/p>\n<p>log-bin<br \/>\nbinlog-do-db=&lt;database name&gt; # input the database which should be replicated<br \/>\nbinlog-ignore-db=mysql # input the database that should be ignored for replication<br \/>\nbinlog-ignore-db=test<\/p>\n<p>server-id=1<\/p>\n<p>[mysql.server]<br \/>\nuser=mysql<br \/>\nbasedir=\/var\/lib<br \/>\n[mysqld_safe]<br \/>\nerr-log=\/var\/log\/mysqld.log<br \/>\npid-file=\/var\/run\/mysqld\/mysqld.pid<br \/>\nStep 3:<br \/>\nOn master 1, create a replication slave account in mysql.<\/p>\n<p>mysql&gt; <em>grant replication slave on *.* to &#8216;replication&#8217;@192.168.16.5 \\<\/em><br \/>\n<em> identified by &#8216;slave&#8217;;<\/em><\/p>\n<p>and restart the mysql master1.<\/p>\n<p>&nbsp;<\/p>\n<p>Step 4:<br \/>\nNow edit my.cnf on Slave1 or Master2 :<\/p>\n<p>[mysqld]<br \/>\ndatadir=\/var\/lib\/mysql<br \/>\nsocket=\/var\/lib\/mysql\/mysql.sock<br \/>\nold_passwords=1<\/p>\n<p>server-id=2<\/p>\n<p>master-host = 192.168.16.4<br \/>\nmaster-user = replication<br \/>\nmaster-password = slave<br \/>\nmaster-port = 3306<\/p>\n<p>[mysql.server]<br \/>\nuser=mysql<br \/>\nbasedir=\/var\/lib<\/p>\n<p>[mysqld_safe]<br \/>\nerr-log=\/var\/log\/mysqld.log<br \/>\npid-file=\/var\/run\/mysqld\/mysqld.pid<br \/>\nStep 5:<br \/>\nRestart mysql slave 1 and at<\/p>\n<p>mysql&gt; <em>start slave;<\/em><br \/>\nmysql&gt; <em>show slave status\\G;<\/em><br \/>\n*************************** 1. row ***************************<\/p>\n<p>Slave_IO_State: Waiting for master to send event<br \/>\nMaster_Host: 192.168.16.4<br \/>\nMaster_User: replica<br \/>\nMaster_Port: 3306<br \/>\nConnect_Retry: 60<br \/>\nMaster_Log_File: MASTERMYSQL01-bin.000009<br \/>\nRead_Master_Log_Pos: 4<br \/>\nRelay_Log_File: MASTERMYSQL02-relay-bin.000015<br \/>\nRelay_Log_Pos: 3630<br \/>\nRelay_Master_Log_File: MASTERMYSQL01-bin.000009<br \/>\nSlave_IO_Running: Yes<br \/>\nSlave_SQL_Running: Yes<br \/>\nReplicate_Do_DB:<br \/>\nReplicate_Ignore_DB:<br \/>\nReplicate_Do_Table:<br \/>\nReplicate_Ignore_Table:<br \/>\nReplicate_Wild_Do_Table:<br \/>\nReplicate_Wild_Ignore_Table:<br \/>\nLast_Errno: 0<br \/>\nLast_Error:<br \/>\nSkip_Counter: 0<br \/>\nExec_Master_Log_Pos: 4<br \/>\nRelay_Log_Space: 3630<br \/>\nUntil_Condition: None<br \/>\nUntil_Log_File:<br \/>\nUntil_Log_Pos: 0<br \/>\nMaster_SSL_Allowed: No<br \/>\nMaster_SSL_CA_File:<br \/>\nMaster_SSL_CA_Path:<br \/>\nMaster_SSL_Cert:<br \/>\nMaster_SSL_Cipher:<br \/>\nMaster_SSL_Key:<br \/>\nSeconds_Behind_Master: 1519187<br \/>\n1 row in set (0.00 sec)<br \/>\nAbove highlighted rows must be indicate related log files and Slave_IO_Running and Slave_SQL_Running: must be to YES.<\/p>\n<p>&nbsp;<\/p>\n<p>Step 6:<br \/>\nOn master 1:<\/p>\n<p>mysql&gt; <em>show master status;<\/em><br \/>\n+&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;+&#8212;&#8212;&#8212;-+&#8212;&#8212;&#8212;&#8212;&#8211;+&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;+<br \/>\n| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |<br \/>\n+&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;+&#8212;&#8212;&#8212;-+&#8212;&#8212;&#8212;&#8212;&#8211;+&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;+<br \/>\n|MysqlMYSQL01-bin.000008 | 410 | adam | |<br \/>\n+&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;+&#8212;&#8212;&#8212;-+&#8212;&#8212;&#8212;&#8212;&#8211;+&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;+<br \/>\n1 row in set (0.00 sec)<\/p>\n<p>The above scenario is for master-slave, now we will create a slave master scenario for the same systems and it will work as master master.<\/p>\n<p>&nbsp;<\/p>\n<p>Step 7:<br \/>\nOn Master2\/Slave 1, edit my.cnf and master entries into it:<\/p>\n<p>[mysqld]<br \/>\ndatadir=\/var\/lib\/mysql<br \/>\nsocket=\/var\/lib\/mysql\/mysql.sock<br \/>\n# Default to using old password format for compatibility with mysql 3.x<br \/>\n# clients (those using the mysqlclient10 compatibility package).<br \/>\nold_passwords=1<br \/>\nserver-id=2<\/p>\n<p>master-host = 192.168.16.4<br \/>\nmaster-user = replication<br \/>\nmaster-password = slave<br \/>\nmaster-port = 3306<\/p>\n<p>log-bin #information for becoming master added<br \/>\nbinlog-do-db=adam<\/p>\n<p>[mysql.server]<br \/>\nuser=mysql<br \/>\nbasedir=\/var\/lib<\/p>\n<p>[mysqld_safe]<br \/>\nerr-log=\/var\/log\/mysqld.log<br \/>\npid-file=\/var\/run\/mysqld\/mysqld.pid<\/p>\n<p>&nbsp;<\/p>\n<p>Step 8:<br \/>\nCreate a replication slave account on master2 for master1:<\/p>\n<p>mysql&gt; <em>grant replication slave on *.* to &#8216;replication&#8217;@192.168.16.4 identified by &#8216;slave2&#8217;;<\/em><\/p>\n<p>&nbsp;<\/p>\n<p>Step 9:<br \/>\nEdit my.cnf on master1 for information of its master.<\/p>\n<p>[mysqld]<br \/>\ndatadir=\/var\/lib\/mysql<br \/>\nsocket=\/var\/lib\/mysql\/mysql.sock<\/p>\n<p># Default to using old password format for compatibility with mysql 3.x<br \/>\n# clients (those using the mysqlclient10 compatibility package).<br \/>\nold_passwords=1<br \/>\nlog-bin<br \/>\nbinlog-do-db=adam<br \/>\nbinlog-ignore-db=mysql<br \/>\nbinlog-ignore-db=test<\/p>\n<p>server-id=1<br \/>\n#information for becoming slave.<br \/>\nmaster-host = 192.168.16.5<br \/>\nmaster-user = replication<br \/>\nmaster-password = slave2<br \/>\nmaster-port = 3306<\/p>\n<p>[mysql.server]user=mysqlbasedir=\/var\/lib<\/p>\n<p>&nbsp;<\/p>\n<p>Step 10:<br \/>\nRestart both mysql master1 and master2.<\/p>\n<p>On mysql master1:<\/p>\n<p>mysql&gt;<em> start slave;<\/em><\/p>\n<p>On mysql master2:<\/p>\n<p>mysql &gt; <em>show master status;<\/em><\/p>\n<p>On mysql master 1:<\/p>\n<p>mysql&gt; <em>show slave status\\G;<\/em><br \/>\n*************************** 1. row ***************************<br \/>\nSlave_IO_State: Waiting for master to send event<br \/>\nMaster_Host: 192.168.16.5<br \/>\nMaster_User: replica<br \/>\nMaster_Port: 3306<br \/>\nConnect_Retry: 60<br \/>\nMaster_Log_File: Mysql1MYSQL02-bin.000008<br \/>\nRead_Master_Log_Pos: 410<br \/>\nRelay_Log_File: Mysql1MYSQL01-relay-bin.000008<br \/>\nRelay_Log_Pos: 445<br \/>\nRelay_Master_Log_File: Mysql1MYSQL02-bin.000008<br \/>\nSlave_IO_Running: Yes<br \/>\nSlave_SQL_Running: Yes<br \/>\nReplicate_Do_DB:<br \/>\nReplicate_Ignore_DB:<br \/>\nReplicate_Do_Table:<br \/>\nReplicate_Ignore_Table:<br \/>\nReplicate_Wild_Do_Table:<br \/>\nReplicate_Wild_Ignore_Table:<br \/>\nLast_Errno: 0<br \/>\nLast_Error:<br \/>\nSkip_Counter: 0<br \/>\nExec_Master_Log_Pos: 410<br \/>\nRelay_Log_Space: 445<br \/>\nUntil_Condition: None<br \/>\nUntil_Log_File:<br \/>\nUntil_Log_Pos: 0<br \/>\nMaster_SSL_Allowed: No<br \/>\nMaster_SSL_CA_File:<br \/>\nMaster_SSL_CA_Path:<br \/>\nMaster_SSL_Cert:<br \/>\nMaster_SSL_Cipher:<br \/>\nMaster_SSL_Key:<br \/>\nSeconds_Behind_Master: 103799<br \/>\n1 row in set (0.00 sec)<\/p>\n<p>ERROR:<br \/>\nNo query specified<\/p>\n<p>===========================================================================<\/p>\n<p><strong><span style=\"text-decoration: underline;\">NOW SETUP MYSQL FOR KEEPALIVED TO BE ABLE TO WRITE TO IT.<\/span><\/strong><\/p>\n<p>Note &#8211; if you setup replication correctly, this will replicate over to your secondary DB and you wont have to worry about setting this up on your secondary.<br \/>\n1. this is so keepalived msc_chk will be able to test to see if the db can be written to and it put a timestamp in the tables.<\/p>\n<p>&#8211; You are now creating a DB for keepalived to write to<\/p>\n<p>Copy Paste at MySQL prompt logged in as MySQL root<\/p>\n<p>===========================================================<\/p>\n<p><em>CREATE DATABASE `healthcheck` DEFAULT CHARACTER SET latin1 COLLATE latin1_swedish_ci;<\/em><br \/>\n<em> CREATE TABLE `healthcheck`.`keepalived` (<\/em><br \/>\n<em> `id` INT( 11 ) NOT NULL AUTO_INCREMENT ,<\/em><br \/>\n<em> `data` VARCHAR( 100 ) NULL ,<\/em><br \/>\n<em> PRIMARY KEY ( `id` )<\/em><br \/>\n<em> ) ENGINE = MYISAM;<\/em><br \/>\n<em> CREATE TABLE `healthcheck`.`keepalived2` (<\/em><br \/>\n<em> `id` INT( 11 ) NOT NULL AUTO_INCREMENT ,<\/em><br \/>\n<em> `data` VARCHAR( 100 ) NULL ,<\/em><br \/>\n<em> PRIMARY KEY ( `id` )<\/em><br \/>\n<em> ) ENGINE = MYISAM;<\/em><\/p>\n<p><em>use healthcheck;<\/em><\/p>\n<p><em>insert into keepalived (data) values (&#8220;hello&#8221;);<\/em><br \/>\n<em> insert into keepalived (data) values (&#8220;hello&#8221;);<\/em><br \/>\n<em> insert into keepalived2 (data) values (&#8220;hello&#8221;);<\/em><br \/>\n<em> insert into keepalived2 (data) values (&#8220;hello&#8221;);<\/em><br \/>\n<em> GRANT ALL ON *.* TO &#8216;lvs1&#8217;@&#8217;lvs1.nicktailor.com&#8217; identified by &#8216;password&#8217;;<\/em><\/p>\n<p><em>GRANT ALL ON *.* TO &#8216;lvs2&#8217;@&#8217;lvs2.nicktailor.com&#8217; identified by &#8216;password&#8217;;<\/em><\/p>\n<p>============================================================<\/p>\n<p>2. A)Log back into LVS1<br \/>\nB)Also do the same for LVS2 with below steps.<\/p>\n<p>touch \/var\/log\/keepalived\/dbcheckprod.log<\/p>\n<p>create a directory called \/root\/keepalived\/<br \/>\n&#8211; this is where your msck_chk script lives<br \/>\n&#8211; it will check tcp, port, mysql service and table write) 3 times and log it if it fails keepalived will fail to the secondary<br \/>\n&#8211; please note the UPDATE table section in this script needs to match the mysql ID you have in mysql below in the script<\/p>\n<p>\/root\/keepalived\/dbcheck.sh<br \/>\nhttps:\/\/www.nicktailor.com\/files\/dbcheck.sh<br \/>\n=======================================<br \/>\n<em>#!\/usr\/bin\/perl<\/em><br \/>\n<em> #use strict;<\/em><br \/>\n<em> #use warnings;<\/em><br \/>\n<em> use DBI;<\/em><br \/>\n<em> use POSIX;<\/em><br \/>\n<em> #use Mysql;<\/em><br \/>\n<em> use Sys::Hostname;<\/em><\/p>\n<p><em>$host = $ARGV[$0] ;<\/em><br \/>\n<em> $lvs_host=hostname;<\/em><\/p>\n<p><em>$database = &#8220;healthcheck&#8221;;<\/em><br \/>\n<em> $tablename = &#8220;keepalived&#8221;;<\/em><br \/>\n<em> $user = &#8220;lvs1&#8221;;<\/em><br \/>\n<em> $pw = &#8220;password&#8221;;<\/em><br \/>\n<em> $dbinfo = &#8220;DBI:mysql:$database;host=$host;mysql_connect_timeout=2&#8221;;<\/em><br \/>\n<em> $timestamp = strftime &#8220;%b%e %Y %H:%M:%S&#8221;, localtime;<\/em><br \/>\n<em> $logdir = &#8220;\/var\/log\/keepalived\/dbcheckprod.log&#8221;;<\/em><\/p>\n<p><em>open (LOG_FILE,&#8221;&gt;&gt;$logdir&#8221;);<\/em><\/p>\n<p><em>if ($lvs_host eq &#8220;lvs1.nicktailor.com&#8221;)<\/em><br \/>\n<em> {<\/em><br \/>\n<em> $tablename = &#8220;keepalived&#8221;;<\/em><br \/>\n<em> }<\/em><br \/>\n<em> elsif ($lvs_host eq &#8220;lvs2.nicktailor.com&#8221;)<\/em><br \/>\n<em> {<\/em><br \/>\n<em> $tablename = &#8220;keepalived2&#8221;;<\/em><br \/>\n<em> }<\/em><br \/>\n<em> #$connect = Mysql-&gt;connect($host, $database, $user, $pw);<\/em><br \/>\n<em> #$dbh = DBI-&gt;connect($dbinfo,$user,$pw,{ RaiseError =&gt; 1 }) or die $DBI::errstr;<\/em><\/p>\n<p><em>$connect_fail = 0;<\/em><\/p>\n<p><em>print LOG_FILE &#8220;\\n\\n\\n$timestamp\\n&#8221;;<\/em><br \/>\n<em> print LOG_FILE &#8220;$host\\n&#8221;;<\/em><br \/>\n<em> # try to connect to db server<\/em><br \/>\n<em> # exit with error if db connection fails 3x<\/em><br \/>\n<em> for ($count=0; $count &lt;= 2; $count++)<\/em><br \/>\n<em> {<\/em><br \/>\n<em> $dbh = DBI-&gt;connect($dbinfo,$user,$pw, {RaiseError =&gt; 0, PrintError=&gt;0});<\/em><\/p>\n<p><em>print LOG_FILE &#8220;Outer loop round $count: error is &#8221; . $DBI::err . &#8220;\\n&#8221;;<\/em><\/p>\n<p><em>if (!$DBI::errstr)<\/em><br \/>\n<em> {<\/em><br \/>\n<em> print LOG_FILE &#8220;connected!\\n&#8221;;<\/em><br \/>\n<em> $myquery = &#8220;UPDATE $tablename SET data=&#8217;$timestamp&#8217; WHERE id = &#8216;1&#8217;&#8221;;<\/em><\/p>\n<p><em>$fail=0;<\/em><\/p>\n<p><em>for ($count = 0; $count &lt;= 2; $count++)<\/em><br \/>\n<em> {<\/em><\/p>\n<p><em>$result = $dbh-&gt;do($myquery);<\/em><\/p>\n<p><em>if ($result &gt; 0)<\/em><br \/>\n<em> {<\/em><br \/>\n<em> print LOG_FILE &#8220;$count &#8220;;<\/em><br \/>\n<em> print LOG_FILE &#8220;success!\\n&#8221;;<\/em><br \/>\n<em> }<\/em><br \/>\n<em> else<\/em><br \/>\n<em> {<\/em><br \/>\n<em> print LOG_FILE &#8220;$count &#8220;;<\/em><br \/>\n<em> print LOG_FILE &#8220;fail!\\n&#8221;;<\/em><br \/>\n<em> $fail++;<\/em><br \/>\n<em> }<\/em><\/p>\n<p><em>sleep(8);<\/em><\/p>\n<p><em>}<\/em><\/p>\n<p><em>$dbh-&gt;disconnect();<\/em><br \/>\n<em> if ($fail == 0)<\/em><br \/>\n<em> {<\/em><br \/>\n<em> print LOG_FILE &#8220;success because failed $fail times\\n&#8221;;<\/em><br \/>\n<em> print LOG_FILE &#8220;exit 0 from table test\\n&#8221;;<\/em><br \/>\n<em> close (LOG_FILE);<\/em><br \/>\n<em> exit 0;<\/em><br \/>\n<em> }<\/em><\/p>\n<p><em>else<\/em><br \/>\n<em> {<\/em><br \/>\n<em> print LOG_FILE &#8220;failed $fail times\\n&#8221;;<\/em><br \/>\n<em> print LOG_FILE &#8220;exit 1 from table test\\n&#8221;;<\/em><br \/>\n<em> close (LOG_FILE);<\/em><br \/>\n<em> exit 1;<\/em><br \/>\n<em> }<\/em><\/p>\n<p><em>}<\/em><br \/>\n<em> else<\/em><br \/>\n<em> {<\/em><br \/>\n<em> $connect_fail++;<\/em><br \/>\n<em> print LOG_FILE &#8220;connnect_fail = $connect_fail\\n&#8221;;<\/em><\/p>\n<p><em>}<\/em><\/p>\n<p><em>sleep(8);<\/em><br \/>\n<em> }<\/em><\/p>\n<p><em>if ($connect_fail &gt; 0)<\/em><br \/>\n<em> {<\/em><br \/>\n<em> print LOG_FILE &#8220;failed $connect_fail times!&#8221;;<\/em><br \/>\n<em> print LOG_FILE &#8220;exit 1 from connect_fail test\\n&#8221;;<\/em><br \/>\n<em> close (LOG_FILE);<\/em><br \/>\n<em> exit 1;<\/em><br \/>\n<em> }<\/em><br \/>\n<em> #else clause may not be necessary because if connect_fail &lt; 3 then table check code would take care of exits<\/em><br \/>\n<em> else<\/em><br \/>\n<em> {<\/em><br \/>\n<em> print LOG_FILE &#8220;connect fail is 3 but still in else clause?&#8221;;<\/em><br \/>\n<em> print LOG_FILE &#8220;exit 0 from connect test\\n&#8221;;<\/em><br \/>\n<em> close (LOG_FILE);<\/em><br \/>\n<em> exit 0;<\/em><br \/>\n<em> }<\/em><br \/>\n================================================================<\/p>\n<p><strong>MYSQL REPLICATION SYNCHORIZATION\/FAILURE PROTECTION<\/strong><br \/>\n================================================================<br \/>\n&#8211; So we want to make sure that MYSQL in the event restarts doesnt allow traffic until both databases servers are caught with replication.<\/p>\n<p>1. Log into mysql server<br \/>\n2. disabled mysql init startup script. If your on redhat its simply &#8216;chkconfig mysql off&#8217;<br \/>\n3. edit \/etc\/rc.local and add the line<br \/>\n\/etc\/init.d\/mysql2 to the end and save it<\/p>\n<p>3. Now your going to setup your start up script that will make sure the replication is caughtup before it allows traffic to the DB<\/p>\n<p>a) Make sure you have iptables setup on your DB servers and that the rules are setup to allow lvs1 and lvs2 ips to connect on port 3306<\/p>\n<p>\/etc\/init.d\/mysql2<br \/>\nhttps:\/\/www.nicktailor.com\/files\/mysql2<br \/>\n==================================<br \/>\n<em>#!\/bin\/bash<\/em><br \/>\n<em> # This will script delete remove the lvs ips if they exist and then add them so there are no duplicate entries<\/em><br \/>\n<em> # It greps for the seconds behind the Master until it hits ZERO if its at ZERO then it will allow the LVS server to connect to the DB.<\/em><br \/>\n<em> # It will also send out an email if replication is broken and keep traffic blocked. My advice before setting this up is get familar with it # and test before using.<\/em><\/p>\n<p><em># Make healthcheck fail<\/em><br \/>\n<em> IPTABLES=&#8221;$(which iptables)&#8221;<\/em><\/p>\n<p><em># Make sure rule doesn&#8217;t exist before we add so we don&#8217;t double up<\/em><br \/>\n<em> $IPTABLES -D RH-Firewall-1-INPUT -s &lt;lvs1ip&gt; -p tcp &#8211;dport 3306 -j ACCEPT<\/em><br \/>\n<em> $IPTABLES -D RH-Firewall-1-INPUT -s &lt;lvs2ip&gt; -p tcp &#8211;dport 3306 -j ACCEPT<\/em><\/p>\n<p><em># start mysql<\/em><br \/>\n<em> service mysql start<\/em><\/p>\n<p><em>GREP=&#8221;$(which grep)&#8221;<\/em><br \/>\n<em> CUT=&#8221;$(which cut)&#8221;<\/em><br \/>\n<em> SLEEP=&#8221;$(which sleep)&#8221;<\/em><\/p>\n<p><em># start values<\/em><br \/>\n<em> POS=4<\/em><br \/>\n<em> #EXEC_POS=5<\/em><br \/>\n<em> #I=0<\/em><\/p>\n<p><em>ECHO=&#8221;\/bin\/echo&#8221;<\/em><br \/>\n<em> #$ECHO &#8220;Waiting for Replication &#8220;<\/em><br \/>\n<em> #while ( test &#8220;$READ_POS&#8221; != &#8220;$EXEC_POS&#8221; )<\/em><\/p>\n<p><em>$SLEEP 10<\/em><br \/>\n<em> while [ &#8220;$POS&#8221; -gt 0 ];<\/em><br \/>\n<em> do<\/em><br \/>\n<em> POS=`\/usr\/bin\/mysql -Bse &#8216;show slave status\\\\G&#8217; -ppassword | grep &#8216;Seconds_Behind_Master&#8217; | grep -o -E &#8216;[0-9]*$&#8217;`<\/em><\/p>\n<p><em>if [[ $POS != [0-9]* ]];<\/em><br \/>\n<em> then<\/em><br \/>\n<em> $ECHO &#8220;Replication is broken so keeping port 3306 blocked.&#8221;<\/em><br \/>\n<em> mail -s &#8216;Replication broken, keeping port 3306 blocked&#8217; nicktailor\\@nicktailor.com &lt; ~root\/mysqlstopped.txt<\/em><br \/>\n<em> exit<\/em><br \/>\n<em> fi<\/em><\/p>\n<p><em># READ_POS=`$ECHO $POS | $CUT -d\\ -f1`<\/em><br \/>\n<em> # EXEC_POS=`$ECHO $POS | $CUT -d\\ -f2`<\/em><\/p>\n<p><em>$ECHO &#8220;Seconds behind master is $POS&#8221;<\/em><br \/>\n<em> #$ECHO &#8220;READ_POS is $READ_POS&#8221;<\/em><br \/>\n<em> #$ECHO &#8220;EXEC_POS is $EXEC_POS&#8221;<\/em><\/p>\n<p><em># Output every ~10 sec<\/em><br \/>\n<em> # if ( test &#8220;`$ECHO $(($I%10))`&#8221; -eq &#8220;0&#8221; )<\/em><br \/>\n<em> # then<\/em><br \/>\n<em> # $ECHO &#8220;inside timer loop READ_POS is $READ_POS&#8221;<\/em><br \/>\n<em> # $ECHO &#8220;inside timer loop EXEC_POS $EXEC_POS&#8221;<\/em><br \/>\n<em> # fi<\/em><br \/>\n<em> # I=$(($I+1))<\/em><br \/>\n<em> $SLEEP 1<\/em><br \/>\n<em> done<\/em><\/p>\n<p><em># Make healthcheck succeed<\/em><br \/>\n<em> $IPTABLES -I RH-Firewall-1-INPUT 12 -s &lt;lvs1ip&gt; -p tcp &#8211;dport 3306 -j ACCEPT<\/em><br \/>\n<em> $IPTABLES -I RH-Firewall-1-INPUT 12 -s &lt;lvs2ip&gt; -p tcp &#8211;dport 3306 -j ACCEPT<\/em><\/p>\n<p>===============================================================<\/p>\n<p>4. Now we on your secondary DB we want to make sure that in the event a failure occurs the replication is not broken or the data would get out of sync. So we have a script that check replication every 5 seconds via a cron, if replication is broken it will change the DB to read only and send out an alert email.<\/p>\n<p>\/root\/repl_check\/dbhealth.pl<br \/>\nhttps:\/\/www.nicktailor.com\/files\/dbhealth.pl<br \/>\n======================================<br \/>\n#!\/usr\/bin\/perl<br \/>\nuse Sys::Hostname;<br \/>\nuse POSIX;<\/p>\n<p>$timestamp = strftime &#8220;%b%e %Y %H:%M:%S&#8221;, localtime;<br \/>\n$host = hostname;<br \/>\n$email_lock = &#8220;\/root\/email.lck&#8221;;<br \/>\n$mysql_socket = &#8220;\/var\/lib\/mysql\/mysql.sock&#8221;;<br \/>\n$show_slave_status = &#8220;\/root\/repl_check\/show_slave_status.txt&#8221;;<br \/>\n$set_read_only = &#8220;\/root\/repl_check\/set_db_read_only.sql&#8221;;<br \/>\n$pword = &#8220;password&#8221;;<\/p>\n<p># check to see if mysql socket exists. if it exists, means that mysql is running. if mysql not running, don&#8217;t need to run slave status check<br \/>\nsub check_mysql_socket<br \/>\n{<br \/>\n# Can&#8217;t connect to local MySQL server through socket &#8216;\/var\/lib\/mysql\/mysql.sock<br \/>\nif (-e $mysql_socket)<br \/>\n{<br \/>\nprint &#8220;MySQL running, will proceed\\n&#8221;;<br \/>\nreturn 1;<br \/>\n}<\/p>\n<p>else<br \/>\n{<br \/>\nprint &#8220;MySQL not running, will do nothing\\n&#8221;;<br \/>\nreturn 0;<br \/>\n}<\/p>\n<p>}<\/p>\n<p># check to see if email.lck exists. if it does, that means email has been sent, no need to keep resending it<br \/>\nsub check_email_lock<br \/>\n{<br \/>\nif (-e $email_lock)<br \/>\n{<br \/>\nprint &#8220;email file exists\\n&#8221;;<br \/>\nreturn 1;<br \/>\n}<br \/>\nelse<br \/>\n{<br \/>\nprint &#8220;no email file exists\\n&#8221;;<br \/>\nreturn 0;<br \/>\n}<br \/>\n}<\/p>\n<p>sub stop_mysql<br \/>\n{<br \/>\nprint &#8220;**Show Read Only Status**\\n&#8221;;<br \/>\nsystem(&#8220;mysql -p$pword &lt; $set_read_only&#8221;);<\/p>\n<p>if (check_email_lock)<br \/>\n{<br \/>\nprint &#8220;email lock exists, keep email lock, no email will be sent &#8220;;<br \/>\n}<br \/>\nelse<br \/>\n{<br \/>\nsystem (&#8220;mail -s &#8216;mysql stopped on $host&#8217; nicktailor\\@nicktailor.com &lt; ~root\/mysqlstopped.txt&#8221;);<br \/>\nsystem (&#8220;touch $email_lock&#8221;);<br \/>\nprint &#8220;email sent, email lock created\\n&#8221;;<br \/>\n}<\/p>\n<p>}<br \/>\nprint $timestamp . &#8220;\\n&#8221;;<\/p>\n<p># if MySQL is running<br \/>\nif (check_mysql_socket)<br \/>\n{<br \/>\n# for testing with text file<br \/>\n# $last_io_errno = `less develscripts\/slavestatus_master_down.txt | grep Last_IO_Errno | \/usr\/bin\/awk &#8216;{print \\$2}&#8217;`;<br \/>\n# $last_sql_errno = `less develscripts\/slavestatus_master_down.txt | grep Last_SQL_Errno | \/usr\/bin\/awk &#8216;{print \\$2}&#8217;`;<\/p>\n<p>system (&#8220;\/usr\/bin\/mysql -Bse &#8216;show slave status\\\\G&#8217; -p$pword &gt; $show_slave_status&#8221;);<\/p>\n<p>$last_io_errno = `less $show_slave_status | grep Last_IO_Errno | \/usr\/bin\/awk &#8216;{print \\$2}&#8217;`;<br \/>\n$last_sql_errno = `less $show_slave_status | grep Last_SQL_Errno | \/usr\/bin\/awk &#8216;{print \\$2}&#8217;`;<br \/>\n$slave_io_running = `less $show_slave_status | grep Slave_IO_Running | \/usr\/bin\/awk &#8216;{print \\$2}&#8217;`;<br \/>\n$slave_sql_running = `less $show_slave_status | grep Slave_SQL_Running | \/usr\/bin\/awk &#8216;{print \\$2}&#8217;`;<\/p>\n<p># trim newline character<br \/>\nchomp($last_io_errno);<br \/>\nchomp($last_sql_errno);<br \/>\nchomp($slave_io_running);<br \/>\nchomp($slave_sql_running);<\/p>\n<p>print &#8220;last io error is &#8221; . $last_io_errno . &#8220;\\n&#8221;;<br \/>\nprint &#8220;last sql errno is &#8221; . $last_sql_errno . &#8220;\\n&#8221;;<br \/>\nprint &#8220;slave io running is &#8221; . $slave_io_running . &#8220;\\n&#8221;;<br \/>\nprint &#8220;slave sql running is &#8221; . $slave_sql_running . &#8220;\\n&#8221;;<br \/>\nif (($last_io_errno &gt; 0) &amp;&amp; ($last_io_errno != 2013))<br \/>\n{<\/p>\n<p>&amp;stop_mysql;<br \/>\n}<\/p>\n<p>elsif ($last_sql_errno &gt; 0)<br \/>\n{<\/p>\n<p>&amp;stop_mysql;<br \/>\n}<\/p>\n<p># if slave not running = Slave_IO_Running and Slave_SQL_Running are set to No<br \/>\nelsif (($slave_io_running eq &#8220;No&#8221;) &amp;&amp; ($slave_sql_running eq &#8220;No&#8221;))<br \/>\n{<br \/>\n&amp;stop_mysql;<br \/>\n}<\/p>\n<p>else<br \/>\n{<br \/>\nif (check_email_lock)<br \/>\n{<br \/>\nsystem (&#8220;rm $email_lock&#8221;);<br \/>\n}<br \/>\nprint &#8220;replication fine or master&#8217;s just down, mysql can keep going, removed lock file\\n&#8221;;<br \/>\n}<br \/>\n}<\/p>\n<p>else<br \/>\n{<br \/>\nprint &#8220;#2 MySQL not running, will do nothing\\n&#8221;;<br \/>\n}<\/p>\n<p>print &#8220;\\n#########################\\n&#8221;;<br \/>\n==================================================================================<\/p>\n<p>Also create the following files for the above script to work<\/p>\n<p>set_db_read_only.sql<br \/>\nhttps:\/\/www.nicktailor.com\/files\/<br \/>\n=========================<br \/>\nshow variables like &#8220;read_only&#8221;;<br \/>\nset global read_only=on;<br \/>\nshow variables like &#8220;read_only&#8221;;<br \/>\n=========================<\/p>\n<p>touch show_slave_status.txt<br \/>\n===============================<\/p>\n<p>unset_db_read_only.sql<br \/>\nhttps:\/\/www.nicktailor.com\/files\/unset_db_read_only.sql<br \/>\n===============================<br \/>\nshow variables like &#8220;read_only&#8221;;<br \/>\nset global read_only=off;<br \/>\nshow variables like &#8220;read_only&#8221;;<br \/>\n===============================<br \/>\nNext setup a cron to run this on the secondary DB however many seconds you like<\/p>\n<p>FINAL SETUP PIECE!!<\/p>\n<p><strong>MYSQL BACKUPS<\/strong><\/p>\n<p>ON noth DB server you want to have mysql backups running via a cron<\/p>\n<p>For multi-threaded backup and restores of InnoDB and myisam check out my&nbsp;<a href=\"https:\/\/www.nicktailor.com\/?p=270\">https:\/\/www.nicktailor.com\/?p=270<\/a>&nbsp;post here.<\/p>\n<p>\/root\/backup\/mysqldumpbackup.sh<br \/>\nhttps:\/\/www.nicktailor.com\/files\/mysqldumpbackup.sh<br \/>\n==============================<br \/>\n<em>#Backup multiple MySQL databases into separate files and one full backup of all of them and then tars them in to archive directory and<\/em><br \/>\n<em> #retains 7 day archives<\/em><br \/>\n<em> #!\/bin\/bash<\/em><br \/>\n<em> # backup each mysql db into a different file, rather than one big file<\/em><br \/>\n<em> # as with &#8211;all-databases &#8211; will make restores easier<\/em><\/p>\n<p><em>USER=&#8221;root&#8221;<\/em><br \/>\n<em> PASSWORD=&#8221;password&#8221;<\/em><br \/>\n<em> OUTPUTDIR=&#8221;mysqldumpoutput&#8221;<\/em><br \/>\n<em> MYSQLDUMP=&#8221;\/usr\/bin\/mysqldump&#8221;<\/em><br \/>\n<em> MYSQL=&#8221;\/usr\/bin\/mysql&#8221;<\/em><br \/>\n<em> purgedir=&#8221;mysqldumptars&#8221;<\/em><br \/>\n<em> now=`date &#8216;+%Y-%m-%d-%H-%M-%S&#8217;`<\/em><\/p>\n<p><em># clean up any old backups &#8211; save space<\/em><br \/>\n<em> \/bin\/rm &#8220;OUTPUTDIR\/*bak&#8221; &gt; \/dev\/null 2&gt;&amp;1<\/em><\/p>\n<p><em># does a all database mysql dump backup<\/em><br \/>\n<em> $MYSQLDUMP &#8211;single-transaction &#8211;force &#8211;opt &#8211;user=$USER &#8211;password=$PASSWORD &#8211;all-databases &gt; &#8220;$OUTPUTDIR\/alldbdump.bak&#8221;<\/em><\/p>\n<p><em># get a list of databases<\/em><br \/>\n<em> databases=`$MYSQL &#8211;user=$USER &#8211;password=$PASSWORD -e &#8220;SHOW DATABASES;&#8221; | tr -d &#8220;| &#8221; | grep -v Database`<\/em><\/p>\n<p><em># dump each database in turn<\/em><br \/>\n<em> for db in $databases; do<\/em><br \/>\n<em> echo $db<\/em><br \/>\n<em> $MYSQLDUMP &#8211;single-transaction &#8211;force &#8211;opt &#8211;user=$USER &#8211;password=$PASSWORD &#8211;databases $db &gt; &#8220;$OUTPUTDIR\/$db.bak&#8221;<\/em><br \/>\n<em> done<\/em><\/p>\n<p><em>#Backup section<\/em><br \/>\n<em> #===================<\/em><\/p>\n<p><em>#this make a tar of the new separate dumpfiles with a time stamp and moves the tar to separate directory upon completion<\/em><br \/>\n<em> cd mysqldumpoutput\/<\/em><br \/>\n<em> \/bin\/tar -zcvf mysqldump.tar .\/*<\/em><br \/>\n<em> \/bin\/mv mysqldumpoutput\/mysqldump.tar \/dd02\/mysqldumptars\/mysqldump_${now}.tar<\/em><\/p>\n<p><em>#This deletes any tar that is more than 7 days old for the dump backups<\/em><br \/>\n<em> \/usr\/bin\/find ${purgedir} -maxdepth 1 -name &#8220;mysqldump_*&#8221; -mtime +6 -exec rm -rf {} \\; -ls<\/em><\/p>\n<p>===============================================================<\/p>\n<p>I HOPE YOU ENJOYED THIS TUTORIAL AND SETUP I KNOW ITS LONG. \ud83d\ude42<\/p>\n<p>If you log into LVS1 now and run<\/p>\n<p>watch ipvsadm -L you should see your servers<\/p>\n<p>To see LVS pool type &#8216;watch ipvsadm -L -n&#8217; &lt;&#8211;This shows all VIPS and real servers bound to them<\/p>\n<p>LVS-HINTS: If you want to watch a specific VIP only:<\/p>\n<p>&#8221; watch -n1 &#8216;ipvsadm -L -n|grep -A$N $VIP:$PORT&#8217; &#8221; &lt;&#8211; this will run the specified command updating every 1 second.<\/p>\n<p>Replace $VIP with the virtual IP $PORT with the service and $N with the number of real servers, ie &#8221; watch -n1 &#8216;ipvsadm -L -n | grep -A2 192.168.0.1:80&#8217; &#8221;<\/p>\n<p>Eg View &#8211; &#8216;watch ipvsadm -L -n&#8217;<br \/>\n=============================================================================================<br \/>\nEvery 2.0s: ipvsadm -L -n Tue Oct 25 16:47:22 2011<br \/>\nIP Virtual Server version 1.2.1 (size=4096)<br \/>\nProt LocalAddress:Port Scheduler Flags<br \/>\n-&gt; RemoteAddress:Port Forward Weight ActiveConn InActConn<br \/>\nTCP 172.16.10.1:3306 wlc &lt;&#8212;-VIP<br \/>\n-&gt; 142.103.18.1:3306 Route 1000 0 45 &lt;&#8211;Real Server<br \/>\n==============================================================================================<br \/>\n&#8216;ip addr&#8217; will show you which lvs the vip is bound to<br \/>\nTo Stop keepalived &#8216;\/etc\/init.d\/keepalived stop&#8217;<br \/>\nTo start keepalived &#8216;\/usr\/local\/sbin\/keepalived&#8217;<br \/>\n===============================<\/p>\n<p>If you have questions you can email me at nick@nicktailor.com<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hello, I&#8217;m sure many people are wondering how to do this. So I decided to write this up on how I setup the architecture. This also for people who cant use mysql multimaster, because the application they are using doesn&#8217;t &#8220;offically&#8221; support mysql multimaster. If it does, you should look at that avenue before attempting this type of setup \ud83d\ude42<a href=\"https:\/\/nicktailor.com\/tech-blog\/using-keepalived-for-mysql-failover\/\" class=\"read-more\">Read More &#8230;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":true,"template":"","format":"standard","meta":{"footnotes":""},"categories":[138,3],"tags":[],"class_list":["post-4","post","type-post","status-publish","format-standard","hentry","category-linux","category-mysql-fail-over-setup"],"_links":{"self":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/4","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/comments?post=4"}],"version-history":[{"count":33,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/4\/revisions"}],"predecessor-version":[{"id":1585,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/4\/revisions\/1585"}],"wp:attachment":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/media?parent=4"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/categories?post=4"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/tags?post=4"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}