# mysql-mha-docker **Repository Path**: zyzheal/mysql-mha-docker ## Basic Information - **Project Name**: mysql-mha-docker - **Description**: No description available - **Primary Language**: Unknown - **License**: Not specified - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2020-06-13 - **Last Updated**: 2020-12-19 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # mysql-mha-docker 环境 docker-19.03.5 mysql-5.7.19 mha4mysql-node:0.57 ## Usage 1. download 2. run containers ```cmd $ docker-compose up -d ``` 3. start ssh service in all containers ```cmd $ sh ./scripts/ssh_start.sh ``` 4. copy ssh public key to each container ```cmd $ sh ./scripts/ssh_share.sh ``` 5. check mha ssh ```cmd $ sh ./scripts/mha_check_ssh.sh ``` 6. set mysql replication (master, backup and slave) ```cmd $ sh ./scripts/mysql_set_mbs.sh ``` 7. check mha repl ```cmd $ sh ./scripts/mha_check_repl.sh ``` 8. start mha manager ```cmd $ sh ./scripts/mha_start_manager.sh ``` 9. docker exec -it mha_node0 /bin/bash (master) ```cmd mysql -u root -p123456 create database lagou; use lagou create table position ( id int primary key, name varchar(48), salary varchar(48), city varchar(48) )engine=innodb charset=utf8; create table position_detail( id int primary key, pid int, description text )engine=innodb charset=utf8; ``` ```cmd mysql> show master status; +------------------+----------+--------------+-------------------------------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +------------------+----------+--------------+-------------------------------------------+-------------------+ | mysql-bin.000001 | 154 | | performance_schema,information_schema,sys | | +------------------+----------+--------------+-------------------------------------------+-------------------+ 1 row in set (0.00 sec) mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | | testing | +--------------------+ 5 rows in set (0.06 sec) mysql> create database lagou; Query OK, 1 row affected (0.00 sec) mysql> use lagou Database changed mysql> create table position ( -> id int primary key, -> name varchar(48), -> salary varchar(48), -> city varchar(48) -> )engine=innodb charset=utf8; Query OK, 0 rows affected (0.04 sec) mysql> mysql> create table position_detail( -> id int primary key, -> pid int, -> description text -> )engine=innodb charset=utf8; Query OK, 0 rows affected (0.05 sec) ``` 10. docker exec -it mha_node1 /bin/bash (slave1) ```cmd mysql> show master status; +------------------+----------+--------------+-------------------------------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +------------------+----------+--------------+-------------------------------------------+-------------------+ | mysql-bin.000001 | 443 | | performance_schema,information_schema,sys | | +------------------+----------+--------------+-------------------------------------------+-------------------+ 1 row in set (0.00 sec) mysql> show master status; +------------------+----------+--------------+-------------------------------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +------------------+----------+--------------+-------------------------------------------+-------------------+ | mysql-bin.000001 | 443 | | performance_schema,information_schema,sys | | +------------------+----------+--------------+-------------------------------------------+-------------------+ 1 row in set (0.00 sec) mysql> show slave status \G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 10.5.0.10 Master_User: myslave Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 154 Relay_Log_File: mysql-relay-bin.000002 Relay_Log_Pos: 367 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 154 Relay_Log_Space: 574 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: 926ee588-ad4f-11ea-8f39-02420a05000a Master_Info_File: /var/lib/mysql/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.00 sec) mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | lagou | | mysql | | performance_schema | | sys | | testing | +--------------------+ 6 rows in set (0.05 sec) mysql> use lagou; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> show tables; +-----------------+ | Tables_in_lagou | +-----------------+ | position | | position_detail | +-----------------+ 2 rows in set (0.01 sec) ``` 11. docker exec -it mha_node2 /bin/bash (slave2) ```cmd mysql> show master status; +------------------+----------+--------------+-------------------------------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +------------------+----------+--------------+-------------------------------------------+-------------------+ | mysql-bin.000006 | 443 | | performance_schema,information_schema,sys | | +------------------+----------+--------------+-------------------------------------------+-------------------+ 1 row in set (0.00 sec) mysql> show slave status \G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 10.5.0.10 Master_User: myslave Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 154 Relay_Log_File: mysql-relay-bin.000006 Relay_Log_Pos: 367 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 154 Relay_Log_Space: 1369 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: 926ee588-ad4f-11ea-8f39-02420a05000a Master_Info_File: /var/lib/mysql/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.00 sec) mysql> use lagou; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> show tables; +-----------------+ | Tables_in_lagou | +-----------------+ | position | | position_detail | +-----------------+ 2 rows in set (0.01 sec) ``` 12. 关闭master docker stop mha_node0 mha_manager 日志输出 ```cmd mysql-mha-docker git:(master) ✗ sh ./scripts/mha_start_manager.sh Sat Jun 13 14:51:29 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sat Jun 13 14:51:29 2020 - [info] Reading application default configuration from /etc/mha/app1.conf.. Sat Jun 13 14:51:29 2020 - [info] Reading server configuration from /etc/mha/app1.conf.. Sat Jun 13 14:51:29 2020 - [info] MHA::MasterMonitor version 0.57. Sat Jun 13 14:51:29 2020 - [info] GTID failover mode = 0 Sat Jun 13 14:51:29 2020 - [info] Dead Servers: Sat Jun 13 14:51:29 2020 - [info] Alive Servers: Sat Jun 13 14:51:29 2020 - [info] 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:51:29 2020 - [info] 10.5.0.11(10.5.0.11:3306) Sat Jun 13 14:51:29 2020 - [info] 10.5.0.12(10.5.0.12:3306) Sat Jun 13 14:51:29 2020 - [info] Alive Slaves: Sat Jun 13 14:51:29 2020 - [info] 10.5.0.11(10.5.0.11:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled Sat Jun 13 14:51:29 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:51:29 2020 - [info] Primary candidate for the new Master (candidate_master is set) Sat Jun 13 14:51:29 2020 - [info] 10.5.0.12(10.5.0.12:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled Sat Jun 13 14:51:29 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:51:29 2020 - [info] Not candidate for the new Master (no_master is set) Sat Jun 13 14:51:29 2020 - [info] Current Alive Master: 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:51:29 2020 - [info] Checking slave configurations.. Sat Jun 13 14:51:29 2020 - [warning] relay_log_purge=0 is not set on slave 10.5.0.11(10.5.0.11:3306). Sat Jun 13 14:51:29 2020 - [warning] relay_log_purge=0 is not set on slave 10.5.0.12(10.5.0.12:3306). Sat Jun 13 14:51:29 2020 - [info] Checking replication filtering settings.. Sat Jun 13 14:51:29 2020 - [info] binlog_do_db= , binlog_ignore_db= information_schema,performance_schema,sys Sat Jun 13 14:51:29 2020 - [info] Replication filtering check ok. Sat Jun 13 14:51:29 2020 - [info] GTID (with auto-pos) is not supported Sat Jun 13 14:51:29 2020 - [info] Starting SSH connection tests.. Sat Jun 13 14:51:31 2020 - [info] All SSH connection tests passed successfully. Sat Jun 13 14:51:31 2020 - [info] Checking MHA Node version.. Sat Jun 13 14:51:31 2020 - [info] Version check ok. Sat Jun 13 14:51:31 2020 - [info] Checking SSH publickey authentication settings on the current master.. Sat Jun 13 14:51:31 2020 - [info] HealthCheck: SSH to 10.5.0.10 is reachable. Sat Jun 13 14:51:31 2020 - [info] Master MHA Node version is 0.57. Sat Jun 13 14:51:31 2020 - [info] Checking recovery script configurations on 10.5.0.10(10.5.0.10:3306).. Sat Jun 13 14:51:31 2020 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/usr/local/mha/save_binary_logs_test --manager_version=0.57 --start_file=mysql-bin.000001 Sat Jun 13 14:51:31 2020 - [info] Connecting to root@10.5.0.10(10.5.0.10:22).. Creating /usr/local/mha if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /var/lib/mysql, up to mysql-bin.000001 Sat Jun 13 14:51:31 2020 - [info] Binlog setting check done. Sat Jun 13 14:51:31 2020 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Sat Jun 13 14:51:31 2020 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.5.0.11 --slave_ip=10.5.0.11 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.7.19-log --manager_version=0.57 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx Sat Jun 13 14:51:31 2020 - [info] Connecting to root@10.5.0.11(10.5.0.11:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to mysql-relay-bin.000004 Temporary relay log file is /var/lib/mysql/mysql-relay-bin.000004 Testing mysql connection and privileges..mysql: [Warning] Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sat Jun 13 14:51:32 2020 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.5.0.12 --slave_ip=10.5.0.12 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.7.19-log --manager_version=0.57 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx Sat Jun 13 14:51:32 2020 - [info] Connecting to root@10.5.0.12(10.5.0.12:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to mysql-relay-bin.000008 Temporary relay log file is /var/lib/mysql/mysql-relay-bin.000008 Testing mysql connection and privileges..mysql: [Warning] Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sat Jun 13 14:51:32 2020 - [info] Slaves settings check done. Sat Jun 13 14:51:32 2020 - [info] 10.5.0.10(10.5.0.10:3306) (current master) +--10.5.0.11(10.5.0.11:3306) +--10.5.0.12(10.5.0.12:3306) Sat Jun 13 14:51:32 2020 - [warning] master_ip_failover_script is not defined. Sat Jun 13 14:51:32 2020 - [warning] shutdown_script is not defined. Sat Jun 13 14:51:32 2020 - [info] Set master ping interval 5 seconds. Sat Jun 13 14:51:32 2020 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes. Sat Jun 13 14:51:32 2020 - [info] Starting ping health check on 10.5.0.10(10.5.0.10:3306).. Sat Jun 13 14:51:32 2020 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. Sat Jun 13 14:51:52 2020 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away) Sat Jun 13 14:51:52 2020 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/usr/local/mha/save_binary_logs_test --manager_version=0.57 --binlog_prefix=mysql-bin Connection to 10.5.0.10 closed by remote host. Sat Jun 13 14:51:52 2020 - [warning] HealthCheck: SSH to 10.5.0.10 is NOT reachable. Sat Jun 13 14:52:02 2020 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.5.0.10' (4)) Sat Jun 13 14:52:02 2020 - [warning] Connection failed 2 time(s).. Sat Jun 13 14:52:07 2020 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.5.0.10' (4)) Sat Jun 13 14:52:07 2020 - [warning] Connection failed 3 time(s).. Sat Jun 13 14:52:12 2020 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.5.0.10' (4)) Sat Jun 13 14:52:12 2020 - [warning] Connection failed 4 time(s).. Sat Jun 13 14:52:12 2020 - [warning] Master is not reachable from health checker! Sat Jun 13 14:52:12 2020 - [warning] Master 10.5.0.10(10.5.0.10:3306) is not reachable! Sat Jun 13 14:52:12 2020 - [warning] SSH is NOT reachable. Sat Jun 13 14:52:12 2020 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mha/app1.conf again, and trying to connect to all servers to check server status.. Sat Jun 13 14:52:12 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sat Jun 13 14:52:12 2020 - [info] Reading application default configuration from /etc/mha/app1.conf.. Sat Jun 13 14:52:12 2020 - [info] Reading server configuration from /etc/mha/app1.conf.. Sat Jun 13 14:52:12 2020 - [info] GTID failover mode = 0 Sat Jun 13 14:52:12 2020 - [info] Dead Servers: Sat Jun 13 14:52:12 2020 - [info] 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:52:12 2020 - [info] Alive Servers: Sat Jun 13 14:52:12 2020 - [info] 10.5.0.11(10.5.0.11:3306) Sat Jun 13 14:52:12 2020 - [info] 10.5.0.12(10.5.0.12:3306) Sat Jun 13 14:52:12 2020 - [info] Alive Slaves: Sat Jun 13 14:52:12 2020 - [info] 10.5.0.11(10.5.0.11:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled Sat Jun 13 14:52:12 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:52:12 2020 - [info] Primary candidate for the new Master (candidate_master is set) Sat Jun 13 14:52:12 2020 - [info] 10.5.0.12(10.5.0.12:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled Sat Jun 13 14:52:12 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:52:12 2020 - [info] Not candidate for the new Master (no_master is set) Sat Jun 13 14:52:12 2020 - [info] Checking slave configurations.. Sat Jun 13 14:52:12 2020 - [warning] relay_log_purge=0 is not set on slave 10.5.0.11(10.5.0.11:3306). Sat Jun 13 14:52:12 2020 - [warning] relay_log_purge=0 is not set on slave 10.5.0.12(10.5.0.12:3306). Sat Jun 13 14:52:12 2020 - [info] Checking replication filtering settings.. Sat Jun 13 14:52:12 2020 - [info] Replication filtering check ok. Sat Jun 13 14:52:12 2020 - [info] Master is down! Sat Jun 13 14:52:12 2020 - [info] Terminating monitoring script. Sat Jun 13 14:52:12 2020 - [info] Got exit code 20 (Master dead). Sat Jun 13 14:52:12 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sat Jun 13 14:52:12 2020 - [info] Reading application default configuration from /etc/mha/app1.conf.. Sat Jun 13 14:52:12 2020 - [info] Reading server configuration from /etc/mha/app1.conf.. Sat Jun 13 14:52:12 2020 - [info] MHA::MasterFailover version 0.57. Sat Jun 13 14:52:12 2020 - [info] Starting master failover. Sat Jun 13 14:52:12 2020 - [info] Sat Jun 13 14:52:12 2020 - [info] * Phase 1: Configuration Check Phase.. Sat Jun 13 14:52:12 2020 - [info] Sat Jun 13 14:52:12 2020 - [info] GTID failover mode = 0 Sat Jun 13 14:52:12 2020 - [info] Dead Servers: Sat Jun 13 14:52:12 2020 - [info] 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:52:12 2020 - [info] Checking master reachability via MySQL(double check)... Sat Jun 13 14:52:13 2020 - [info] ok. Sat Jun 13 14:52:13 2020 - [info] Alive Servers: Sat Jun 13 14:52:13 2020 - [info] 10.5.0.11(10.5.0.11:3306) Sat Jun 13 14:52:13 2020 - [info] 10.5.0.12(10.5.0.12:3306) Sat Jun 13 14:52:13 2020 - [info] Alive Slaves: Sat Jun 13 14:52:13 2020 - [info] 10.5.0.11(10.5.0.11:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled Sat Jun 13 14:52:13 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:52:13 2020 - [info] Primary candidate for the new Master (candidate_master is set) Sat Jun 13 14:52:13 2020 - [info] 10.5.0.12(10.5.0.12:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled Sat Jun 13 14:52:13 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:52:13 2020 - [info] Not candidate for the new Master (no_master is set) Sat Jun 13 14:52:13 2020 - [info] Starting Non-GTID based failover. Sat Jun 13 14:52:13 2020 - [info] Sat Jun 13 14:52:13 2020 - [info] ** Phase 1: Configuration Check Phase completed. Sat Jun 13 14:52:13 2020 - [info] Sat Jun 13 14:52:13 2020 - [info] * Phase 2: Dead Master Shutdown Phase.. Sat Jun 13 14:52:13 2020 - [info] Sat Jun 13 14:52:13 2020 - [info] Forcing shutdown so that applications never connect to the current master.. Sat Jun 13 14:52:13 2020 - [warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address. Sat Jun 13 14:52:13 2020 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master. Sat Jun 13 14:52:14 2020 - [info] * Phase 2: Dead Master Shutdown Phase completed. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] * Phase 3: Master Recovery Phase.. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] * Phase 3.1: Getting Latest Slaves Phase.. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] The latest binary log file/position on all slaves is mysql-bin.000001:154 Sat Jun 13 14:52:14 2020 - [info] Latest slaves (Slaves that received relay log files to the latest): Sat Jun 13 14:52:14 2020 - [info] 10.5.0.11(10.5.0.11:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled Sat Jun 13 14:52:14 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:52:14 2020 - [info] Primary candidate for the new Master (candidate_master is set) Sat Jun 13 14:52:14 2020 - [info] 10.5.0.12(10.5.0.12:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled Sat Jun 13 14:52:14 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:52:14 2020 - [info] Not candidate for the new Master (no_master is set) Sat Jun 13 14:52:14 2020 - [info] The oldest binary log file/position on all slaves is mysql-bin.000001:154 Sat Jun 13 14:52:14 2020 - [info] Oldest slaves: Sat Jun 13 14:52:14 2020 - [info] 10.5.0.11(10.5.0.11:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled Sat Jun 13 14:52:14 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:52:14 2020 - [info] Primary candidate for the new Master (candidate_master is set) Sat Jun 13 14:52:14 2020 - [info] 10.5.0.12(10.5.0.12:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled Sat Jun 13 14:52:14 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:52:14 2020 - [info] Not candidate for the new Master (no_master is set) Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase.. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [warning] Dead Master is not SSH reachable. Could not save it's binlogs. Transactions that were not sent to the latest slave (Read_Master_Log_Pos to the tail of the dead master's binlog) were lost. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] * Phase 3.3: Determining New Master Phase.. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] Finding the latest slave that has all relay logs for recovering other slaves.. Sat Jun 13 14:52:14 2020 - [info] All slaves received relay logs to the same position. No need to resync each other. Sat Jun 13 14:52:14 2020 - [info] Searching new master from slaves.. Sat Jun 13 14:52:14 2020 - [info] Candidate masters from the configuration file: Sat Jun 13 14:52:14 2020 - [info] 10.5.0.11(10.5.0.11:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled Sat Jun 13 14:52:14 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:52:14 2020 - [info] Primary candidate for the new Master (candidate_master is set) Sat Jun 13 14:52:14 2020 - [info] Non-candidate masters: Sat Jun 13 14:52:14 2020 - [info] 10.5.0.12(10.5.0.12:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled Sat Jun 13 14:52:14 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306) Sat Jun 13 14:52:14 2020 - [info] Not candidate for the new Master (no_master is set) Sat Jun 13 14:52:14 2020 - [info] Searching from candidate_master slaves which have received the latest relay log events.. Sat Jun 13 14:52:14 2020 - [info] New master is 10.5.0.11(10.5.0.11:3306) Sat Jun 13 14:52:14 2020 - [info] Starting master failover.. Sat Jun 13 14:52:14 2020 - [info] From: 10.5.0.10(10.5.0.10:3306) (current master) +--10.5.0.11(10.5.0.11:3306) +--10.5.0.12(10.5.0.12:3306) To: 10.5.0.11(10.5.0.11:3306) (new master) +--10.5.0.12(10.5.0.12:3306) Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] * Phase 3.3: New Master Diff Log Generation Phase.. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] * Phase 3.4: Master Log Apply Phase.. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed. Sat Jun 13 14:52:14 2020 - [info] Starting recovery on 10.5.0.11(10.5.0.11:3306).. Sat Jun 13 14:52:14 2020 - [info] This server has all relay logs. Waiting all logs to be applied.. Sat Jun 13 14:52:14 2020 - [info] done. Sat Jun 13 14:52:14 2020 - [info] All relay logs were successfully applied. Sat Jun 13 14:52:14 2020 - [info] Getting new master's binlog name and position.. Sat Jun 13 14:52:14 2020 - [info] mysql-bin.000001:443 Sat Jun 13 14:52:14 2020 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.5.0.11', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=443, MASTER_USER='myslave', MASTER_PASSWORD='xxx'; Sat Jun 13 14:52:14 2020 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address. Sat Jun 13 14:52:14 2020 - [info] Setting read_only=0 on 10.5.0.11(10.5.0.11:3306).. Sat Jun 13 14:52:14 2020 - [info] ok. Sat Jun 13 14:52:14 2020 - [info] ** Finished master recovery successfully. Sat Jun 13 14:52:14 2020 - [info] * Phase 3: Master Recovery Phase completed. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] * Phase 4: Slaves Recovery Phase.. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase.. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] -- Slave diff file generation on host 10.5.0.12(10.5.0.12:3306) started, pid: 822. Check tmp log /usr/local/mha/10.5.0.12_3306_20200613145212.log if it takes time.. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] Log messages from 10.5.0.12 ... Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sat Jun 13 14:52:14 2020 - [info] End of log messages from 10.5.0.12. Sat Jun 13 14:52:14 2020 - [info] -- 10.5.0.12(10.5.0.12:3306) has the latest relay log events. Sat Jun 13 14:52:14 2020 - [info] Generating relay diff files from the latest slave succeeded. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase.. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] -- Slave recovery on host 10.5.0.12(10.5.0.12:3306) started, pid: 824. Check tmp log /usr/local/mha/10.5.0.12_3306_20200613145212.log if it takes time.. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] Log messages from 10.5.0.12 ... Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] Starting recovery on 10.5.0.12(10.5.0.12:3306).. Sat Jun 13 14:52:14 2020 - [info] This server has all relay logs. Waiting all logs to be applied.. Sat Jun 13 14:52:14 2020 - [info] done. Sat Jun 13 14:52:14 2020 - [info] All relay logs were successfully applied. Sat Jun 13 14:52:14 2020 - [info] Resetting slave 10.5.0.12(10.5.0.12:3306) and starting replication from the new master 10.5.0.11(10.5.0.11:3306).. Sat Jun 13 14:52:14 2020 - [info] Executed CHANGE MASTER. Sat Jun 13 14:52:14 2020 - [info] Slave started. Sat Jun 13 14:52:14 2020 - [info] End of log messages from 10.5.0.12. Sat Jun 13 14:52:14 2020 - [info] -- Slave recovery on host 10.5.0.12(10.5.0.12:3306) succeeded. Sat Jun 13 14:52:14 2020 - [info] All new slave servers recovered successfully. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] * Phase 5: New master cleanup phase.. Sat Jun 13 14:52:14 2020 - [info] Sat Jun 13 14:52:14 2020 - [info] Resetting slave info on the new master.. Sat Jun 13 14:52:14 2020 - [info] 10.5.0.11: Resetting slave info succeeded. Sat Jun 13 14:52:14 2020 - [info] Master failover to 10.5.0.11(10.5.0.11:3306) completed successfully. Sat Jun 13 14:52:14 2020 - [info] Deleted server0 entry from /etc/mha/app1.conf . Sat Jun 13 14:52:14 2020 - [info] ----- Failover Report ----- app1: MySQL Master failover 10.5.0.10(10.5.0.10:3306) to 10.5.0.11(10.5.0.11:3306) succeeded Master 10.5.0.10(10.5.0.10:3306) is down! Check MHA Manager logs at 1a83c543536c for details. Started automated(non-interactive) failover. The latest slave 10.5.0.11(10.5.0.11:3306) has all relay logs for recovery. Selected 10.5.0.11(10.5.0.11:3306) as a new master. 10.5.0.11(10.5.0.11:3306): OK: Applying all logs succeeded. 10.5.0.12(10.5.0.12:3306): This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. 10.5.0.12(10.5.0.12:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.5.0.11(10.5.0.11:3306) 10.5.0.11(10.5.0.11:3306): Resetting slave info succeeded. Master failover to 10.5.0.11(10.5.0.11:3306) completed successfully. ```