Add node back into galera cluster (brute force)

admin

Here is the scenario...

You have a 3 node galera (g1,g2,g3) cluster and there is some network maintenance which stops connectivity between the nodes causing them to lose their quorum and stop serving.

To fix this you disable 2 nodes (g2,g3) and bring just one back (g1)...

you should be able to bring the 1 node back with the following command:
galera_new_cluster
If that causes a failure, you may also need to update a line in your /var/lib/mysql/grastate.dat file.
to do this change:
safe_to_bootstrap: 0
to
safe_to_bootstrap: 1
now run the galera_new_cluster command again and it "should" work...

Once your network settles down and connectivity is regained you will have 2 galera nodes which will no longer gracefully start up, what do you do?

Feel free to use the following hammer:

On g2

cd /var/lib/mysql
rm grastate.dat
rm ib_logfile*
sudo -u mysql mysqld --wsrep_sst_donor=g1

where "g1" is the wsrep_node_name of your working/primary galera node

follow the above instructions for g3 and you should be all set...