Here is the scenario...
You have a 3 node galera (g1,g2,g3) cluster and there is some network maintenance which stops connectivity between the nodes causing them to lose their quorum and stop serving.
To fix this you disable 2 nodes (g2,g3) and bring just one back (g1)...
you should be able to bring the 1 node back with the following command:
galera_new_cluster
If that causes a failure, you may also need to update a line in your /var/lib/mysql/grastate.dat file.
to do this change:
safe_to_bootstrap: 0
to
safe_to_bootstrap: 1
now run the galera_new_cluster command again and it "should" work...
Once your network settles down and connectivity is regained you will have 2 galera nodes which will no longer gracefully start up, what do you do?
Feel free to use the following hammer:
On g2
cd /var/lib/mysql
rm grastate.dat
rm ib_logfile*
sudo -u mysql mysqld --wsrep_sst_donor=g1
where "g1" is the wsrep_node_name of your working/primary galera node
follow the above instructions for g3 and you should be all set...
