We have made a hot backup solution for our raq, I'd love for some people to ponder what we did and see if I missed anything here. Here is the script I run daily at 4AM.

#!/bin/bash
rsync -ae ssh --delete --exclude="mysql.sock" --exclude="ns*" /home/mysql/ root@[IP of my backup]:/home/mysql/

(in the command above, assume my machine names start with ns, gotta exclude the .pid file and the .err file)

rsync -ae ssh --delete /home/spool/mail/ root@[IP of my backup]:/home/spool/mail/

rsync -ae ssh --delete --exclude="name.of.main.server" --exclude="name.of.backup.server" /home/sites/ root@[IP of backup]:/home/sites/

The only "problems" I've found:

- Everytime we add a new domain on the main server, we have to add it on the backup server. Same for users. Which isn't a problem, since we don't do this too often. (users aren't actually needed for this, since the backup is just queuing mail for the main server, but if we ever had to switch to use the backup as our main server, it would be vital.)

- MySQL on the backup needs a restart to recognize the rsync'ed files. Flush privs doesn't do it. I just do a restart daily on the backup.

- Passwordless SSH as root is available from our main server to our backup. I can't think of any other way of scripting a backup without this. We keep a seperate backup copy of our main server, apart from the hot copy, so if things really went sour, we could just wipe both machines and be back up within ~2 hours.


What I'd really like to do is rsync as much of the cobalt as needed, so anything done on the GUI on our main server is mirrored on our backup. All I'd really have to do after that is replace the IP's and machine name of our main server with those of our backup. That would be easy. I just don't know where Cobalt keeps all this stuff, and I'm pretty sure I'd wreck the GUI on the backup. Anyone ever done this successfully?

Also, aside from this script, we have another that pings the main server from the backup. If the main is down, and the backup has an internet connection, we switch on named, which has everything DNS'ed to our backup, and queues emails till the main server gets back up. We have not implemented this one yet. Can anyone forsee any major problems with this scenario?

looks like a good solution, as a matter of point, cobalt keeps quite a bit of information in postgres, take a look at the databases you can do quite a bit with it.

Originally posted by driverdave
We have made a hot backup solution for our raq, I'd love for some people to ponder what we did and see if I missed anything here. Here is the script I run daily at 4AM.

#!/bin/bash
rsync -ae ssh --delete --exclude="mysql.sock" -- - Everytime we add a new domain on the main server, we have to add it on the backup server. Same for users. Which isn't a problem, since we

Well, we're using partly the same command in our backup system (but ours is quite a bit more advanced however).

I have a few recommendations:

1. Use the --numeric-ids parameter for rsync. This makes it work correctly even when you haven't quite got the /etc/passwd file synchronized yet.

2. Use -e 'ssh -C' to get compression (or use -z if you rather like compression there).

3. Use -pogt to get the file metadata correct.

Apart from that, I would recommend you to try and actually use your backup (i.e. to see that everything works from the backup).

For example you cannot really just rsync the MySQL databases (unless you use binary logging) - you actually have to use for example mysqlhotcopy to get a correct and valid copy of the MySQL databases.

If you want the "backup raq" to function identical to the original RaQ -- why not simply copy over all the files? (except ofcourse for the crontab/backup scripts/ip-address configuration).

Originally posted by driverdave
Also, aside from this script, we have another that pings the main server from the backup. If the main is down, and the backup has an internet connection, we switch on named, which has everything DNS'ed to our backup, and queues emails till the main server gets back up. We have not implemented this one yet. Can anyone forsee any major problems with this scenario?

Oops, forgot to answer the last part of your message:

If you just want to queue mail, this is clearly way overkill. Simply use two MX records on your domains for that.

If you also want to mirror websites, there's some things you need to be aware of:

1. First of all be sure that not just anything can trigger a DNS change. For example if the backup server loses it's internet connection (and thus cannot ping the production server), don't let it switch DNS.

2. To make the DNS switch work correctly - you have to have the backup server listed as an authoritative server for domain, and the production server itself _must_ be the _only_ other DNS server for the domain.

3. Further, you need to keep very low TTLs on your zones.

4. Remember to only include one NS record with the answers from the zone, when the backup has taken over (to prevent situations where people may be accessing both servers at the same time)

5. If you have sites that rely on databases, you need to include some extra wizardry to ensure that the two databases stay in-sync. And also so that you can update the production server from the backup server after a crash.

I've implemented something very similar to this earlier. It's used with our databased DNS system - where it automagically changes DNS records to point to the "up" servers, or the "least loaded" servers or the "geographically nearest" servers. It's quite effectfull actually!

Thanks for the suggestions. I have tested our backup raq, I've even gone through and checked out each site. Everything is working and mirrored properly.

By the way, there are several goals we have in mind with this server
- Queue mail for our main server if it happens to go down
- Serve HTTP/MySQL requests while our main server is down
- Replace our main server in the event of a total loss of our main server, with minimal effort.

MySQL is happy, as long as I re-start it after rsyncing the files. I'll look into mysqlhotcopy, I try to do this at a time where no-one should be updating MySQL, but mysqlhotcopy sounds like what we should be doing. That locks the databases while it copies, right?

The ping script I wrote only turns on DNS for the backup if it has an internet connection AND it can't find the main server.

I've done a bunch of tests playing with DNS, I think everything on that end should be fine. I will lower those TTLs though, I overlooked that. But other than that, our DNS is configured exactly like you described it.

Use -pogt to get the file metadata correct.
I'm not sure what that is, but I'll look into it.


If you want the "backup raq" to function identical to the original RaQ -- why not simply copy over all the files? (except ofcourse for the crontab/backup scripts/ip-address configuration).
This is my goal, but I just don't know where the raq keeps it's IP/Hostname configuration. If I could just rsync everything and exclude files with those settings, I'd be set.


The only reason I'm keeping track of users and passwords on the backup, but just queing mail is for the scenario wher we all of a sudden have to use our backup server as our main server. Keeping the users synced up will lower the pain of such a switch.

I've implemented something very similar to this earlier. It's used with our databased DNS system - where it automagically changes DNS records to point to the "up" servers, or the "least loaded" servers or the "geographically nearest" servers. It's quite effectfull actually!
That sounds like a nice setup. Do you have lots of users making updates to your databases? How do you figure out which servers to sync up from? Or do you have everyone update one server and sync to that?

Originally posted by driverdave
MySQL is happy, as long as I re-start it after rsyncing the files. I'll look into mysqlhotcopy, I try to do this at a time where no-one should be updating MySQL, but mysqlhotcopy sounds like what we should be doing. That locks the databases while it copies, right?

Correct!


This is my goal, but I just don't know where the raq keeps it's IP/Hostname configuration. If I could just rsync everything and exclude files with those settings, I'd be set.


It's set multiple places :-(

The most important are probably /etc/sysconfig/network and the /etc/httpd/conf/httpd.conf files.


That sounds like a nice setup. Do you have lots of users making updates to your databases? How do you figure out which servers to sync up from? Or do you have everyone update one server and sync to that?

Well, when I say "databased DNS" I actually mean that the _DNS_ itself is served from a database (run my own special version of Bind).

To spread for example MySQL across several servers, we use the normal replication feature of MySQL. Nothing "fancy" there :-)

Thanks again. I'll look into mysql hotcopy, and lower those TTLs.