Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Redundant mail servers

[allera]

I've got a problem and I'm not sure how to go about solving it.

I plan to use 2 mail servers, each at different network centers on different lines. mail1 will handle normal everyday email, and should mail1 go down, mail2 starts accepting all email until mail1 comes back.

That's easily doable with MX records. The problem is, how do I get mail2's email to mail1 when mail1 resurfaces?

Or, how can I set it so that when people check their email, it checks mail1 and mail2 (I don't think that will work, but it's an idea).

Another problem, when mail1 is down and mail2 is accepting email, how do I get customers to check mail2 for their new email while mail1 is down without them making any configuration changes?

What I'm aiming for is a transparent redundant mail system. The customers won't know if the email is coming from mail1 or mail2, all they know is it's coming from "mail.whatever.com." In the background, mail1 always provides the email, and mail2 kicks in when mail1 goes down. I don't really care about the data on mail1 so long as the new email gets sent to mail2 and customers get the new email off of mail2. When mail1 resurfaces, mail2 sends its unPOP'd captured email over to mail1 and customers automagically get their mail from mail1 again and mail2 stands by again, empty and waiting.

I would rather the solution be as bandwidth friendly as possible, aka not mirroring servers continuously. Mirroring periodically might be a solution. I would also like to stay away from NFS, but if that's the only way, that's the old way...

I know this problem has been solved before but I cannot find it on the internet anywhere. All the forums I've tried had threads with the problem, but no posted solutions.

Don't be shy, even if you don't know how to fully do it, if we can get pieces together maybe we can work the whole system out.

I appreciate your help!

[Jm4n]

I'm not sure if you're planning on using Sendmail or (better yet) Qmail, but in either case you should be able to configure mail2 as a relay-only mail server.

For example, when mail2 (for whatever reason) receives an incoming message, it simply relays/forwards it to mail1. Of course if mail1 is down, mail2 would simply queue the messages for a configurable amount of time.

Now, if the end user is unable to connect to mail1 to *retrieve* their mail, redirecting them to mail2 would not work in this case (since it would be a relay-only server with no POP facilities). But they can rest assured that their mail has in fact been accepted...

Another option, if both machines are in the same facility (which I'm guessing they aren't, given your bandwidth concern), would be to use ip masquarading/forwarding. Actually I think this could be done remotely as well...

You could set up a gateway box. Its IP (externally accessible, eg a public IP) is what the users are given. For example, mail.domain.com points to an IP on the gateway. The gateway's only function is to be a gateway (plus firewall if desired).

Now, say you have two mail servers both behind the gateway box. mail1 and mail2 each have an internal IP.

The gateway runs a cron to periodically check mail1. Normally, gateway forwards (via ip forwarding) all port 25 and 110 requests to mail1. If the cron detects that mail1 is down or inaccessible, it updates the forwarding rules to point to mail2 instead.

Perhaps (via NFS or coda) you could have a fileserver hold the mail centrally, where the POP/SMTP servers do not actually store data...

Anyway, the ip forwarding can be done easily enough with ipchains and the ipmasqadm family of tools; For example, I forward all requests on port 7777 to my sometimes-windows box for Unreal Tournament games on my home network This could very easily be configured to automatically switch, and I believe you can in fact do this with external IPs as well (eg, mail servers in separate facilities).

Now, OTOH, if the network leading to the gateway box is down... that will break the whole system even if mail2 is perfectly accessible...

Just a thought anyway

<EDIT>
Further thinking... suppose you set up mail2 as the actual gateway box as well. If mail1 is okay, forward the requests on to mail1, and mail2's mail system is not touched. If mail1 is down, however, mail2 accepts the mail itself until such time it can forward them to mail1...

Something like this will require a lot of planning/thinking to make sure everything has been considered. I'd be happy to ponder this some more, given the following:

- What OS and mail software are you using/will you use?
- Is a third gateway box out of the question?
- Are they to be in the same facility?
</EDIT>

[allera]

Originally posted by Jm4n
Further thinking... suppose you set up mail2 as the actual gateway box as well. If mail1 is okay, forward the requests on to mail1, and mail2's mail system is not touched. If mail1 is down, however, mail2 accepts the mail itself until such time it can forward them to mail1...

Something like this will require a lot of planning/thinking to make sure everything has been considered. I'd be happy to ponder this some more, given the following:

- What OS and mail software are you using/will you use?
- Is a third gateway box out of the question?
- Are they to be in the same facility?

Bingo! That's the solution I was looking for! I never thought of using the second mail server as the gateway. Although, what happens when mail2 is down? It won't forward the user to mail1 like it should. Then you're kinda stuck again, but at least I think we're getting closer.

OS: FreeBSD 4.x (probably 4.4 by the time we put all of it in motion, 4.3 for now)

Mail: Qmail of course (with vpopmail)

The only problem with the gateway box is that if that network were to go down, both mail servers are SOL. Even if I were to put mail1 in one NOC, mail2 in a second NOC, and gateway1 in a third NOC, the third NOC better have a damn good uptime.

The two mail servers will be at different facilities on different provider lines (one on UUNet while the other on Qwest, for example). It just so happens that the NOCs I'm looking at are set up that way already, it wasn't a prerequisite of mine. I suppose that way if either backbone has issues, the other will pick up the slack for us.

I think that gateway is the key though. I would just use an existing server somewhere and use an IP just for mail.whatever.com and forward requests from that box to mail1 and mail2.

Then again, if I found a network center with as good an uptime as that one gateway box needs, I wouldn't need to go through all of this.

Sounds good though, thanks for the help! I'll let you know how it proceeds from here.

[Jm4n]

Now that I think about it, the gateway idea probably isn't the best. If I understand MX records correctly, the sender's mail server will always try the higher-priority server first, right?

So mail1 is higher priority over mail2. Users POP using mail1.

mail2's only job is to relay mail on to mail1, nothing else (no POP). If and only if mail1 is down will mail2 actually receive any messages (or if it's unreachable by a particular sender anyway).

Qmail I think will retry up to 72 hours before rejecting a message, which hopefully should be more than sufficient.

Going back to the gateway idea -- I don't know how to do IP forwarding in FreeBSD. I do rather like 'ipfilter' as a firewall, but have never tried forwarding with it. I'm sure it's possible...

So with FreeBSD acting only as a gateway (no web services etc running on it), you should have no trouble keeping this box up 100% of the time. As far as network uptime, I have to plug RackSpace here; they have quite a few different backbones, and I've not had a problem yet with them. You could lease a dedicated server with them, running FreeBSD (I think they offer this; if not, perhaps RedHat Linux using the ipmasq tools I had mentioned earlier would work for this purpose?)...

Having the three systems in three separate locations could lead to new problems; for example, all 3 boxes might be perfectly accessible, but perhaps the route from one to the others might be down (a particular router somewhere perhaps)...

So I'm actually thinking the gateway idea isn't the best one. The simple "relay-only" SMTP server on mail2, who has a lower MX priority, forwarding everything over to mail1 and/or queuing it for later delivery, should be great.

If mail1 goes down, users won't be able to pop but their mail will be accepted rather than bounced or queued at the sender's server...

There are so many possibilities here. The gateway idea would give you the option of letting users POP from either server transparently, but you would necessarily have to replicate for this to work... but simply relaying from mail2-->mail1 would definately prevent bounces when mail1 is out.

[allera]

So I'm actually thinking the gateway idea isn't the best one. The simple "relay-only" SMTP server on mail2, who has a lower MX priority, forwarding everything over to mail1 and/or queuing it for later delivery, should be great.

I just might end up doing this.

If mail1 goes down, users won't be able to pop but their mail will be accepted rather than bounced or queued at the sender's server...

This was my initial intent (to have them be able to get mail no matter what), but I suppose I could have those die-hard emailers change their email preferences while mail1 is down and let everyone else wait until it comes back up...

There are so many possibilities here. The gateway idea would give you the option of letting users POP from either server transparently, but you would necessarily have to replicate for this to work... but simply relaying from mail2-->mail1 would definately prevent bounces when mail1 is out.

I'll still research it further. I still like the gateway idea and I may still go with it, simply for the "transparent" look. I'll let you know what we ultimately decide and how we implemented it so that others with the same problem will profit from it.

Thanks again for all of your help!