Hi there,

We've got about 500,000 accounts on our website. We store a bunch of details on users and every now and again (every few months) we like to send an email to all of the users.

The problem I'm having is the user count is going up, so more emails need to be sent. Our current set up doesn't seem to be able to cope with this.

What's the best way to go about this? This is the current method I use (PHP/MSSQL):
- Select 250 emails from the database who we haven't sent an email to yet
- Loop through every email, and send the email off
- Update the database (MSSQL) to say that the email has been sent

I currently use Cent OS 5 with cPanel/Exim for this and we now have a huge queue on the server (of about 70,000 emails) -- this would be way higher but I eventually stopped it after I noticed the emails were backlogging.

What's the best configuration for sending an email to this many users? Should I be using PHP to do this? Should I use something else, perhaps Python (I heard this was good)?

Would I be right in thinking that using a SMTP server (rather than php's mail()) would be a better option? What would be the best Windows SMTP server to handle this (I only have one Linux server with cPanel/Exim)?

Given a suitable Windows SMTP server, would it be better to set up multiple SMTP servers rather than a single one? I have around 10 Windows based servers that I could use to 'spread the load'.

Looking for some input on this.

Regards

That's a lot lot of emails.

As you said you can do it in batch - every 250 emails per 2-3 minutes would be good (put sleep(120) after each batch). It does not matter which language you are going to use.

If you use php, you can use a php class like phpmailer or similar - those classes can easily use sendmail, smtp, redundant smtp etc.

Python, or any scripting language that supports threading would your best bet with this many emails.

A sleep is insufficient and sloppy coding for a thing such as this.

My suggestion is either writing it perl or python, or even writing a c or c++ app that compiles on either linux or windows, and do it that way, threading this application would be your best route, and use SMTP.

Not using threading would be slow, and insufficient with these numbers, and rather rubbish.

But the point is not to overload the smtp server.

It doesnot matter if you use sleep or do it with javascript or run the same script many times concurrently etc.

But the point is not to overload the smtp server.

It doesnot matter if you use sleep or do it with javascript or run the same script many times concurrently etc.

Point is, it would be threaded, and pretty sure the person coding it would know that the best way to do it would be using a queue. Make it work just like printer pooling (an example, i know we aren't printing)

EDIT: Probably shouldn't even use just one SMTP server, probably a couple, for 500k emails to be sent out.

Thanks for the suggestions guys.

The next question is finding a suitable and best Windows SMTP server.

Any suggestions for this? I believe Microsoft have an SMTP server you can install; would this be sufficient?

Thanks for the suggestions guys.

The next question is finding a suitable and best Windows SMTP server.

Any suggestions for this? I believe Microsoft have an SMTP server you can install; would this be sufficient?

It would be efficient enough yes.

If I was you I would look for some sort of system currently in place and that you can purchase, these will usually send a limited number of emails each time, so that it doesnt conflict with the server.

You could code one yourself and simply make it run about 50 at a time and include a simple next button - similar to how WHMCS does it.

As PHP don't have multithreading itself there is nice way to simulate that with pcntl_fork. For this task i will try to use that combined with cron, since you're sendind emails regularly.