Hello,

I have a client with a massive mailing list and when ever it sends out email, the load goes to around 60-80 (100% CPU usage).

What can I do to lessen the strain on the server.

Celeron 2.4Ghz, 512mb DDR Ram, etc. I cannot upgrade hardware.

ANY AND ALL SUGGESTIONS ARE WELCOMED!

Thanks and God Bless!

have the stagger the mailings over a period of a few hours instead of dumping them all into the queue at once.

Originally posted by eBoundary
have the stagger the mailings over a period of a few hours instead of dumping them all into the queue at once.
or ask him/her politely to dump them all in the queue at 3am or something...

alex

I agree.

Isn't using 100% CPU against your TOS?

Hello,

LOL, The thing is, he owns the server and is using it for specifically for emailing and no web page serving. And the subscriber count is too large to only send at a specific time

Thanks and God Bless!

then get a bigger server or stop sending spam :) or get multple servers and distribute the sending over multiple servers. The answers are all pretty simple.

Big mailing list :/

If the server is dedicated to sending emails then the high load is to be expected. You might be able to tweak a few things but it sounds like this guy is pushing out emails as fast as he can...

Hello,

I would reccomend using mojomail or another newsletter manager to span and queue the email list sending over a specific interval so that it doesn't slam your mail server daemon.

I setup the mailing list and it is setup correctly etc. What would you recommend I tweak?

Thanks and God Bless!

I don't know enough about the program(s) involved but you can usually optimise stuff do whatever.. What's this guy sending out anyway?

I suggest changing the permission of /var/spool/exim/msglog directory to 0
If you are using Exim.

Format,

What will that do? I am cerious.

Thanks and God Bless!

if u using cpanel u can limit a daily email sending in each account

Hello Blackmoont,

That is not my problem. He is the server owner, he doesn't need a limit. I need to find someone to get it working correctly. I come to find out two things.

1) Exim is using allot of resources (DUH!)
2) The cronjob runs fine for a few thousand then simply dies with a "Segmentation Error".

What does that error mean, any ideas?

Thanks and God Bless!

Hello,

One more thing. Would it be a bad idea to use SMTP instead of Exim?

Thanks and God Bless!

AHhh! I forgot again. LOL

Would downgrading to Exim 3 also be a better idea then Exim 4?

Thanks and God Bless!

When you change the permission of /var/spool/exim/msglog directory to 0 the only message you see in your exmin_mainlog is "cannot write to directory" this way, server load goes down a bit.
I did it before and it worked.
Do it at your own risk.

If you are using WHM then in the "Tweak Settings" area you can limit as to how many eMail are sent per hour. This gives immediate relief and all eMails will be sent -- just not as fast.

If they aren't doing any sort of website hosting then CPanel is itself adding undue load to the server. What they'd be better off with is a fresh OS install and a more efficient MTA installing, such as qmail and tweaking the server for email delivery if that's all it does.

If you limit the emails in Tweak Settings, you will get huge exim_mainlog which doesn't helps. IMHO

Hello,

If I do this, will the mailing list script (ARP3) still receive bounce back messages, but just won't store them in the queue?

Also, I disabled antivirus.exim from the exim.conf and the load only went to 8 when sending a block of 5,000 and then went down. Which is wonderful! ANY and ALL other tweaks are aksed for.

This is for Mail only, but I am not that much of a savy admin, so I need cPanel ;-)

Thanks and God Bless!

Any ideas on running the mailque from SSH?

Thanks and God Bless!

Originally posted by Format c: /u
When you change the permission of /var/spool/exim/msglog directory to 0 the only message you see in your exmin_mainlog is "cannot write to directory" this way, server load goes down a bit.
I did it before and it worked.
Do it at your own risk.

Just disable the logging:

Add this line to your /etc/exim.conf (Use the editor in WHM or Cpanel will overwrite your additions during updates)

message_logs = false

Restart exim: /etc/init.d/exim restart

You are done. No messy workarounds or heckling in the OS.

Originally posted by CCF Hosting
Hello,

If I do this, will the mailing list script (ARP3) still receive bounce back messages, but just won't store them in the queue?


Yes. You will always receive bounces. What you can do is to reduce the number of days they are stuck in the queue.

Go to WHM and use the Exim configurator (advanced mode) and add this line:

ignore_bounce_errors_after = 1d

and if you don't want them to be in the queue for too long, add:

timeout_frozen_after = 1d

Originally posted by CCF Hosting
Any ideas on running the mailque from SSH?

Thanks and God Bless!

Can you define "running the mailqueue" ? Do you mean forcing it to deliver?

Hello Sprintserve,

If I understand correctly, the bounce message will be sent to the return_path or what ever, but will NOT be stored in the mailque?

Also, I mean. Is there a command in in SSH to delivery the mailque?

My browser times out when I try it from the browser.

Thanks and God Bless!

It depends. One of the problems I seen before is invalid return paths... and you can easily end up with thousands (yes I seen it before) of bounced messages in the mail queue that can't go anywhere since the return path is invalid and it just get stuck there retrying.

You can run in SSH

exim -M <msgid1> <msgid2> ...

in order to deliver emails

Hmmm. Since the load seems to be able to withstand it's self now, I don't have to worry about locking the mail que.

Would it be OK to contact you by IM this one time? Because I am having a hard time explaining myself here, I apologize.

Thanks and God Bless!

The cronjob runs fine for a few thousand then simply dies with a "Segmentation Error".
I'm assuming you mean 'segmentation fault', not 'segmentation error' here.

Here's a couple of things that can be done.

Firstly, run /scripts/fixeverything on this server. This will go through and fix common problems (including mail list problems).

Secondly, you can modify these settings in /etc/exim.conf . While this won't provide a huge difference, it should provide enough of one.


smtp_connect_backlog = 50 <<-- lower this
smtp_accept_max = 100 << -- lower this


deliver_queue_load_max = 3 << -- lower this


Validate the email addresses on this list.

Consider splitting the list up a bit.

Here's the logistics of what's happening here, and WHY it's happening.

Client is sending mail to hundreds (maybe thousands, who knows) of users. Whether or not this is legitimate email isn't the issue, the issue is it's loading the server up.

Any time you send mail to THIS many users, you're going to have a ton of rejected emails (ie: people like me that send in fake email addresses anywhere that stuff's shown publicly).

When these emails are sent, not only is the email being SENT to the user, but if the user is providing a NOT valid email account, what happens is that this users email provider says "Ok, now we need to mail this user back and tell them this"... Great, HOWEVER most people don't want to see this (I mean, really, do you want to see hundreds of "your mail could not be sent because blah blah blah at blah blah blah.org isn't valid"? Most send it to a script which (again) is run by exim, parses this mail out and sends IT to the provider (which usually responds with a message, resulting in one hell of a big loop).

Since you're using CPanel, it's probably safe to assume that you're using mailman which is a pretty robust system. I've seen it handle 500+ users easily without problem. Try to have your client go into the administration section and see if they can pull up the full list, looking for known problem addresses.

HTH

Can I charge for the support? :D

That's a joke. But personally I am not inclined to people contacting us over our IM mediums set aside for customers. The difference is that on the board, I post when I have some time. If it is IM, it may disrupt service to customers.

Thanks for your reply. The thing is, I have set the list to only send 5,000 an hour and it takes 16 minutes and 23 seconds to send them all and the load goes to at most 8.

The mailing list is a 3rd party Perl program called Auto Response Plus (http://www.arp3.com/).

ARP3 has a solution for bounced email, where all bounced messages are piped into this script which will deal with them.

I believe the script only will record a failure on each attempt to send a mailing list message and NOT based on the attempts of the mail queue.

Here is the thing I DO NOT understand:




If a message is sent, and it bounces back with the mail queue being LOCKED, will a copy of the bounced message still be sent to the sender of the message OR will the bounce message bounce from the server because the queue is locked?




Something else I am having difficulty with is, I am receiving a copy of ALL bounced messages in the mail queue, which is A WHOLE BUNCH OF (*censored* ;-) ). How can I stop this, other then changing the nobody address to a :fail email address, because that is not even working.

Also wolfstream, I don't want to lower the mail queue server load max because the load does not go too high anymore, however what will lowering these do and what are these?

smtp_connect_backlog = 50 <<-- lower this
smtp_accept_max = 100 << -- lower this

The script is sending using Sendmail (Exim).

Thanks for ALL of your help and God Bless!

A couple of things briefly:

Perl is well known for high load and the like. Not perl precisely, but perl scripts, and generally poorly written ones. It's possible that the handlers calling various functions in this script are causing server load ( 8 isn't bad, I've seen a lot worse).

You may also look into the fact that your mailing client doesn't accept exim. The sendmail link provided isn't true sendmail, it's an exim link, there are a vast majority of differences between the binaries, especially when calling them from the CLI.

smtp_connect_backlog:
This specifies a maximum number of waiting SMTP connections


smtp_accept_max:
This specifies the maximum number of simultaneous incoming SMTP calls that Exim will accept.

This would help a little bit in this case, but I'm not sure how much.

You are running your perl script as nobody? You probably set up a forwarding rule in your WHM.

Login to WHM and click "Change System Mail Preferences"

Remove your email from the forwarding of Nobody's mails.

When emails are bounced, the sender should receive a bounced message when their email is valid.

Hello,

Here is an odd situation. When I run the script from ROOT, the load goes no higher then 8.

When running it from the user account, it goes above 11. Please help me understand why?

Thanks and God Bless!

Hello,

It has now gone above 21. I don't understand.

Thanks and God Bless!

There shouldn't be a difference whether it's running as root or as a user account.

I am not sure why, but there is. A load of 17 difference. I don't want to run it as root for security reasons.

If you have ANY advise, I would greatly appreciate it.

Thanks and God Bless!

Hello,

I am still having issues with this, I have even purchased another server with ServerMatrix and SAME issue. Except, it actually worked for about 8 hours then went down the tubes on me. Any ideas on this?

Thanks and God Bless!

Hi David,
Sorry to hear that your problems haven't been resolved after 2 weeks. Did you consider it may be a bad script? Afterall, you did upgrade the server hardware to something else.

I didn't have the time to look at the code, but perhaps someone else can and give it a quick critique.

If you're still having problems after moving servers, I'd say there's no question, the problem is the script.

Yes. It is highly likely a script problem.

Hello,

The thing is, there software is very well known in the market industry. It works wonders for other people, but I am one of the unlucky few. If it is the script problem, what can I do to locate the issue?

I can't give out the code, because it is licensed, but if anyone would like to help me, I can provide part of the Code via email or IM.

We did switch to ServerMatrix and the hardware is now Celeron 1.7Ghz instead of 2.4Ghz.

Software Web Site: http://www.arp3.com/

Thanks and God Bless!

If you know that the software works for others, but not you, perhaps you need to see if it is your implementation of it that is causing it.

You change a server, and you downgrade the CPU? I assumed you upgraded... then obviously it may still be the hardware issue.

Talk to me again, I may be able to help with a test. Details will be discussed.

if it is licensed software why are you not contacting the vendor for support?

Hello,

I have the datacenter running dionastic's on the server ATM. Yes, we did DOWNGRADE. One reason is the server hardware with BurstNet was faulty, Hard drive and memory were bad.

And see when everything DID work, the load was staying under 6, (better then BurstNet running as root). But now with this service (after it stopped working), I must run it as root to be able to keep it under 6.

Thanks and God Bless!

Hello,

I am in contact with Neil (vendor), but I am going back and forth trying to figure out; HARDWARE or SOFTWARE. My fingers are getting tired from all of the travelling I must do to figure out if it is either, or.

Thanks and God Bless!

Hello,

Here is the deal. I asked ServerMatrix to run a hardware test on the server, and guess what. They ddin't have time!!!

BUT** They did decided to replace my 1.7Ghz machine with a 2.4Ghz, for the same monthly price!

Can't beat that. However, I am still receiving the same issue.

I do plan on reinstalling the script with the same database, but new software install (same version) to see how that does.

Does anyone have any suggestions to optimize Exim (settings, config etc.). When it did work I received about 6.5 to 7 emails a second, however I would like to achieve 8-10.

Thanks and God Bless!

We notice Exim loads goes up alot when handling mailing lists, perhaps you could consider switching to Qmail, qmail seems to be more resource friendly. You could also try renicing the exim processes.

You could use SPRI from Rfxnetworks.com to do it for you automatically every X minutes. http://www.rfxnetworks.com. Simply add exim and associated processes to the low priority list.

Also ensure that in general your server is optimized to, what is the result of hdparm -tT /dev/hda? What kernel are you using (uname -a)?

When you're usinc CPanel you can't "switch" to qmail, you've got exim, and , well, exim, that's it ;)

As far as the mailing list, I hate to say it, but it really sounds like your software. two, three servers have given the same problem, there's not a lot of question, it's the software. Try the mailing list software supplied with CPanel, it's really NOT that bad.

Any time you have mail sent out to a large amount of people, you will find that you're going to have problems. I mean you're causing your server a LOT of work here. To boot, you're doing this rather large amount of work in a language that's known for causing server issues (well, MANY of the scripts written for it are), so it's not something that should come as a surprise really that the servers are loading down.

Have you EVER succesfully had this working?

Try this:

Switch off unnecessary mailer add-ons, Example virus scanning / spam assassin and such if you have them.

Do you get a lot of bounces each mailing? I found that bounces affect the load drastically. A server we managed spiked to loads of 120+ the other day before we rebooted it, not from sending... but apparently from email bounces. Someone spoof the headers to set the return path to a domain on the server, and we were receiving a few thousand bounces in a matter of minutes. Added processes such as Spamassassin went crazy and on an overdrive leading to the spikes.

And well, I have suspicion that the software can't handle real volume (did you check with the software vendor what's the max known list sent and compare with your list)? What did you say your size of the list is again?

The mailing list software that I use to manage a large list on one of my servers doesn't have a negative effect. With around 350,000+ subscribers (it's a popular teen site), the load will climb to about a maximum of 7 and CPU usage is almost at 100%, but it doesn't slow things down really. That server is only an Athlon 1800 with 512MB RAM too... You may want to ask the software developer to see if there's anything that you or they could do to decrease the load.

- Matt

Hello,

I have made a reply to each person that has asked a question for today. Please read bellow for my responce.

----------

mainarea:

What mailing list are you using? What is the teen site, I am interested to see how you intergrated the mailing list with your web site.

----------

sprintserve:

I have commented out antivirus.exim in exim.conf, and SPAMD is disabled. What changes do you recommend I make in the exim.conf/server? Perhaps that will help some.

I have heard of people running a list of over 500,000+ subscribers with ARP3 (the web site says NO limit). Our list size is 200,000 rougly.

We do not receive a whole lot of bounce backs, just your tipical mailbox full and invalid users. Would you recommend having the script use a special return-header when sending email by default?

----------

MattF:

Here are the results of the commands you asked me to run:

root@serv3 [~]# hdparm -tT /dev/hda

/dev/hda:
Timing buffer-cache reads: 128 MB in 0.35 seconds =365.71 MB/sec
Timing buffered disk reads: 64 MB in 1.15 seconds = 55.65 MB/sec


-----AND-----

root@serv3 [~]# uname -a
Linux serv3.*************.com 2.4.20-20.9 #1 Mon Aug 18 11:45:58 EDT 2003 i686 i686 i386 GNU/Linux

What does it all mean? Is that bad or good?

Thanks and God Bless!

Hello,

I forgot to ask, how can I optimize mySQL to run more effeciently, since subscribers are stored in mySQL? How do I setup deferred sending with EXIM?

Thanks and God Bless!

u can try the 2.6 test 9 kernel it improves performance in alot of things

Hello,

What do you mean "2.6 test 9 kernel "? What command do I use to ONLY update the kernal, because cPanel will break updating Perl and PHP I think.

Thanks and God Bless!

Originally posted by sprintserve
Try this:

Switch off unnecessary mailer add-ons, Example virus scanning / spam assassin and such if you have them.

Do you get a lot of bounces each mailing? I found that bounces affect the load drastically. A server we managed spiked to loads of 120+ the other day before we rebooted it, not from sending... but apparently from email bounces. Someone spoof the headers to set the return path to a domain on the server, and we were receiving a few thousand bounces in a matter of minutes. Added processes such as Spamassassin went crazy and on an overdrive leading to the spikes.

And well, I have suspicion that the software can't handle real volume (did you check with the software vendor what's the max known list sent and compare with your list)? What did you say your size of the list is again?

sprintserve,

Can you suggest on how to turn off external mail handlers such as MailScanner/SpamAssassin?

I've gone so far as to uninstall MailScanner with the "rpm -e mailscanner" command, and I removed SpamAssassin from my perl modules. But for some strange reason, they're still rooted into my sendmail and add themselves to the mail headers.

I've even spoken to Julian Field (the programmer of MailScanner) to see why this might be happening. But even he's at a loss as to why it's still showing up in my mail headers. When I say it's in my mail headers, it's showing up as such:

X-MailScanner-Information: Please contact the ISP for more information
X-MailScanner: Found to be clean
X-MailScanner-SpamCheck: not spam (whitelisted), SpamAssassin (score=-22.7,
required 5, AWL 0.00, BAYES_01 -6.60, IN_REP_TO -3.20,
MSGID_GOOD_EXCHANGE -5.70, ORIGINAL_MESSAGE -6.30,
QUOTED_EMAIL_TEXT -3.20)



But both MailScanner and SpamAssassin are completely gone from the system, and the system has been rebooted a few times as well to make sure no residual processes were running.

I'm guessing my only option is to somehow modify the sendmail.cf file to remove these from the header options?

Any help would be greatly appreciated on this one.

Originally posted by Jake Weg
u can try the 2.6 test 9 kernel it improves performance in alot of things

Indeed. But nobody knows what bugs exist and how stable it is. When it is not under widespread usage, bugs are likely to slip by unnoticed, as well as security holes if any.

Originally posted by CCF Hosting
Hello,

I forgot to ask, how can I optimize mySQL to run more effeciently, since subscribers are stored in mySQL? How do I setup deferred sending with EXIM?

Thanks and God Bless!

Paste the contents of my.cnf here. I am sure some tweaks can be added to help optimise it some.

Are you using Mysql 3.23 or Mysql 4?

Originally posted by kgibbons
sprintserve,

Can you suggest on how to turn off external mail handlers such as MailScanner/SpamAssassin?

<snip>

But both MailScanner and SpamAssassin are completely gone from the system, and the system has been rebooted a few times as well to make sure no residual processes were running.

I'm guessing my only option is to somehow modify the sendmail.cf file to remove these from the header options?

Any help would be greatly appreciated on this one.


How did you install Mailscanner in the first place? If you used the file from Layer2.cpanel.net, do remember that what Cpanel does is to basically integrate it within Exim. So normal means of uninstalling may not work.

If you had used the package from Layer2, you should use their uninstall script. Let me know if that helps.

Originally posted by CCF Hosting
sprintserve:

I have commented out antivirus.exim in exim.conf, and SPAMD is disabled. What changes do you recommend I make in the exim.conf/server? Perhaps that will help some.

I have heard of people running a list of over 500,000+ subscribers with ARP3 (the web site says NO limit). Our list size is 200,000 rougly.

We do not receive a whole lot of bounce backs, just your tipical mailbox full and invalid users. Would you recommend having the script use a special return-header when sending email by default?



The kernel version you are using is fine.

Did you find out what kind of servers those sending out 500,000 is using? 200,000 is a huge list nevertheless. And if it is using Mysql that would be the double whammy. Did you view your processes interactively when you are sending to see which are the ones that are taking up resources?

How many processes of Exim do you spawn?

How many processes of Mysql do you spawn?

What about Apache? Is the script run via the browser? If yes, that may take resources as well.

Did you ask the developer of the software what's the recommended hardware for a list of your size?

If you don't get many bounces, it's fine. But given that you are talking about 200,000+ list, it's relative. So how many bounces average?

Lastly, each of the above on the own may not cause it, but when added together, it may be an issue.

Originally posted by sprintserve
How did you install Mailscanner in the first place? If you used the file from Layer2.cpanel.net, do remember that what Cpanel does is to basically integrate it within Exim. So normal means of uninstalling may not work.

If you had used the package from Layer2, you should use their uninstall script. Let me know if that helps.

Actually, my server is a plain RedHat 8 system with no Cpanel. And I installed MailScanner from RPM. So as far as I would believe, the "rpm -e" should have wiped it out.

As far as wiping out SpamAssassin... I use Webmin as my control panel. So I removed SpamAssassin from the Others>Perl Modules section of Webmin. So that should be gone as well.

But somehow, someway... the MailScanner header is still showing up. And no clue why...

What mail daemon are you using?

Originally posted by sprintserve
What mail daemon are you using?

I'm using sendmail 8.12.8-9.80 along with "imap 2001a-15"

Hello,

mySQL spawns just one process, but that will pop-up every 30 seconds or so and take up 30% CPU. Exim makes as many threads as it can, I will guess about 5-6 a second (speed my script is sending out; some going into the mail queue for later delivery).

The script does not use Apache, but cron-job. On a 5000 sending block, perhaps 100 at the very most (the rest are just queued for later sending).

The ones that are sending 500,000 are using 2GB RAM and a dual XEON processor I believe.

I have reviewed the live processes, Exim for sure has perhaps anywhere from .8 to 4.5% of the CPU and mySQL as stated above.

Here is my my.cnf file:



[mysqld]
set-variable = max_connections=500


NOT MUCH IS THERE! :-)


As for the TOP display, well I need to figure out why I have 23,000+ messages in my mail queue and why Exim failed. I received an email from SM stating Exim and Named has failed, but server status states BIND down and everything else up.

And now Exim cannot complete a DNS lookup to send the email out, that is why it has failed. This is becoming a very large headache.

Thanks and God Bless!

Perhaps a Dual Xeon is what you need. Did you try reducing it even further? 1000 sending block.. and keep reducing until you reach an acceptable level?

Hello,

The load goes up almost instantly when it runs and then dies down after it is finished, so all that would do is decrease the time the load is running.

I am also going to reinstall it today to see how it goes.

Thanks and God Bless!

Does the script use a mySQL backend? Don't recall if already mentioned. If yes, set persistent connections to 'off' if you haven't already tried that.

Just a thought. ;)

Hello,

Yes it does use mySQL backend. What is the mySQL default?
What is the command stuff to place in the my.cnf file?

Thanks and God Bless!

Originally posted by CCF Hosting
Hello,

I have a client with a massive mailing list and when ever it sends out email, the load goes to around 60-80 (100% CPU usage).

What can I do to lessen the strain on the server.

Celeron 2.4Ghz, 512mb DDR Ram, etc. I cannot upgrade hardware.

ANY AND ALL SUGGESTIONS ARE WELCOMED!

Thanks and God Bless!

Celerons = rejected P4's. You might have ALOT better luck with either a true P4 2.4 or a Dual Xeon with a heck load of RAM (>2GB).

Just my 0.02 cents.

Hello,

Here is what I am thinking, lower the amount that is sent, increase how often it is sent and stop it from placing slow email in the mail queue. ANY ideas on stopping slow (like Netscape) (not failed DNS) from entering the mail queue or retry like twice instead of once?

Also, I am not sure if this was answered before but, if I lock the mail queue, would the sender still receive the bounce message, but it would not be stored in the queue?

If so, how do I lock the mail queue?

Thanks and God Bless!

remote_max_parallel parallel SMTP delivery per message

This feature may lower the load... It an exim config setting that allows one connection per unique host.

Phil: Any reason you brought a year old thread back to life?

oops! I must have used the search to find the thread which i forgot about...

Anyway it should be good information when others search :-)

hehe.

No worries