We have been experiencing a massive Joe Job attack against 6 of our servers over the last 72 hours. This is not a normal spammer type Joe Job...this is a very deliberate and malicious attempt to take out these 6 servers. Setting everyone's default email to :fail: does little to help, since the servers still have to process 100s of 1000s of requests per minue. It started to ease up yesterday, but today they are back at it again in full force.
Does anyone have any real solutions or suggestions to get this under control?
I'm at my wits end and this has cost me so much money personally and my reputation, not to mention the 1000s and 1000s of people who depend on their email for business. I wonder if the people who do these sorts of things realize how many people they affect.
Originally posted by Stormhosts Hello,
Have you investigated the use of SPF records and checking for those and filtering out emails that dont match the originating (approved) server. The format for these is...
That is weird... I will report it so the mods can put them back in order. <<< All fixed now. >>>
It seems that the receiving mail servers are not doing tests for SPF's if it's allowing them in to their networks so that they bounce back to the "so called originating addy".
Do any of the bounce back messages have the email headers of the originating email? And if so, do the headers list the originating IP of the original emails? and if they do... are the IP's the same for all emails (obviously you cant check thousands of emails) but for a selection of maybe 10 - 20 messages?
If so then I would report the IP immediatly to the net block owner and would try call.
Hopefully you can find this information if it hasnt been forged and is present.
Last edited by SoftWareRevue; 04-13-2005 at 02:00 PM.
Originally posted by AH-Tina Ugh - we're talking 1000s of domains. I'm not sure if this can be automated, but to manually set MX records of those would take days.
You could probably easily script a mass update of the zone files to change MX's that are local to go to a gateway server, that then filters and delivers on to the actual servers. Not the simplest approach, but if your servers are getting choked out (due to so many domains on each server getting bounces) then it's at least one possible solution.
I'll refrain from the arm chair quarterback, back seat driving comments on number of domains per server to limit the severity of such email DDOS's (if it really is targetted and not just a coincidence of spammers choosing similarly located domains).
So the NDRs are coming back to 1000's of domain which are hosted on these 6 servers ?
The only thing I can think of is to throttle the smtp connection from ips that are sending a large number of emails to your the specifc server.
Are the NDRs coming from Yahoo/Hotmail ?
This would allow you to send back a 400, and the email will be retried. At this time you would need to increase the number of smtp connections you can accept.
Yes this type of attack sucks.
I take it email is being delayed.
Its not 1000s of domains...its 1000s of email addresses. Say blah.com is hosted on one of the attacked servers...the Joe Jobber is sending email with a replyto of [email protected], [email protected], [email protected]....100s of addresses that logically wouldn't exist for that domain. The emails are being sent to bad email addresses, so that network is bouncing all those emails back to us.
So, if there are 200 domains hosted on one of the server that is being attacked...there's probably 100,000 fake email addresses and 1,000,000 bounced emails for those 200 domains.
The bounced emails are coming from many, many, many different networks.
Its more than just email being delayed...the SMTP connections are overloading the server.
like was said above, could you send all mail to a central powerful server, and strip out all mail that has the characteristics of a bounced email? like those that "appear" to be from Mail Delivery Subsystem, MAILER-DAEMON, etc. ?
Sounds like changing your MX records for the time being would work temporarily. To a single server and do your processing there.
This would at least free up the 6 webservers.
Also, if you're receiving the bounced messages you can still pull the originating IP address and additional helpful information. Look at some of the messages you are receiving and post them on the board. A sample of 5 or 6 messages from the headers should be sufficient for the moment.
The e-mail address itself maybe forged, but we could collect more data on the method they are using from the headers...
Also, what system are you using for your e-mail? Sendmail, Qmail, Postfix, Exim, etc.
Or are they forging your IP address and your domains in their mailings?
You say its forged and not usable, but could you please post them anyways? If you want real answers we have to have real data to parse through.
Another option would be to have a catchall address for the non-existant users on the systems that go to /dev/null/ that in conjunction with a ConnectionFactorByMX limit should throttle them down to a trickle on your system and backlog theirs.
But if you want to find out who's doing this, at least post the header information.
There are no emails generated on your server when using that script and :fail:, that is how :fail: should work and the mail server is not then over loaded processing all the joe job emails, with :blackhole: the mail is still processed
The sending smtp server would be rejected at RCPT by your server, but the sender would get an error and bounce message generated from their mail server, not yours.