Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Server being taken down by Joe Job

[Tina J]

We have been experiencing a massive Joe Job attack against 6 of our servers over the last 72 hours. This is not a normal spammer type Joe Job...this is a very deliberate and malicious attempt to take out these 6 servers. Setting everyone's default email to :fail: does little to help, since the servers still have to process 100s of 1000s of requests per minue. It started to ease up yesterday, but today they are back at it again in full force.

Does anyone have any real solutions or suggestions to get this under control?

I'm at my wits end and this has cost me so much money personally and my reputation, not to mention the 1000s and 1000s of people who depend on their email for business. I wonder if the people who do these sorts of things realize how many people they affect.

--Tina

[MatthewN]

Hello,
Have you investigated the use of SPF records and checking for those and filtering out emails that dont match the originating (approved) server. The format for these is...

"v=spf1 a mx ~all"

I would read up about it first at http://spf.pobox.com/index.html

I havent dealt too much with joejob attacks but the information above may be of assistance.

[Tina J]

Originally posted by Stormhosts
Hello,
Have you investigated the use of SPF records and checking for those and filtering out emails that dont match the originating (approved) server. The format for these is...

"v=spf1 a mx ~all"

I would read up about it first at http://spf.pobox.com/index.html

I havent dealt too much with joejob attacks but the information above may be of assistance.

Wow, how did your post get ahead of mine...when its my thread? That's weird.

Yes, we have SPF records and check them. THe problem is that these are bounced emails, from legitimate mail servers saying "this email address isn't on our network".

--Tina

[MatthewN]

That is weird... I will report it so the mods can put them back in order. <<< All fixed now. >>>

It seems that the receiving mail servers are not doing tests for SPF's if it's allowing them in to their networks so that they bounce back to the "so called originating addy".

Do any of the bounce back messages have the email headers of the originating email? And if so, do the headers list the originating IP of the original emails? and if they do... are the IP's the same for all emails (obviously you cant check thousands of emails) but for a selection of maybe 10 - 20 messages?

If so then I would report the IP immediatly to the net block owner and would try call.

Hopefully you can find this information if it hasnt been forged and is present.

[Tina J]

Its forged and not usable.

--Tina

[JohnCrowley]

If you're using sendmail, you could try rejecting email in the access file for the affected domains, and only allow in selective addresses in the virtusertable.

If it does not stop, set the MX's of affected domains to a separate server, and configure that server to relay back to the actual servers these accounts are on for valid addresses.

- John C.

[Tina J]

Originally posted by JohnCrowley
If you're using sendmail, you could try rejecting email in the access file for the affected domains, and only allow in selective addresses in the virtusertable.

If it does not stop, set the MX's of affected domains to a separate server, and configure that server to relay back to the actual servers these accounts are on for valid addresses.

- John C.

Ugh - we're talking 1000s of domains. I'm not sure if this can be automated, but to manually set MX records of those would take days.

--Tina

[JTY]

Rejecting would probably do little, as you'd still have the SMTP connections clogging your servers.

Unfortunately, I don't have any good advice to give.

[Tina J]

Originally posted by JTY
Rejecting would probably do little, as you'd still have the SMTP connections clogging your servers.

That's EXACTLY the problem.

--Tina

[JohnCrowley]

Originally posted by AH-Tina
Ugh - we're talking 1000s of domains. I'm not sure if this can be automated, but to manually set MX records of those would take days.

--Tina

You could probably easily script a mass update of the zone files to change MX's that are local to go to a gateway server, that then filters and delivers on to the actual servers. Not the simplest approach, but if your servers are getting choked out (due to so many domains on each server getting bounces) then it's at least one possible solution.

I'll refrain from the arm chair quarterback, back seat driving comments on number of domains per server to limit the severity of such email DDOS's (if it really is targetted and not just a coincidence of spammers choosing similarly located domains).

- John C.

[datums]

Hello Tina

So the NDRs are coming back to 1000's of domain which are hosted on these 6 servers ?

The only thing I can think of is to throttle the smtp connection from ips that are sending a large number of emails to your the specifc server.

Are the NDRs coming from Yahoo/Hotmail ?

This would allow you to send back a 400, and the email will be retried. At this time you would need to increase the number of smtp connections you can accept.

Yes this type of attack sucks.
I take it email is being delayed.

[Tina J]

Originally posted by datums
Hello Tina

So the NDRs are coming back to 1000's of domain which are hosted on these 6 servers ?

The only thing I can think of is to throttle the smtp connection from ips that are sending a large number of emails to your the specifc server.

Are the NDRs coming from Yahoo/Hotmail ?

This would allow you to send back a 400, and the email will be retried. At this time you would need to increase the number of smtp connections you can accept.

Yes this type of attack sucks.
I take it email is being delayed.

Its not 1000s of domains...its 1000s of email addresses. Say blah.com is hosted on one of the attacked servers...the Joe Jobber is sending email with a replyto of 23432@blah.com, aisfkljflk@blah.com, kjdsf@blah.com....100s of addresses that logically wouldn't exist for that domain. The emails are being sent to bad email addresses, so that network is bouncing all those emails back to us.

So, if there are 200 domains hosted on one of the server that is being attacked...there's probably 100,000 fake email addresses and 1,000,000 bounced emails for those 200 domains.

The bounced emails are coming from many, many, many different networks.

Its more than just email being delayed...the SMTP connections are overloading the server.

--Tina

[CD Burnt]

like was said above, could you send all mail to a central powerful server, and strip out all mail that has the characteristics of a bounced email? like those that "appear" to be from Mail Delivery Subsystem, MAILER-DAEMON, etc. ?

[ikeo]

Sounds like changing your MX records for the time being would work temporarily. To a single server and do your processing there.

This would at least free up the 6 webservers.

Also, if you're receiving the bounced messages you can still pull the originating IP address and additional helpful information. Look at some of the messages you are receiving and post them on the board. A sample of 5 or 6 messages from the headers should be sufficient for the moment.

The e-mail address itself maybe forged, but we could collect more data on the method they are using from the headers...

Also, what system are you using for your e-mail? Sendmail, Qmail, Postfix, Exim, etc.

Or are they forging your IP address and your domains in their mailings?

You say its forged and not usable, but could you please post them anyways? If you want real answers we have to have real data to parse through.

Another option would be to have a catchall address for the non-existant users on the systems that go to /dev/null/ that in conjunction with a ConnectionFactorByMX limit should throttle them down to a trickle on your system and backlog theirs.

But if you want to find out who's doing this, at least post the header information.

[Steven]

http://configserver.com/free/eximdeny.html

maybe something like that?