Results 1 to 16 of 16

Thread: APF / BFD bug

  1. #1

    APF / BFD bug

    Greetings,
    While doing a random, manual check on my server, I found a brute force attack in progress that had been going on for some time. I have APF and BFD installed - BFD had *tried* to firewall the IP addr, but an apparent bug caused it to put a mal-formed IP addr in the APF deny file. The mal-formed IP addr wasn't accepted by the iptables stuff, so .... BFD thought it was firewalled, APF thought it was firewalled, but it wasn't ... When I found the attack in progress, it had been going on for just over an hours - many, many 1000's of log-in attempts (particularily root).

    So, here's what the first few lines of encounter looked like:

    Nov 17 22:46:58 plesk sshd[15146]: reverse mapping checking getaddrinfo for 66-195-205-25.static.twtelecom.net failed - POSSIBLE BREAKIN ATTEMPT!
    Nov 17 22:46:59 plesk sshd[15145]: reverse mapping checking getaddrinfo for 66-195-205-25.static.twtelecom.net failed - POSSIBLE BREAKIN ATTEMPT!
    Nov 17 22:47:01 plesk sshd[15146]: Failed password for root from ::ffff:66.195.205.25 port 40061 ssh2
    Nov 18 04:47:01 plesk sshd[15148]: Failed password for root from ::ffff:66.195.205.25 port 40061 ssh2
    Nov 18 04:47:01 plesk sshd[15148]: Received disconnect from ::ffff:66.195.205.25: 11: Bye Bye
    Nov 17 22:47:01 plesk sshd[15151]: reverse mapping checking getaddrinfo for 66-195-205-25.static.twtelecom.net failed - POSSIBLE BREAKIN ATTEMPT!
    Nov 17 22:47:02 plesk sshd[15145]: Failed password for root from ::ffff:66.195.205.25 port 39780 ssh2
    Nov 18 04:47:02 plesk sshd[15147]: Failed password for root from ::ffff:66.195.205.25 port 39780 ssh2
    Nov 18 04:47:02 plesk sshd[15147]: Received disconnect from ::ffff:66.195.205.25: 11: Bye Bye
    Nov 17 22:47:02 plesk sshd[15154]: reverse mapping checking getaddrinfo for 66-195-205-25.static.twtelecom.net failed - POSSIBLE BREAKIN ATTEMPT!
    Nov 17 22:47:03 plesk sshd[15151]: Failed password for root from ::ffff:66.195.205.25 port 40172 ssh2
    Nov 18 04:47:03 plesk sshd[15152]: Failed password for root from ::ffff:66.195.205.25 port 40172 ssh2
    Nov 18 04:47:03 plesk sshd[15152]: Received disconnect from ::ffff:66.195.205.25: 11: Bye Bye
    Nov 17 22:47:04 plesk sshd[15157]: reverse mapping checking getaddrinfo for 66-195-205-25.static.twtelecom.net failed - POSSIBLE BREAKIN ATTEMPT!
    Nov 17 22:47:05 plesk sshd[15154]: Failed password for root from ::ffff:66.195.205.25 port 39943 ssh2
    Nov 18 04:47:05 plesk sshd[15155]: Failed password for root from ::ffff:66.195.205.25 port 39943 ssh2
    Nov 18 04:47:05 plesk sshd[15155]: Received disconnect from ::ffff:66.195.205.25: 11: Bye Bye
    Nov 17 22:47:05 plesk sshd[15160]: reverse mapping checking getaddrinfo for 66-195-205-25.static.twtelecom.net failed - POSSIBLE BREAKIN ATTEMPT!
    Nov 17 22:47:06 plesk sshd[15157]: Failed password for root from ::ffff:66.195.205.25 port 40281 ssh2
    Nov 18 04:47:06 plesk sshd[15158]: Failed password for root from ::ffff:66.195.205.25 port 40281 ssh2
    Nov 18 04:47:06 plesk sshd[15158]: Received disconnect from ::ffff:66.195.205.25: 11: Bye Bye

    And here's what BFD eventually tried to insert into the APF deny list:
    (this is from the BFD log)
    Nov 17 22:50:02 plesk BFD(15539): {sshd} 66-195-205-25.static.twtelecom.net exceeded login failures; executed ban command '/etc/apf/apf -d 66-195-205-25.static.twtelecom.net {bfd.sshd}'.


    So, BFD fed '66-195-205-25.static.twtelecom.net' to APF, which accepted it and put it into the deny file. But iptables wouldn't accept it, cause it's not a properly formated IP addr.

    When I saw the attack in progress, I tried to execute the ban command manually, and APF would come back and say "already exists in the deny list". I'm guessing that it's using a regular expression to grep for the addr in the list and 66-195-205-25 matched 66.195.205.25 because in a regular expression, the '.' means 'match any character'.

    So, the hacker was hacking away, unimpeded .. Once I figured out why the IP addr wasn't being blocked, even though APF said it was, I manually edited the deny list, removing the malformed IP and putting in the proper one, then re-started the firewall ... and the hacker was history, at least for now. But, this means that my server has a basic vulnerability because of this BFD/APF combo bug.

    So all ya'll out there running BFD/APF - BEWARE !!!!

    - john silvey

  2. #2
    Join Date
    Mar 2003
    Posts
    663
    Hiya, don't mind can you provide us the way you do it in step by step in case anyone experience this? Like the command to check for possible attacks and editing the deny list manually?

    Thanks for the update!

  3. #3
    Sure - to manually check for activity, go to /var/log and look at the messages and secure logfiles. These are text files. Be careful not to open/edit them and lock them up as system processes will be writing to them continually. I usually do a quick copy of the file-of-interest to a dummy file name, then open the dummy file in vi or emacs to browse it.

    The deny file is in /etc/apf and is called deny_hosts.rules. It's also an ascii file that you can edit. If you make manual changes to it, you need to restart APF by typing "/etc/apf/apf -r".

    To manually insert an IP addr to block, you can either edit the deny file, then restart the firewall, or more simply, execute the APF ban command:
    /etc/apf/apf -d 66.205.23.2 (change the IP addr to the one you're interested in)

    BFD is a relatively simple script that wakes up every once in awhile via cron, has a look at the files, and executes an apf -d command on an IP addr's that if finds that violate rules.

    Because of this bug, until there is a fix for BFD, automatically detecting brute force attacks is not completely working if you're using BFD as the only detector. I'm guessing that there are other detection utilities out there, but I'm not yet familiar with them ... time to learn !

    - js

  4. #4
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,683
    Configserver Firewall (CSF) has blocking built into it, which works reasonably well. It's a little trigger happy in this version so I suggest you turn off pop3 blocking until that resolves.

  5. #5
    Thanks - just had a look at it. It claims to be an 'exclusive' add-on for cpanel ... but it looks kind of like BFD and APF in terms of operation and installation. The only real thing that I could find that seemed to be cpanel specific is that it is that all ports associated with cpanel are properly pre-configured. If that's the only link between it and cpanel, then it doesn't seem that it's really exclusive to cpanel, and I could use it on my Plesk-based, RH9 system too? Do you think this would work ?

  6. #6
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,683
    The command line form would probably work on Plesk, but I'd ask the author before you use it as there are a lot of cpanel assumptions built into it and one of those may end up biting you if you're not careful.

  7. #7
    Thanks - I'll contact them.

    I tried to contact the BFD/APF author/website to submit a bug report, but their site looks half-dead. A bunch of the links don't work .. the forums won't come up .. the link to tech support produced a page-not-found. I found a working link to "sales" and sent a short note, but haven't heard anything back.

    Also, with a bit of help from somebody else, I discovered that I was not completely accurate in my description. IPTABLES *will* accept a host/network name. So, that wasn't the problem - the problem is that the host/network name is bogus (as warned in the secure log - "reverse name lookup failed"). The bug is actually that BFD picks up the bogus host/network name and executes the APF command with it, and since it's bogus, the actual IP does not get blocked. So, the fix for the bug would be to make sure BFD always picks up the actual IP addr, not the host/network name out of the appropriate logfile. So, it seems to boil down to a hacker being able to thwart BFD/APF simply by providing a bogus host/network name.

    - js

  8. #8
    Join Date
    Feb 2001
    Location
    West Michigan, USA
    Posts
    9,675
    Quote Originally Posted by digimon
    So, it seems to boil down to a hacker being able to thwart BFD/APF simply by providing a bogus host/network name.

    - js
    If that's true...that's beyond hilarious.

    --Tina
    ||| 99.999% Uptime SLA!!!
    Plenty of space and bandwidth to fit your needs!
    www.AEIandYou.com - - (WP Friendly - Premium Reseller Hosting and Cheap Dedicated Servers)

  9. #9
    Join Date
    Nov 2005
    Location
    Minneapolis, MN
    Posts
    1,648
    That address isn't supplied by the connecting machine; the log messages are a bit misleading. When SSHD gets an incoming connection it looks up the PTR record for the connecting machine, and then does a forward lookup for the name it gets back to make sure that the forward and reverse DNS records are in agreement. You will see those log messages if the name returned in the PTR record either doesn't have a matching A record or if the IP address is different than that of the connection initiator.

    The simple solution for this is to edit sshd_config and set:

    UseDNS no
    Eric Spaeth
    Enterprise Network Engineer :: Hosting Hobbyist :: Master of Procrastination
    "The really cool thing about facts is they remain true regardless of who states them."

  10. #10
    A couple of questions ...

    Quote Originally Posted by spaethco
    When SSHD gets an incoming connection it looks up the PTR record for the connecting machine
    - is this the same as reverse DNS lookup, where the originating machine is queried for it's domain name?


    Quote Originally Posted by spaethco
    and then does a forward lookup for the name it gets
    - and this is done through name-servers, all the way up to The Great Nameserver In The Sky, if necessary, right?

    So, can a person of ill-intent accomplish this by having a bogus reverse-dns setup on the server he is operating from?

    thanks,
    js

  11. #11
    Join Date
    Nov 2005
    Location
    Minneapolis, MN
    Posts
    1,648
    Quote Originally Posted by digimon
    - is this the same as reverse DNS lookup, where the originating machine is queried for it's domain name?
    The reverse DNS lookup hits the servers that are authoritative for that netblock; in most cases that is not the machine originating the connection.
    Quote Originally Posted by digimon
    So, can a person of ill-intent accomplish this by having a bogus reverse-dns setup on the server he is operating from?
    That person would need to have the netblock owner's DNS servers serve up a PTR record without a corresponding A record match. Usually this is a configuration oversight and not a malicious action.

    Like I said, the easiest way to get around this is just to disable DNS lookups for SSHD. The whole reason for that lookup is so that you could put hostnames in hosts.allow and hosts.deny, and SSHD would perform the cursory DNS checks (forward and reverse) to make sure a host is who they seem to be. Since nearly everybody references IPs directly for firewall/allowed-host applications these days, the DNS lookups are just unnecessary processing baggage anyway.
    Eric Spaeth
    Enterprise Network Engineer :: Hosting Hobbyist :: Master of Procrastination
    "The really cool thing about facts is they remain true regardless of who states them."

  12. #12
    Thanks ... is there any other reason to resolve IP's to domain names (other than human reading convenience). It seems that finding non-matching names has some value in terms of identifying suspicious activity. On the other hand, it seems like it would slow things down - all the look-up activity that is going on.
    - js

  13. #13
    Join Date
    Apr 2002
    Location
    Troy, MI
    Posts
    309
    http://www.r-fx.ca/downloads/sshd
    mv -f sshd /usr/local/bfd/rules

    make sure your running BFD 0.9 and APF 0.9.6; if so and issue continues please empty /etc/apf/deny_hosts.rules (:> /etc/apf/deny_hosts.rules) and then download above mentioned module and it should correct the issue.
    Ryan MacDonald
    Lead Administrator | TotalChoice Hosting
    Choice Does Matter! | Serving over 26,000 clients

  14. #14
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,683
    I've seen this trick too. I think it works by having the reverse DNS for the IP in question not resolve to anything, from memory!

  15. #15
    rfxn: thanks - that looks like it should do the trick. I've replaced my orig sshd rule file with this one.

    So, I'm running paranoid - our server was hacked a few months ago, and I've been keeping a close eye on it since. One thing I noticed - when lines such as the following show up in my /var/log/secure file, about 60-70% of the time, a brute force attack is eventually launched from the same IP. And the reverse is even more true - every brute force attack is almost always preceded by this ... maybe 10% of the brute force attacks occur without this occuring first.

    Nov 15 06:36:49 plesk sshd[21643]: Did not receive identification string from ::ffff:65.160.106.85
    Nov 15 06:36:49 plesk sshd[21644]: Did not receive identification string from ::ffff:65.160.106.85

    There is usually from one to three occurances of the above message for a given IP. I'm assuming these messages indicate a port scan, or at least a probe of the SSH port. The attack follows anywhere from a few minutes to a day later, but it's very consistent. So, for the past few months, I've been manually executing the APF ban command on any such IP. My brute force attacks dropped to 10% of what they used to be.

    While the past few months have been a learning and evaluation session, it's time to automate the policy of immediately banning an IP that generates such a message in my secure file. It seems that BFD would work just fine for this. If I add a new rule file in the BFD rules directory and tweak a copy of the sshd file, such that it is looking for the "Did not receive identification string ..." string and set the TRIG = 1, could I accomplish this? It looks like BFD uses any file it finds in the $RPATH directory.

    - js

  16. #16
    rfxn - this worked - I was able to clone the sshd rule file and tweak it such that BFD now bans any IP addr which generates the 'no identification string' message in the /var/log/secure file. TRIG is set to '0', so a single occurance of this will result in an IP addr being firewalled. I didn't dig too deeply in the BFD code, but it looks like it also needs a user name to go with an IP addr for an event to be logged - I hard-coded the dummy user name "sshd_no_id" - so it still logs it as 'too many failed log-in attempts', but ... it's working.

    Quote Originally Posted by digimon
    If I add a new rule file in the BFD rules directory and tweak a copy of the sshd file, such that it is looking for the "Did not receive identification string ..." string and set the TRIG = 1, could I accomplish this? It looks like BFD uses any file it finds in the $RPATH directory.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •