Page 1 of 3 123 LastLast
Results 1 to 25 of 61
  1. #1
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681

    IPMI -- public ips -- What the hell are you thinking?

    <rant>

    For the past year if not longer there has been a string of IPMI exploits. Hosts have been warned repeatedly to take IPMI off the public internet yet there is numerous providers who have not done this.

    I started getting emails last night/today from providers warning about the newest exploit. Now mind you this is DAYS after it was announced.. Come on seriously?

    What is wrong with you guys? Do you have any care or respect for your customers?

    Do some searching on various forums, there is a bunch of hosting companies who have had their servers wiped clean due to this most recent exploit in the past few days.

    If you offer dedicated servers, and ipmi that is still to this day publicly accessible. I will openly say you are a company no one should ever host with.. because you are flat out clueless.

    Its really not that hard to limit this stuff to a VPN.

    </rant>
    Last edited by Steven; 06-26-2014 at 03:27 PM.

  2. #2
    I liked what someone else said in the exploit thread, went something like "Get your IPMI accessible on from a VPN and be much better off, those customers that dont like this are customers you probably dont want." Which I think made a lot of sense.

  3. #3
    Thanks for your post , fortunately I was aware of the exploit before it get announced on the websites, 7 - 8 days ago , so we sent email to our colo customers and ask them to upgrade their firmware if the IPMI is accessible over public IP address . for dedicated servers , first we blocked the port 49152 on private network and started to upgrade firmwares and we get it done yesterday .

    We had zero report for a problem even for colo customers ...

  4. #4
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Quote Originally Posted by HSN-Saman View Post
    Thanks for your post , fortunately I was aware of the exploit before it get announced on the websites, 7 - 8 days ago , so we sent email to our colo customers and ask them to upgrade their firmware if the IPMI is accessible over public IP address . for dedicated servers , first we blocked the port 49152 on private network and started to upgrade firmwares and we get it done yesterday .

    We had zero report for a problem even for colo customers ...
    I repeat. If ipmi is on public ips and you offer dedicated servers you are doing your customers an injustice.

    I dont care if you blocked ports or whatever. There will likely be more exploits.

    Its time for you to be proactive. Only a matter of time before there is a private 0 day that you can't protect against.

  5. #5
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    With regards to VPN being the end-all-be-all... humor me this -- What prevents a rouge customer/signup from hijacking servers from the inside? Many if not most of the IPMI VPN solutions I've seen allow customers access to more than their own IPMI, e.g. large flat private network ready for easy mass exploit. I surely hope strong ACL's are implemented in any setup, and certainly worth investigating beyond 'oh we have a VPN, we're good'.

  6. #6
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Quote Originally Posted by FastServ View Post
    With regards to VPN being the end-all-be-all... humor me this -- What prevents a rouge customer/signup from hijacking servers from the inside? Many if not most of the IPMI VPN solutions I've seen allow customers access to more than their own IPMI, e.g. large flat private network ready for easy mass exploit. I surely hope strong ACL's are implemented in any setup, and certainly worth investigating beyond 'oh we have a VPN, we're good'.
    Agreed. But there is plenty of hosts that do not even give out access yet are on public ranges.
    Getting off public ranges is very important. Being public is just as easy if not easier than a vpn compromise.

    Aka zmap of the internet.

    You can make a home brew vpn solution that ties ipmi user and pass into vpn auth.. then you can restrict ip access... It is not that hard.

    Ionity has a similar setup with their SRX works great.

    Definately not rocket science to make something secure.
    Last edited by Steven; 06-26-2014 at 04:01 PM.

  7. #7
    Quote Originally Posted by Steven View Post
    I repeat. If ipmi is on public ips and you offer dedicated servers you are doing your customers an injustice.

    I dont care if you blocked ports or whatever. There will likely be more exploits.

    Its time for you to be proactive. Only a matter of time before there is a private 0 day that you can't protect against.
    Our IPMIs are not accessible over world and it's on private network But what if some one order a server to get VPN access and exploit servers ? that's why we blocked the port and start to upgrade the firmware

  8. #8
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Quote Originally Posted by Steven View Post
    Agreed. But there is plenty of hosts that do not even give out access yet are on public ranges.
    Getting off public ranges is very important. Being public is just as easy if not easier than a vpn compromise.

    Aka zmap of the internet.
    True...VPN will prevent drive-by attacks. However the recent mass-deletion event appears to be the targeting a specific provider. VPN is one extra step for a determined attacker, seeking to discredit their competition. I really hope that everyone takes a good look at their current setup, VPN or not.

  9. #9
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Quote Originally Posted by FastServ View Post
    True...VPN will prevent drive-by attacks. However the recent mass-deletion event appears to be the targeting a specific provider. VPN is one extra step for a determined attacker, seeking to discredit their competition. I really hope that everyone takes a good look at their current setup keeping that in mind.
    What sparked this thread is several drive bys we are dealing with atm. Not targeted.

  10. #10
    Quote Originally Posted by FastServ View Post
    With regards to VPN being the end-all-be-all... humor me this -- What prevents a rouge customer/signup from hijacking servers from the inside? Many if not most of the IPMI VPN solutions I've seen allow customers access to more than their own IPMI, e.g. large flat private network ready for easy mass exploit. I surely hope strong ACL's are implemented in any setup, and certainly worth investigating beyond 'oh we have a VPN, we're good'.
    It's not easy to configure, but you can limit a vpn session to accessing a specific internal IP address. We give customers a specific VPN login for each and every server they have, and each login only allows them to reach one server.

  11. #11
    Quote Originally Posted by Steven View Post
    Agreed. But there is plenty of hosts that do not even give out access yet are on public ranges.
    Getting off public ranges is very important. Being public is just as easy if not easier than a vpn compromise.

    Aka zmap of the internet.

    You can make a home brew vpn solution that ties ipmi user and pass into vpn auth.. then you can restrict ip access... It is not that hard.

    Ionity has a similar setup with their SRX works great.

    Definately not rocket science to make something secure.
    I have to completely disagree with your assertion that "it's not that hard". It's absolutely positively 100% WORTH THE EFFORT, but to claim it's not hard is completely wrong. It took our lead technician a lot of hours to get a vpn solution set up that he was happy with from a security and functionality standpoint. Time well spent, but I can understand why others have dragged their feet up to this point. That said, difficult or not, it absolutely must be done.

  12. #12
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Quote Originally Posted by funkywizard View Post
    It's not easy to configure, but you can limit a vpn session to accessing a specific internal IP address. We give customers a specific VPN login for each and every server they have, and each login only allows them to reach one server.
    That's the problem (in bold)... Most VPN solutions I've seen aren't nearly as tight. I'm glad you're doing it the right way.

  13. #13
    Quote Originally Posted by FastServ View Post
    That's the problem (in bold)... Most VPN solutions I've seen aren't nearly as tight. I'm glad you're doing it the right way.
    Given the fact that being able to access the IPMI login page is equivalent to giving someone root access on a server, after the string of exploits that were published a few months back, we decided we couldn't give clients any access to IPMI at all without something at least that secure.

  14. #14
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Quote Originally Posted by funkywizard View Post
    Given the fact that being able to access the IPMI login page is equivalent to giving someone root access on a server, after the string of exploits that were published a few months back, we decided we couldn't give clients any access to IPMI at all without something at least that secure.
    Exactly how it should be.

  15. #15
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    A few months ago. Someone who hangs out in a chat room I frequent did a global scan for ipmi.
    What he found was thousands of ipmi intances with admin/admin default login. Our ids has been seeing hundreds of scans per day on our tiny net blocks (/22x2). Which just recently ramped up.

    What I don't think some people realize is how bad this is going to get.

  16. #16
    I'd go a step further and say that *all* of your management should be on an isolated management network that's not internet accessible without a VPN or otherwise being on-site. I don't expose hypervisors to the Internet, I don't expose internal tracking tools or out of band management or anything of the sort. Does it make certain problems (like console access for customers) a bit harder to solve? Sure. Are those problems solvable? Absolutely - it really wasn't that hard. It absolutely boggles my mind that there are people out there with their IPMI/DRACs/iLOs/etc on the public Internet. That's just... extremely lazy. I don't know how else to put it.

  17. #17
    Join Date
    Aug 2004
    Location
    Kauai, Hawaii
    Posts
    3,799
    Quote Originally Posted by Steven View Post
    Do some searching on various forums, there is a bunch of hosting companies who have had their servers wiped clean due to this most recent exploit in the past few days.
    I've seen urpad.net and jonessolutions affected, both at Quadranet I think. Who else was affected?

  18. #18
    Join Date
    Aug 2003
    Location
    Edinburgh/London
    Posts
    5,789
    Semoweb too I think?

  19. #19
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Quote Originally Posted by gordonrp View Post
    I've seen urpad.net and jonessolutions affected, both at Quadranet I think. Who else was affected?
    Semoweb, there is someone else in a thread here (dedicated section) affected and I have 3 open tickets from 3 companies we are doing DR on right now that I can't release the names of.

  20. #20
    Join Date
    Mar 2009
    Location
    CA
    Posts
    9,350
    Quote Originally Posted by gordonrp View Post
    I've seen urpad.net and jonessolutions affected, both at Quadranet I think. Who else was affected?
    Urpad.net is not hosted with QuadraNet, they're in Coresite IIRC (Alameda)

  21. #21
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Quote Originally Posted by Dustin Cisneros View Post
    Urpad.net is not hosted with QuadraNet, they're in Coresite IIRC (Alameda)
    They are an RLT company so most likely in the Blacklotus space (thats where they moved into last year)

  22. #22
    Join Date
    Mar 2009
    Location
    CA
    Posts
    9,350
    Quote Originally Posted by Steven View Post
    They are an RLT company so most likely in the Blacklotus space (thats where they moved into last year)
    Makes sense, SemoWeb shows BlackLotus on the order forms.

  23. #23
    Quote Originally Posted by funkywizard View Post
    I have to completely disagree with your assertion that "it's not that hard". It's absolutely positively 100% WORTH THE EFFORT, but to claim it's not hard is completely wrong. It took our lead technician a lot of hours to get a vpn solution set up that he was happy with from a security and functionality standpoint. Time well spent, but I can understand why others have dragged their feet up to this point. That said, difficult or not, it absolutely must be done.
    Here's an important question for you: Are the IPMI controllers isolated from each other as well? If you're giving someone access to the web UI (or even if you're not), it's fairly easy to gain shell access to it. If you haven't secured the IPMI controllers from each other, you could then start attacking other machines.

    Granted, it requires a bit more knowledge to exploit, but it's not impossible.

  24. #24
    Join Date
    Aug 2007
    Location
    L.A., CA
    Posts
    3,710
    Quote Originally Posted by devicenull View Post
    Here's an important question for you: Are the IPMI controllers isolated from each other as well? If you're giving someone access to the web UI (or even if you're not), it's fairly easy to gain shell access to it. If you haven't secured the IPMI controllers from each other, you could then start attacking other machines.

    Granted, it requires a bit more knowledge to exploit, but it's not impossible.
    Thats why any real company now a days needs 2 networks.
    Public and Private.
    Each client would have a public VLAN and a private VLAN.
    The private VLAN would contain IPMI and any inter-server internal connectivity and isolate each client to their own private, secure LAN.

    This means that you need redundant public routers (for internet access) and redundant private routers (for the private network).

    Besides QuadraNet, I don't know any other larger volume host here on WHT that actually has a legitimate private network and supplies clients with both a public vlan and a private vlan as a default offering across their server range.

    Feel free to pitch in names if you are 100% positive they operate private network with private vlan per customer.

  25. #25
    Join Date
    Feb 2014
    Posts
    103
    Not everybody is rich and tech savvy clients to get into complicated ipmi setup, with recent events i guess one has to secure ipmi network and probably rather for those who live in countries where vpn itself is banned or the customer who is has hassle of extra step of getting around complex vpn setup or when he is outside or when he is on guest machine with no admin privilege, and need emergency access, all would filter down to support ticket system, where customer would seek help, all the reason for IPMI was to get down or save money effort on support system so that client is doing self service. and ask support for only complex matters,

    there is no need to say what the hell is wrong with you keeping ipmi on public lan, because only few people have static ip even if we need to use access list, and vpn being a luxury/complex for say 50% of customers, and providers themself do not have luxury to loose that percentage of customers these days like someone said they are better off not having such customers. Provider perspective it is extra effort but only a one time setup and bit of extra management which is better than dealing with emergency situations.

Page 1 of 3 123 LastLast

Similar Threads

  1. IPMI safe (secure) to use over public / WAN IP?
    By john2k in forum Colocation, Data Centers, IP Space and Networks
    Replies: 12
    Last Post: 08-16-2013, 09:56 PM
  2. Number of public IPs
    By preciouspigsystems in forum Hosting Software and Control Panels
    Replies: 5
    Last Post: 01-13-2013, 02:41 AM
  3. VM template without public IPs
    By WEBCRAFT-TUCKY in forum Hosting Security and Technology
    Replies: 4
    Last Post: 05-08-2011, 04:11 PM
  4. Juniper SSG5 and Public IPs
    By zynfella in forum Colocation, Data Centers, IP Space and Networks
    Replies: 0
    Last Post: 12-19-2007, 08:59 AM
  5. Godaddy's thinking about going public....
    By Glanhosts in forum Web Hosting Lounge
    Replies: 8
    Last Post: 07-17-2004, 10:46 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •