Results 1 to 24 of 24
  1. #1

    IPMI with KVM Over LAN

    What are the benefits of IPMI with KVM Over LAN? Is it more of a benefit for the datacenter? Or does it have benefits for someone that has collocated there server a far distance away?

  2. #2
    Join Date
    May 2006
    Location
    NJ, USA
    Posts
    6,456
    You can reboot / restart / reformat a server from remote. It makes management a lot easier from a DC level - eliminates need for onsite techs for everything but racking/unracking.
    simplywww: directadmin and cpanel hosting that will rock your socks
    Need some work done in a datacenter in the NYC area? NYC Remote Hands can do it.

    Follow my "deals" Twitter for hardware specials.. @dougysdeals

  3. #3
    Definitely saves one's ass when a machine in Frankfurt dies, and you are in California. Especially the virtual media to do a reinstall right away. I've only used the Supermicro IP/KVM, but it works great either through HTTPS or their Java client (which just uses HTTPS anyways).

  4. #4
    The new supermicro Core i7 boards (F designation i.e. X8STi-F) have them built in now as a third network port on the back of the server. This means you no longer have to burn a chassis slot, or piggy back on the second nic (for the integrated cards). The web interface is nice, if sparse and works very well and is remotely upgradeable and works "lights out" on the box.

    We have started deploying these instead of using Raritan KVM over IP units. The price differential works out to about the same if you are purchasing Raritan used (new the integrated card is a no brainer). Plus you are saying U space by not installing a KVM unit. Downside is you are burning another network port.

    Spend the extra money to get one of these boards, it is well worth it.
    ExpressHosting.net - Fast. Reliable. Affordable.
    Shared Hosting | Dedicated Servers | Colocation | Managed Cloud | AS53255

  5. #5

    main difference in IPMI

    IPMI is great. The web interface with java screen cast and keyboard works well in windows. The only draw back is that the java remote control does not seem to work on OSX. (I have a vmware xp image on my laptop, so not too big of a drawback. Power cycling works nice.

    In my experience the biggest difference is the virtual media support in ipmi. You can virtually mount any ISO or image as a CD or floppy that the bios recognizes, and can boot from.
    This allows the following two advantages over kvm.
    - upgrade you bios remotely
    - just mount the bios upgrade floppy virtually and reboot to reflash. Can't do that with kvm.

    - mount CD install floppy and reinstall OS
    - just mount a virtual CD with netinstall. I had an image mounted over a 3g at+t modem and managed to reinstall the system.
    - you could do this with kvm if you had a pxe/bootp configuration on the network.

    When i install new servers, i just make sure the IPMI is working, and then just go home and configure everything i need remotely.

    (i have noticed a few times that the IPMI ip configuration was reset from the static IP address that I configured, back to DHCP mode. Can't pin point the reason, but it has happened. If you do have the OS running already, you can reset and configure the IPMI addresses from the OS so it's fixable. Of course if you didn't get that far you have to do it locally. SO it's not 100%, but i have never been bitten by a full disconnect. For best results leave it in dhcp mode, and map the MAC to an ip)

  6. #6
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    Quote Originally Posted by expresscolo View Post
    The new supermicro Core i7 boards (F designation i.e. X8STi-F) have them built in now as a third network port on the back of the server. This means you no longer have to burn a chassis slot, or piggy back on the second nic (for the integrated cards). The web interface is nice, if sparse and works very well and is remotely upgradeable and works "lights out" on the box.

    We have started deploying these instead of using Raritan KVM over IP units. The price differential works out to about the same if you are purchasing Raritan used (new the integrated card is a no brainer). Plus you are saying U space by not installing a KVM unit. Downside is you are burning another network port.

    Spend the extra money to get one of these boards, it is well worth it.
    For me, 1U of space and 1 network port is less expensive that 16 network ports (which may even require an additional 1U switch).
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  7. #7
    Quote Originally Posted by expresscolo View Post
    The new supermicro Core i7 boards (F designation i.e. X8STi-F) have them built in now as a third network port on the back of the server. This means you no longer have to burn a chassis slot, or piggy back on the second nic (for the integrated cards). The web interface is nice, if sparse and works very well and is remotely upgradeable and works "lights out" on the box.
    I have one of these boards. They're very nice.

    Getting the IPMI online can be a huge hassle if you don't know a few things.

    The IPMI ip and server's IP have to be on the same subnet. Otherwise, the IPMI just won't work. (It'll accept your configuration, but won't be accessible on the lan)

    After setting the IPMI lan configuration or changing them, I had to shutdown, remove the power cord, let set for 15 seconds, then plug it back in. IPMI runs whenever the power is plugged in, and when the power is plugged in, it "Initializes" itself. Keep that in mind if you ever have issues.

  8. #8
    Quote Originally Posted by IceDog View Post
    Getting the IPMI online can be a huge hassle if you don't know a few things.
    The IPMI ip and server's IP have to be on the same subnet. Otherwise, the IPMI just won't work. (It'll accept your configuration, but won't be accessible on the lan)
    That is not a limitation of the IPMI at all. That's just you not configuring your lan correctly.
    We deploy all our IPMI on a completely separate switch and subnet.

  9. #9
    Quote Originally Posted by rburkat View Post
    That is not a limitation of the IPMI at all. That's just you not configuring your lan correctly.
    We deploy all our IPMI on a completely separate switch and subnet.
    Is your IPMI built directly on the motherboard or on a seperate card?

  10. #10
    Quote Originally Posted by IceDog View Post
    Is your IPMI built directly on the motherboard or on a seperate card?
    The one I'm talking about are part of the mobo. Several different supermicro x8 boards. I believe the IPMI is a separate microcontroller that interfaces with the mobo video, keyboard. The IPMI module has a separate lan controller on it, so it has nothing to do with the OS that is running and the IP that you assign to the other ether ports that are under control of that OS.
    This would be true on the modular AIO ones too.

  11. #11
    You don't *have* to use the dedicated nic on the -f motherboards, you can still share with LAN1 if you want.

    Likewise, you don't have to have them set to use an IP on the same subnet.

    We use them all the time - prefer the interface on the older ones (SIMLP/SIMSO), but the new ones are still good.

  12. #12
    Hello,

    Well, I unearthed this post because I wanted to re-use the info already gathered on here for future reference.
    anyway, I have a server at a colo in FL and Im in PA.

    Let me say that the colo charges for anything and everything I will need to ask them to. If I need a temp or perm IP KVM they will charge me for.
    My server board has Integrated IPMI 2.0 with KVM and
    Dedicated LAN. (Supermicro X9SCL-F)
    I would like some input on how to set this up remotely if i can, maybe via ssh.
    Do the DC need to run another eth cable to the IPMI port from their switch, or does it loop from the second eth port on my server.
    (will they charge me for the use of my built in IP KVM ?)
    I have 5 IP's which 3 usable.

    Andy

  13. #13
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,656
    Quote Originally Posted by amlavor View Post
    Hello,

    Well, I unearthed this post because I wanted to re-use the info already gathered on here for future reference.
    anyway, I have a server at a colo in FL and Im in PA.

    Let me say that the colo charges for anything and everything I will need to ask them to. If I need a temp or perm IP KVM they will charge me for.
    My server board has Integrated IPMI 2.0 with KVM and
    Dedicated LAN. (Supermicro X9SCL-F)
    I would like some input on how to set this up remotely if i can, maybe via ssh.
    Do the DC need to run another eth cable to the IPMI port from their switch, or does it loop from the second eth port on my server.
    (will they charge me for the use of my built in IP KVM ?)
    I have 5 IP's which 3 usable.

    Andy
    Using the dedicated port is recommended, but by default, the IPMI will fail over to whichever port is plugged in. Not sure whether it will prefer lan1 or lan2. Just install ipmitool, and then run these commands:
    modprobe ipmi_si
    modprobe ipmi_devintf
    ipmitool lan set 1 ipaddr <IP address to use>
    ipmitool lan set 1 netmask <netmask to use>
    ipmitool lan set 1 defgw ipaddr <gateway to use>
    ASTUTE HOSTING: Advanced, customized, and scalable solutions with AS54527 Premium Canadian Optimized Network (Level3, PEER1, Shaw, Tinet)
    MicroServers.io: Enterprise Dedicated Hardware with IPMI at VPS-like Prices using AS63213 Affordable Bandwidth (Cogent, HE, Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  14. #14
    Quote Originally Posted by hhw View Post
    Using the dedicated port is recommended, but by default, the IPMI will fail over to whichever port is plugged in. Not sure whether it will prefer lan1 or lan2. Just install ipmitool, and then run these commands:
    modprobe ipmi_si
    modprobe ipmi_devintf
    ipmitool lan set 1 ipaddr <IP address to use>
    ipmitool lan set 1 netmask <netmask to use>
    ipmitool lan set 1 defgw ipaddr <gateway to use>
    Hey.
    what do you mean it'd fail? that means if my server bugs out, and it is looped onto the lan2(vacant) there will be no access?
    Or that would be better if it is connected direct to the DC switch. (im a little confused on where to plug it to)

    I saw a pdf where it shows the use of the supermicro IPMIVIEW and how to find the device. I think that is not bad, but the wiring and setting up remotely had me a bit confused.

    thanks for your input
    Andy

  15. #15
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,656
    Quote Originally Posted by amlavor View Post
    Hey.
    what do you mean it'd fail? that means if my server bugs out, and it is looped onto the lan2(vacant) there will be no access?
    Or that would be better if it is connected direct to the DC switch. (im a little confused on where to plug it to)
    The ipmi will run on whichever port is connected. If the dedicated port isn't plugged in, it will automatically run on one of the regular LAN ports. In other words, if you were to not plug anything in, it would still work by sharing the LAN port that your server uses.
    ASTUTE HOSTING: Advanced, customized, and scalable solutions with AS54527 Premium Canadian Optimized Network (Level3, PEER1, Shaw, Tinet)
    MicroServers.io: Enterprise Dedicated Hardware with IPMI at VPS-like Prices using AS63213 Affordable Bandwidth (Cogent, HE, Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  16. #16
    Quote Originally Posted by hhw View Post
    The ipmi will run on whichever port is connected. If the dedicated port isn't plugged in, it will automatically run on one of the regular LAN ports. In other words, if you were to not plug anything in, it would still work by sharing the LAN port that your server uses.
    So you saying just by now, having the lan1 only connected to my server, I can run the IPMVIEW and access it? if so, sure is a cool thing.

    But i'm planning to connect direct to the IPMI por and to the switch and having one of my free IP's set to it.

    In anyhow then I'm good to configure the IPMI over ssh with the instruction you gave then.


    Really appreciate your inputs

    Andy

  17. #17
    Join Date
    Aug 2002
    Location
    Seattle
    Posts
    5,512
    We use IPMI primarily as a DC ops tool, with the IPMI access restricted to the secure LAN. It's also great for customers because we can provide them IPMI proxies to work on their servers via console in an emergency.

  18. #18
    FWIW - Here are a few walkthroughs I've done (these are a bit older):
    Supermicro IPMIview (lets you manage multiple servers from one interface)
    Tyan IPMI WebGUI
    Gigabyte IPMI WebGUI (Avocent based)
    A bit on ASUS iKVM near the bottom of that piece (going to do a full walkthrough as I review some of the newer boards over the next few weeks.)
    I did a Tom's Hardware article on three Xeon E5 systems, and did a little bit on their WebGUI and that link shows Intel's implementation.

    If you look through those pieces you are likely to see a bunch of the features that you get with IPMI, and some of the things you can do with it.

    Normally, I never have to hook up a mouse or keyboards to any of the boards I review.

    One of the MAJOR benefits aside from being able to remotely bounce a server is the ability to work on servers remotely below the OS level. Let's say you want to swap to iSCSI boot because of a failed local drive, or you have a kernel panic so you cannot SSH, you can virtually pull a chair up to the server and work on it like you just plugged in a low end monitor, keyboard/ mouse.
    Last edited by pjkenned; 01-05-2013 at 06:58 PM.
    My site dedicated to server and workstation hardware: http://www.servethehome.com

  19. Quote Originally Posted by amlavor View Post
    Hello,

    Well, I unearthed this post because I wanted to re-use the info already gathered on here for future reference.
    anyway, I have a server at a colo in FL and Im in PA.

    Let me say that the colo charges for anything and everything I will need to ask them to. If I need a temp or perm IP KVM they will charge me for.
    My server board has Integrated IPMI 2.0 with KVM and
    Dedicated LAN. (Supermicro X9SCL-F)
    I would like some input on how to set this up remotely if i can, maybe via ssh.
    Do the DC need to run another eth cable to the IPMI port from their switch, or does it loop from the second eth port on my server.
    (will they charge me for the use of my built in IP KVM ?)
    I have 5 IP's which 3 usable.

    Andy
    the LAN1(82574L GbE)/LAN2(82579LM GbE) on X9SCL-F board are in reversed MAC order under linux so that LAN1 would be "eth1" and LAN2 would be "eth0". this can make any user confused to hell very fast. so, next time, use X9SCL+-F or X9SCM-iiF board so that this confusion could be voided.

    the beauty of IPMI/BMC is that it's designed to be totally OS independent so that you still can remotely manage/control your server in "light-out" conditions such as powered-down, crashed OS, no OS or pre OS...etc. we have many clients who use just single NIC connection to a single switch port to run all hosting traffic, PXE boot, and IPMI function from single GbE NIC port so that a dedicated IPMI connection is not absolutely necessary. though, you do need a dedicated IP address for IPMI traffic even you share everything from one single physical NIC port (LAN1 or "eth1" on X9SCL-F board).

    instead of managing server from server itself, get yourself familiarized with the idea of using Web GUI (java console in "https://" session by any popular web browser) on a remote workstation to run your IPMI thingies. here is the PDF manual for supermicro's IPMI/BMC:
    http://www.supermicro.com/manuals/ot...PMI_Manual.pdf
    it's a must-read for new user.

  20. #20
    Quote Originally Posted by [email protected] View Post
    the LAN1(82574L GbE)/LAN2(82579LM GbE) on X9SCL-F board are in reversed MAC order under linux so that LAN1 would be "eth1" and LAN2 would be "eth0". this can make any user confused to hell very fast. so, next time, use X9SCL+-F or X9SCM-iiF board so that this confusion could be voided.

    the beauty of IPMI/BMC is that it's designed to be totally OS independent so that you still can remotely manage/control your server in "light-out" conditions such as powered-down, crashed OS, no OS or pre OS...etc. we have many clients who use just single NIC connection to a single switch port to run all hosting traffic, PXE boot, and IPMI function from single GbE NIC port so that a dedicated IPMI connection is not absolutely necessary. though, you do need a dedicated IP address for IPMI traffic even you share everything from one single physical NIC port (LAN1 or "eth1" on X9SCL-F board).

    instead of managing server from server itself, get yourself familiarized with the idea of using Web GUI (java console in "https://" session by any popular web browser) on a remote workstation to run your IPMI thingies. here is the PDF manual for supermicro's IPMI/BMC:
    http://www.supermicro.com/manuals/ot...PMI_Manual.pdf
    it's a must-read for new user.
    So far Im able to see the sensor and configs.
    ie

    balbalblabalbalablba......
    IP Address Source : DHCP Address
    IP Address : xx.xx.xx.xx8 (it is not the same as my eth0 IP)
    Subnet Mask : 255.255.255.0
    MAC Address : xx:xx:xx:xx:xx:xx
    SNMP Community String : public
    IP Header : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
    BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
    Default Gateway IP : xx.xx.xx.1
    Default Gateway MAC : 00:00:00:00:00:00
    Backup Gateway IP : 0.0.0.0
    Backup Gateway MAC : 00:00:00:00:00:00
    802.1q VLAN ID : Disabled
    802.1q VLAN Priority : 0
    RMCP+ Cipher Suites : 1,2,3,6,7,8,11,12
    Cipher Suite Priv Max : aaaaXXaaaXXaaXX
    : X=Cipher Suite Unused
    : c=CALLBACK
    : u=USER
    : o=OPERATOR
    : a=ADMIN
    : O=OEM
    blablablblabalbalbla...

    I'm trying to use a single cable to the server(shared), since my DC is charging me some crazy fees for network drop and other things.
    Now, since locally (ssh) seems to be working , I'm trying to get it working remotely. So even though its a shared connection using eth0, IPMI still needs its own IP. but remotely using IPMVIEW it cannot see it, I cant even ping that IP.
    I have used IPKVM, but its the first time setting up the IPMI onboard myself. I must be missing something.

  21. #21
    Join Date
    Apr 2010
    Posts
    491
    Quote Originally Posted by amlavor View Post
    So far Im able to see the sensor and configs.
    ie

    balbalblabalbalablba......
    IP Address Source : DHCP Address
    IP Address : xx.xx.xx.xx8 (it is not the same as my eth0 IP)
    Subnet Mask : 255.255.255.0
    MAC Address : xx:xx:xx:xx:xx:xx
    SNMP Community String : public
    IP Header : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
    BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
    Default Gateway IP : xx.xx.xx.1
    Default Gateway MAC : 00:00:00:00:00:00
    Backup Gateway IP : 0.0.0.0
    Backup Gateway MAC : 00:00:00:00:00:00
    802.1q VLAN ID : Disabled
    802.1q VLAN Priority : 0
    RMCP+ Cipher Suites : 1,2,3,6,7,8,11,12
    Cipher Suite Priv Max : aaaaXXaaaXXaaXX
    : X=Cipher Suite Unused
    : c=CALLBACK
    : u=USER
    : o=OPERATOR
    : a=ADMIN
    : O=OEM
    blablablblabalbalbla...

    I'm trying to use a single cable to the server(shared), since my DC is charging me some crazy fees for network drop and other things.
    Now, since locally (ssh) seems to be working , I'm trying to get it working remotely. So even though its a shared connection using eth0, IPMI still needs its own IP. but remotely using IPMVIEW it cannot see it, I cant even ping that IP.
    I have used IPKVM, but its the first time setting up the IPMI onboard myself. I must be missing something.
    Seems rather odd that your hosting provider is providing public IP's via DHCP. Try configuring a static IP on it out of one of your assigned public IP's.

    PS public IP's on IPMI are really a horrid idea security wise. The current gen seems to be ok but the earlier ones had a lot of issues locking up when exposed to general public internet levels of noise.

  22. #22
    Quote Originally Posted by silasmoeckel View Post
    Seems rather odd that your hosting provider is providing public IP's via DHCP. Try configuring a static IP on it out of one of your assigned public IP's.

    PS public IP's on IPMI are really a horrid idea security wise. The current gen seems to be ok but the earlier ones had a lot of issues locking up when exposed to general public internet levels of noise.
    I have changed the DHCP to STATIC short after posting. I'm also sharing the eth0 connection.
    But I cant even ping that ip locally(on server ssh nor remotely).

    The IP, Netmask, Gateway are all set.
    Different IP's and MAC's

    But I read somewhere that it needs to have the same MAC for either NIC and IPMI.

  23. Quote Originally Posted by amlavor View Post
    I have changed the DHCP to STATIC short after posting. I'm also sharing the eth0 connection.
    But I cant even ping that ip locally(on server ssh nor remotely).

    The IP, Netmask, Gateway are all set.
    Different IP's and MAC's

    But I read somewhere that it needs to have the same MAC for either NIC and IPMI.
    again, because you are using X9SCL-F board, the "eth0" is actually LAN2 (Intel 82579LM; higher MAC address) which can't be routed to IPMI traffic. IPMI can only be accessed via either LAN1 or dedicated IPMI port, but not LAN2.

    if you were to share IPMI traffic with on-board LAN port, then you must choose LAN1 (Intel 82574L lower MAC) which is seen by CentOS/RHEL as "eth1". by factory default, IPMI (regardless LAN1 or dedicated port) does have an unique MAC address which can be found on a label on motherboard, and it should be displayed in BIOS unless you had changed it, and you should not do so.

  24. #24
    Quote Originally Posted by [email protected] View Post
    again, because you are using X9SCL-F board, the "eth0" is actually LAN2 (Intel 82579LM; higher MAC address) which can't be routed to IPMI traffic. IPMI can only be accessed via either LAN1 or dedicated IPMI port, but not LAN2.

    if you were to share IPMI traffic with on-board LAN port, then you must choose LAN1 (Intel 82574L lower MAC) which is seen by CentOS/RHEL as "eth1". by factory default, IPMI (regardless LAN1 or dedicated port) does have an unique MAC address which can be found on a label on motherboard, and it should be displayed in BIOS unless you had changed it, and you should not do so.
    Alright, so let see if I got this straight.

    Instead using lan2(eth0)which is now connected, I need to have it moved over to lan1(eth1) .(I use centos btw)
    Then it indeed would see the ip 9(IPMI) Nic(8)
    Im going to do this change now.
    thanks

Similar Threads

  1. IPMI Simso+
    By kevhosting in forum Colocation and Data Centers
    Replies: 2
    Last Post: 11-05-2008, 10:21 PM
  2. Ipmi
    By Dualism in forum Colocation and Data Centers
    Replies: 2
    Last Post: 04-16-2007, 10:46 AM
  3. What is IPMI, how does it help?
    By voipfc in forum Dedicated Server
    Replies: 1
    Last Post: 03-04-2007, 11:39 AM
  4. IPMI or not?
    By gate2vn in forum Colocation and Data Centers
    Replies: 52
    Last Post: 10-04-2006, 02:11 AM
  5. IPMI Question
    By freebase in forum Colocation and Data Centers
    Replies: 19
    Last Post: 09-04-2004, 11:59 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •