Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Effect of the network card

[MaB]

Hi All,

I've been reading more and more posts about how a quality network card is important. We build all of our servers from new, quality parts but honestly we always rely on the built in NIC on the motherboard (which are brand name motherboards, but nonetheless)

How can a lesser quality NIC effect a server other than total failure of network activity? What signs should we look for and what are quality brands? My first guess is dropped packets or low throughput. I've also heard of people saying that load can go up from a bad NIC - is there any truth to that?

Thanks!

[CiscoMike]

Originally Posted by MaB

Hi All,

I've been reading more and more posts about how a quality network card is important. We build all of our servers from new, quality parts but honestly we always rely on the built in NIC on the motherboard (which are brand name motherboards, but nonetheless)

How can a lesser quality NIC effect a server other than total failure of network activity? What signs should we look for and what are quality brands? My first guess is dropped packets or low throughput. I've also heard of people saying that load can go up from a bad NIC - is there any truth to that?

Thanks!

yes there is. A NIC, like most other devices, will want to interrupt the CPU for assistance (thus, the need for an IRQ). The difference is some NICs don't need to send an attention signal too often, only when they need to buffer (like for congestion or a massive reorder operation) and rediscover portions of the network like a PMUTD request. The NIC might also need to signal the CPU during a "back-off" operation when the TCP window size has decreased and it needs the CPU to buffer additional frames in RAM.

Some cheap NICs have zero buffer size and require the CPU to not only process all packets but also buffer everything before it's sent out. While this is only a 5%-10% hit on the CPU, that could be significant. Some NICs have no cache to store ARP or IP entries and so for every lookup they need to request for assistance from the OS which again is a CPU interrupt.

The good news is the days of NICs like that are pretty much long gone. Anything by Broadcom has some buffering capabilities and a small amount of lookup cache for route/forwarding entries. Most Intel based products also have those same features as do AMD branded network-quality chips (not the home desktop chips, those are still crap).

[david510]

Mike,

Good one. Can you please explain a bit more on PMUTD request.

[CiscoMike]

Originally Posted by david510

Mike,

Good one. Can you please explain a bit more on PMUTD request.

Path MTU discovery - it's a method used to determine the smallest MTU size between you and your host. It uses ICMP at each hop (it's kinda like a traceroute in nature but it's not a time-exceed message). The OS, in order to properly size any and all frames, will use a PMTUD sequence which generally requires collaboration between the OS (initiate), the CPU (format) and the NIC (send/receive). It's not too CPU intensive unless you have multiple paths to inspect. Generally you don't need PMTUD unless you get orders to fragment from an upstream device that cannot fragment on it's own.

The primary place you'll see PMTUD is on VPN but also anywhere where you suffer serilization (spelled wrong, too lazy to look it up) delay when going between media types (like ethernet to ATM aka DSL) or when trying to pass jumbo/giant frames OR you have the DF bit set in the IP header.

[Steven]

Originally Posted by MaB

Hi All,

I've been reading more and more posts about how a quality network card is important. We build all of our servers from new, quality parts but honestly we always rely on the built in NIC on the motherboard (which are brand name motherboards, but nonetheless)

How can a lesser quality NIC effect a server other than total failure of network activity? What signs should we look for and what are quality brands? My first guess is dropped packets or low throughput. I've also heard of people saying that load can go up from a bad NIC - is there any truth to that?

Thanks!

A good example is servers we had at softlayer, 9mbit outgoing it would kernel panic and die due to the poor onboard nic that was on the board. there was also various messages on message boards with the same problem with the nic in question..

Personally I prefer a intel nic even if I have to add a addon card.

[wKkaY]

What board/NIC/OS combos are those panic'ed servers, Steven?

[CiscoMike]

SuperMicro boxes as that's what SL deploys.

[Steven]

Originally Posted by wKkaY

What board/NIC/OS combos are those panic'ed servers, Steven?

It only happened on their AMD machines which used nforce forcedeth nics. This was on centos 5. We ordered 5 machines at once, and every box exhibited the same behavior. We move to Intel based machines which have e1000 nics and we have been pushing 300-500mbit on each on ever since.


At first we thought it was bad hardware when it happened to the first one (we have encountered bad hardware on over 30 boxes in the past few months, most of the time when its first deployed to us, for this reason I never recommend softlayer to clients and I often get them to move elsewhere). But after all 5 did the same thing and some research on the net it became obvious it was an issue with the particular network adaptor.

[CiscoMike]

nforce and AMD/ATI on-board controllers are good examples of NICs that require excessive CPU interrupts. I still swear by the 3Com 905 adapters even though they haven't been sold for years. Some of the modern Intel NICs are pretty good but again, Broadcom has pretty much commoditized the market and shy of driver support, the Broadcom based NICs are fairly stable, standard and reliable.

[Steve_Arm]

However the interrupt frequency isn't something that can be set from the system?

[MaB]

We have a server that experienced 2 random outages before we moved the raid card and harddrives to a new chasis. The first time the machine stopped responding to IP traffic including pings - the second time it responded to pings but nothing worked (including ssh) - unfortunately we didn't have access to the console to see what was going on and we simply APC'd it. We tested the RAM for 48 hours, we tested the CPU for a few hours and we now have 2 high CPU usage VPSs on the node and I couldn't figure out why we couldn't make it crash. My thought is now that the NIC may have been causing the troubles, but according to MRTG data there wasn't a significant traffic burst around the times of either crash.

The adapter is:
01:08.0 Ethernet controller: Intel Corporation 82562EZ 10/100 Ethernet Controller (rev 02)

It's an ASUS motherboard with an Intel chipset. I just used netperf to push and pull nearly 100mbps from the network card - but it didn't crash

[CiscoMike]

Originally Posted by Steve_Arm

However the interrupt frequency isn't something that can be set from the system?

Some drivers allow that but why not use a purpose built server NIC that is optimized for I/O, doesn't ask the CPU for help every other second and can store basic network data without having to provision RAM and risk a flush erasing all ARP and IP info?

Moral of the story - nForce and RV based NICs are not server solutions, not optimized for I/O and can be detrimental to server based network usage. On-board server NICs using Broadcom, AMD or Intel silicon is ok but it's still better to get a dedicated NIC manufactured for server environments for the ultimate solution.