Results 1 to 22 of 22
  1. #1
    Join Date
    Nov 2002
    Posts
    536

    10GbE terrible performance.

    hello,

    I have two servers with dual Intel X520 10GbE network cards. Servers are connected directly to each-other with fiber. I'm making some tests with iperf / ntttcp and i have about 70MB/s speed:

    [172] 0.0- 1.0 sec 73.4 MBytes 73.4 MBytes/sec
    [172] 1.0- 2.0 sec 70.8 MBytes 70.8 MBytes/sec
    [172] 2.0- 3.0 sec 70.2 MBytes 70.2 MBytes/sec
    [172] 3.0- 4.0 sec 66.9 MBytes 66.9 MBytes/sec
    [172] 4.0- 5.0 sec 67.0 MBytes 67.0 MBytes/sec

    I should get about 700 / 800, no? I tried to enable/disable jumbo frames but didnt help. I'm using intel latest firmware, and atm dunno what is cause this.

    What's your advice?

    thanks
    yeah.. i'm useless!

  2. #2
    Join Date
    Oct 2005
    Location
    Tucson AZ
    Posts
    367

  3. #3
    Join Date
    Nov 2010
    Location
    Tampa
    Posts
    41
    Unless I am just short on sleep today, 10GbE is in bits, not bytes. Your output is measuring in bytes, which is a decrease by 8x. Would still seem low, however.

  4. #4
    Join Date
    Oct 2005
    Location
    Tucson AZ
    Posts
    367

  5. #5
    I'm ot overly familiar with iperf/ntttcp but if they use any type of data stored on a disk drive, the drive subsystem would be a limiting factor on server to server connectivity. 70 MB/s seems to just about the speed your standard 10k SAS drive and/or RAID1 SATA drives drive system would perform at.

  6. #6
    Join Date
    Oct 2005
    Location
    Tucson AZ
    Posts
    367
    Quote Originally Posted by colomondo View Post
    I'm ot overly familiar with iperf/ntttcp but if they use any type of data stored on a disk drive, the drive subsystem would be a limiting factor on server to server connectivity. 70 MB/s seems to just about the speed your standard 10k SAS drive and/or RAID1 SATA drives drive system would perform at.
    (iperf/ntttcp is not disk dependent)
    But, if your 10k SAS drive is only performing at 70MB/s you should get some new drives or controllers or something...

  7. #7
    Join Date
    Jul 2009
    Location
    UK
    Posts
    1,308
    Try multithreading iperf.
    Normally you can tell it how many threads to run.

    Sometimes only running on one thread will limit the throughput.
    Live Chat Support Software for your Business website - IMsupporting.com

  8. #8
    Join Date
    Nov 2002
    Posts
    536
    Hello,

    Thanks for your replies.

    I dont see any errors in interfaces. Also, iperf/ntttcp does not use disks. Any other suggestions?

    On Linux servers i have:

    [ 3] 9.0-10.0 sec 534 MBytes 534 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 10.0-11.0 sec 529 MBytes 529 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 11.0-12.0 sec 652 MBytes 652 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 12.0-13.0 sec 659 MBytes 659 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 13.0-14.0 sec 647 MBytes 647 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 14.0-15.0 sec 660 MBytes 660 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 15.0-16.0 sec 662 MBytes 662 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 16.0-17.0 sec 661 MBytes 661 MBytes/sec

    Thanks
    yeah.. i'm useless!

  9. #9
    Join Date
    Oct 2005
    Location
    Tucson AZ
    Posts
    367
    Quote Originally Posted by phactor View Post
    Hello,

    Thanks for your replies.

    I dont see any errors in interfaces. Also, iperf/ntttcp does not use disks. Any other suggestions?

    On Linux servers i have:

    [ 3] 9.0-10.0 sec 534 MBytes 534 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 10.0-11.0 sec 529 MBytes 529 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 11.0-12.0 sec 652 MBytes 652 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 12.0-13.0 sec 659 MBytes 659 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 13.0-14.0 sec 647 MBytes 647 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 14.0-15.0 sec 660 MBytes 660 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 15.0-16.0 sec 662 MBytes 662 MBytes/sec
    [ ID] Interval Transfer Bandwidth
    [ 3] 16.0-17.0 sec 661 MBytes 661 MBytes/sec

    Thanks
    So these results are not of the same servers but rather other servers where this is working properly?

    Are the new servers winderz?

  10. #10
    Join Date
    Nov 2002
    Posts
    536
    This last results are in centos 5.5 boxes. These are normal, no?

    The first, and where the slowness exist, are windows 2k8 r2.
    yeah.. i'm useless!

  11. #11
    Join Date
    Oct 2005
    Location
    Tucson AZ
    Posts
    367

  12. #12
    Join Date
    Nov 2002
    Posts
    536
    I must check ntttcp. With iperf i use:

    IN server:

    iperf -s -P 0 -i 1 -p 5001 -f M

    iperf -c 10.3.0.3 -P 1 -i 1 -p 5001 -f M -t 50

    10.3.0.3 is the server ip address.
    yeah.. i'm useless!

  13. #13
    Join Date
    May 2006
    Location
    New Zealand
    Posts
    6,482
    CPU isn't maxing out on either client or server is it?

    You need to increase from the default window size and or increase the number of threads running.

    Set "-w 2M -p 10" and increase as necessary.

    Also remove "-f M" it will only confuse things
    █ | Brad - Hoopla Hosting - Email or add me to Gtalk! brad [at] hooplahosting.com
    █ | Web Hosting New Zealand - Reseller Hosting - cPanel - Zurmo Hosting - Softaculous - CloudFlare
    █ | VPS - SolusVM - E3-1230 - RAID10 - 1Gbit

  14. #14
    Iperf is probably maxing out your cpus.
    Phoenix Dedicated Servers -- IOFLOOD.com
    Email: sales [at] ioflood.com
    Skype: iofloodsales
    Backup Storage VPS -- 1TBVPS.com

  15. #15
    Join Date
    Nov 2002
    Posts
    536
    Quote Originally Posted by funkywizard View Post
    Iperf is probably maxing out your cpus.
    Hello,

    No, cpu usage is near none. This is a dual E5620 box, that's not the problem.
    yeah.. i'm useless!

  16. #16
    Join Date
    Nov 2002
    Posts
    536
    Quote Originally Posted by Hoopla-Brad View Post
    CPU isn't maxing out on either client or server is it?

    You need to increase from the default window size and or increase the number of threads running.

    Set "-w 2M -p 10" and increase as necessary.

    Also remove "-f M" it will only confuse things

    Hello,

    No, CPU isnt maxing out. Results with 100M

    [172] 5.0- 6.0 sec 358 MBytes 358 MBytes/sec
    [172] 6.0- 7.0 sec 357 MBytes 357 MBytes/sec
    [172] 7.0- 8.0 sec 358 MBytes 358 MBytes/sec
    [172] 8.0- 9.0 sec 357 MBytes 357 MBytes/sec
    [172] 9.0-10.0 sec 363 MBytes 363 MBytes/sec
    [172] 0.0-10.3 sec 2734 MBytes 266 MBytes/sec
    Done.

    Why the difference?
    yeah.. i'm useless!

  17. #17
    the cpu could be maxing out if iperf is only using one core. try typing "top" and then type "1" and see if the load is balanced between cores, or just maxing out one of them.
    Phoenix Dedicated Servers -- IOFLOOD.com
    Email: sales [at] ioflood.com
    Skype: iofloodsales
    Backup Storage VPS -- 1TBVPS.com

  18. #18
    Join Date
    Nov 2002
    Posts
    536
    Quote Originally Posted by funkywizard View Post
    the cpu could be maxing out if iperf is only using one core. try typing "top" and then type "1" and see if the load is balanced between cores, or just maxing out one of them.
    Hello,

    Problem is in windows. I have task manager active, and CPU is about 10% in server and 1% in client.
    yeah.. i'm useless!

  19. #19
    Join Date
    Apr 2009
    Posts
    1,143
    might be a stupid question.. but are jumboframes enabled on the network? - aswell as on the nic's

    Have you looked at windows tcp tweaking?
    http://www.dslreports.com/tweaks

  20. #20
    Join Date
    Jan 2003
    Location
    Chicago, IL
    Posts
    6,889
    Quote Originally Posted by phactor View Post
    Hello,

    Problem is in windows. I have task manager active, and CPU is about 10% in server and 1% in client.
    Using 1 core at 100% out of 8 would be 12.5%, or if it is hyperthreaded you're looking at 16 virtual cores, where 10% is almost certainly using one physical core. I don't believe it is a multi-threaded application. Not saying that is the problem, but just by saying you're at 10% CPU load also doesn't prove that isn't the problem.
    Karl Zimmerman - Steadfast: Managed Dedicated Servers and Premium Colocation
    karl @ steadfast.net - Sales/Support: 312-602-2689
    Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation
    Now Open in New Jersey! - Contact us for New Jersey colocation or dedicated servers

  21. #21
    Join Date
    Nov 2002
    Posts
    536
    I enabled jumbo frames on both nics. Since its a direct connection, its enough right?
    yeah.. i'm useless!

  22. #22
    Join Date
    May 2006
    Location
    New Zealand
    Posts
    6,482
    I know you want to run TCP but have you tried UDP? It's much easier to troubleshoot where the problem lies.

    You will need to set the speed with UDP.
    █ | Brad - Hoopla Hosting - Email or add me to Gtalk! brad [at] hooplahosting.com
    █ | Web Hosting New Zealand - Reseller Hosting - cPanel - Zurmo Hosting - Softaculous - CloudFlare
    █ | VPS - SolusVM - E3-1230 - RAID10 - 1Gbit

Similar Threads

  1. Replies: 0
    Last Post: 01-20-2011, 10:53 PM
  2. switch that supports wire rate 10gbe sflow / netflow
    By funkywizard in forum Colocation and Data Centers
    Replies: 16
    Last Post: 01-23-2010, 03:42 AM
  3. Replies: 3
    Last Post: 05-26-2008, 10:49 PM
  4. Terrible Situation in a Terrible Partnership
    By Trophimus in forum Web Hosting
    Replies: 140
    Last Post: 01-05-2006, 12:30 AM
  5. Unixhoster.com...terrible service, terrible people
    By hank9481 in forum Web Hosting
    Replies: 0
    Last Post: 01-08-2003, 03:36 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •