Page 3 of 4 FirstFirst 1234 LastLast
Results 51 to 75 of 86
  1. #51
    Join Date
    Nov 2009
    Location
    Reading, UK
    Posts
    17
    @ditlev

    We're running as many tests as we can at the moment, but one thing that stands out straight away is the testing of the raw drives vs the testing of them as a single drive in a Vdisk.

    Firstly ran the tests on the raw drives :

    Code:
    dd if=/dev/zero of=/dev/sd* bs=10M count=100 oflag=direct
    dd if=/dev/sd*  of=/dev/null bs=10M count=100 iflag=direct
    Then made a new Datastore based on 1 drive, 0 stripes, 0 copies, and then ran these tests on that drive ( The Drive is local to HV1 )

    Code:
    dd if=/dev/zero of=/dev/nbd* bs=10M count=100 oflag=direct
    dd if=/dev/nbd* of=/dev/null bs=10M count=100 iflag=direct
    There seems to be a pretty large discrepancy:

    Code:
                                  Vdisk             Raw                    Differrence
    Hypervisor 1    W: 285 MB/s       W: 320 MB/s       W: -35 MB/sec
                            R: 335 MB/s       R: 457 MB/s       R: -127 MB/sec
                    
    Hypervisor 2    W: 268 MB/s       W: 320 MB/s       W: -52 MB/sec
                            R: 349 MB/s       R: 457 MB/s       R: -108 MB/sec
    
    Hypervisor 3    W: 274 MB/s       W: 320 MB/s       W: -46 MB/sec
                            R: 320 MB/s       R: 457 MB/s       R: -101 MB/sec

  2. #52

    *

    Quote Originally Posted by Krystal-Tom View Post
    @ditlev

    We're running as many tests as we can at the moment, but one thing that stands out straight away is the testing of the raw drives vs the testing of them as a single drive in a Vdisk.

    Firstly ran the tests on the raw drives :

    Code:
    dd if=/dev/zero of=/dev/sd* bs=10M count=100 oflag=direct
    dd if=/dev/sd*  of=/dev/null bs=10M count=100 iflag=direct
    Then made a new Datastore based on 1 drive, 0 stripes, 0 copies, and then ran these tests on that drive ( The Drive is local to HV1 )

    Code:
    dd if=/dev/zero of=/dev/nbd* bs=10M count=100 oflag=direct
    dd if=/dev/nbd* of=/dev/null bs=10M count=100 iflag=direct
    There seems to be a pretty large discrepancy:

    Code:
                                  Vdisk             Raw                    Differrence
    Hypervisor 1    W: 285 MB/s       W: 320 MB/s       W: -35 MB/sec
                            R: 335 MB/s       R: 457 MB/s       R: -127 MB/sec
                    
    Hypervisor 2    W: 268 MB/s       W: 320 MB/s       W: -52 MB/sec
                            R: 349 MB/s       R: 457 MB/s       R: -108 MB/sec
    
    Hypervisor 3    W: 274 MB/s       W: 320 MB/s       W: -46 MB/sec
                            R: 320 MB/s       R: 457 MB/s       R: -101 MB/sec
    When you get a chance, run those tests on a stripe datastore. Try 2 stripes and then 4 stripes. The numbers drop even more.

  3. #53
    Join Date
    Nov 2009
    Location
    Reading, UK
    Posts
    17
    Same Hypervisor, all drives are local, running the test from onapp web interface.

    ddwrapper BS=4096 Disk Size=10GB
    1 Stripe : W: 246 MB/s R: 283 MB/s
    2 Stripes: W: 379 MB/s R: 434 MB/s
    4 Stripes: W: 426 MB/s R: 528 MB/s

  4. #54
    Join Date
    Mar 2009
    Location
    NL
    Posts
    594
    Quote Originally Posted by nwilkens View Post
    I just picked up a pair of ex4550's, and will be using Intel X540-T2 adapters. I will be testing OnApp in a very similar setup in the next week. I'd be interested in testing the same scenario, and will let you know what happens.
    We run on ex4550's with intel nics with RR. Works just fine, see our results earlier.
    YISP - High Bandwidth dedicated servers and colocation in YISP-AS(Amsterdam)!

    Website: http://www.yisp.nl
    Contact: info "(AT)" yisp.nl

  5. #55
    Quote Originally Posted by Krystal-Tom View Post
    Same Hypervisor, all drives are local, running the test from onapp web interface.

    ddwrapper BS=4096 Disk Size=10GB
    1 Stripe : W: 246 MB/s R: 283 MB/s
    2 Stripes: W: 379 MB/s R: 434 MB/s
    4 Stripes: W: 426 MB/s R: 528 MB/s
    Can you try doing a remote path? If those disks are on (EX) HV1, try testing it from HV2. That way you know what the actual performance will be.

    When doing at least 1 replica, the data will be copied to a different server. So if the disks are local, when you do a Write test it has to write to the disks locally, and to the remote copy at the same time. The remote path is what is killing performance. Once you have 8 disks involved (4 local, 4 remote - 1 replica + 4 stripes) the write/read is very low.

  6. #56
    Join Date
    Nov 2009
    Location
    Reading, UK
    Posts
    17
    ddwrapper BS=4096 Disk Size=10GB ( All Tested from HV 1 )
    1 Stripe &1 Copy: W: 228 MB/s R: 336 MB/s ( 1x HV1 - 1x HV2 )
    2 Stripes &1 Copy: W: 336 MB/s R: 484 MB/s ( 2x HV1 - 2x HV2 )
    2 Stripes &1 Copy: W: 342 MB/s R: 119 MB/s ( 1x HV1 - 1x HV2 - 2x HV3 )
    4 Stripes &1 Copy: W: 371 MB/s R: 564 MB/s ( 4x HV1 - 4x HV2 )
    4 Stripes &1 Copy: W: 338 MB/s R: 180 MB/s ( 3x HV1 - 2x HV2 - 3x HV3)

    118MB/sec or 180MB/sec obviously isn't anywhere near good enough for production

    Also, why are we losing 100MB/sec of read performance on local drives even when they are just on their own ?

  7. #57
    Quote Originally Posted by Krystal-Tom View Post
    ddwrapper BS=4096 Disk Size=10GB ( All Tested from HV 1 )
    1 Stripe &1 Copy: W: 228 MB/s R: 336 MB/s ( 1x HV1 - 1x HV2 )
    2 Stripes &1 Copy: W: 336 MB/s R: 484 MB/s ( 2x HV1 - 2x HV2 )
    2 Stripes &1 Copy: W: 342 MB/s R: 119 MB/s ( 1x HV1 - 1x HV2 - 2x HV3 )
    4 Stripes &1 Copy: W: 371 MB/s R: 564 MB/s ( 4x HV1 - 4x HV2 )
    4 Stripes &1 Copy: W: 338 MB/s R: 180 MB/s ( 3x HV1 - 2x HV2 - 3x HV3)

    118MB/sec or 180MB/sec obviously isn't anywhere near good enough for production

    Also, why are we losing 100MB/sec of read performance on local drives even when they are just on their own ?
    The last test you did, 4 stripes & 1 Replica with all your HV is a more realistic test since you will have more than 2 HV's eventually in production. I do think there is an issue with the dm-multipath. With 4 stripes, and 1 copy, you should be getting awesome performance. We brought up this performance issue 3 months ago.

  8. #58
    Michail's been testing a duplicate environment I setup to our production configuration and believes the issue is related to TCP Incast:

    h t t p : //bradhedlund.com/2011/05/01/tcp-incast-and-cloud-application-performance/

    Lowering the MTU helped slightly, but we're still seeing terrible remote read performance. With 4 stripes, 2 copies on SAS 15K disks and a MTU of 9000, we were getting 60MB/s write and 15MB/s read remotely. Dropping the MTU to 1500 got us to about 30MB/s read remotely (write stayed the same). Switching that to 2 stripes and 2 copies gets me to 67MB/s read remotely.

    We're using a stack of two 2960X switches which each have a pair of ASIC's with 2MB buffers from what I read. I am seeing output drops which seems to confirm the TCP Incast issue (though I'd expect to see input drops since the nature of the issue is that the receiving system cannot process traffic fast enough).

    Things I've tried:
    - Add additional storage nic to only the remote node. Tried no etherchannel, etherchannel and lacp configurations on the switch. Performance results were ~5% of a single nic.
    - Increase the hold-queues on the IS switchports. However, from what I read this only applies to process switched packets so that didn't help either.
    - QoS is disabled on our switches, so I tried enabling it and tweaking the buffers, but that just caused more drops. Performance was roughly the same.
    - Various MTU settings of 800, 500, and 284. Performance was roughly the same as 1500.
    - Bonding on local and remote nodes. The only configuration that increased performance was a split VLAN (two nics on each system, different VLANs per nic). Split VLAN doubled my numbers, but requires that ALL systems have nics in those VLANs (ie. your backup and provisioning servers need two nics too).

    What are others using for switches? I've heard the 4948's have larger buffers which can help.

  9. #59
    Quote Originally Posted by owjeff View Post
    Michail's been testing a duplicate environment I setup to our production configuration and believes the issue is related to TCP Incast:

    h t t p : //bradhedlund.com/2011/05/01/tcp-incast-and-cloud-application-performance/

    Lowering the MTU helped slightly, but we're still seeing terrible remote read performance. With 4 stripes, 2 copies on SAS 15K disks and a MTU of 9000, we were getting 60MB/s write and 15MB/s read remotely. Dropping the MTU to 1500 got us to about 30MB/s read remotely (write stayed the same). Switching that to 2 stripes and 2 copies gets me to 67MB/s read remotely.

    We're using a stack of two 2960X switches which each have a pair of ASIC's with 2MB buffers from what I read. I am seeing output drops which seems to confirm the TCP Incast issue (though I'd expect to see input drops since the nature of the issue is that the receiving system cannot process traffic fast enough).

    Things I've tried:
    - Add additional storage nic to only the remote node. Tried no etherchannel, etherchannel and lacp configurations on the switch. Performance results were ~5% of a single nic.
    - Increase the hold-queues on the IS switchports. However, from what I read this only applies to process switched packets so that didn't help either.
    - QoS is disabled on our switches, so I tried enabling it and tweaking the buffers, but that just caused more drops. Performance was roughly the same.
    - Various MTU settings of 800, 500, and 284. Performance was roughly the same as 1500.
    - Bonding on local and remote nodes. The only configuration that increased performance was a split VLAN (two nics on each system, different VLANs per nic). Split VLAN doubled my numbers, but requires that ALL systems have nics in those VLANs (ie. your backup and provisioning servers need two nics too).

    What are others using for switches? I've heard the 4948's have larger buffers which can help.
    We were told the same thing today (same link) and have tried many other settings like you. We tried dropping the Round Robin and went the ol-reliable LACP route, and also tried TLB and ALB. We can't get it to improve performance. Host to Host bandwidth throughput is great, a solid 20Gbps (2+ threads). Throughput to the storage controllers are awful, maybe 500Mbps. Only thing we haven't tried is the MTU changes. Just seems like we are running in circles here. Seems a bit odd that we would have line rate capacity issues on a bonded 2x 10Gig port, when only doing 500Mbps. Also tried going a single 10Gig to the SAN network on all servers (no bonding) didn't help much either.

    We are going to spend a bit of time investigating the buffer issue theory to see if we can improve it. Nothing is off the table at this point, and I hope Julian (OnApp Storage Dev) keeps investigating this issue.

  10. #60
    Based on additional testing, it seems that only the number of replicas affects this issue. A datastore with 2 copies and 4 stripes has the same read performance characteristics as a datastore with 2 copies and 2 stripes.

    There is a command that can affinitize traffic to a single node on the remote server: /usr/pythoncontroller/setPrio. When I used that command to direct a vdisk to a single hosted, I was able to get line rate read speed.

    From the remote node:

    Get the remote node's uuid:
    onappstore getid

    Find the vdisk id and members:
    onappstore list

    Find the hostid of a member:
    onappstore nodeinfo uuid=<member id>

    Online the disk on the remote node:
    onappstore online uuid=<vdisk id> frontend_uuid=<node id>

    Affinitize vdisk to specific node:
    setPrio uuid=<vdisk id> hostid=<host id>

    Run the test:
    /.rw/ddwrapper device=/dev/mapper/<vdisk id> bs=4096K

    Hoping OnApp can add a work-around leveraging this affinity to a single node.

  11. #61
    The issue is mainly that TCP Incast issue, we can confirm it being an issue. It explains that when you have more replicas, it kills the performance since the packets can't get transmitted. The issue is the many-to-one (READS) that is killing the performance. So that SetPrio script helps limit the amount of servers responding to the request. We saw a huge performance increase on READS when using that. It doesn't explain why the WRITES are slow (one-to-many). Must be some more TCP Incast issues.

    OnApp storage team will have to get very creative to over come this limitation, as there is no switch that can fix this...not even the expensive EX4550's we are using. The SetPrio helped with the reads, but to get anything faster a regular RAID Controller would be preferred as it could limit the TCP Incast issues (fewer disks responding to a query).
    Cloud IaaS Solutions Provider - www.CloudVZ.com
    SSD SANs | High IOPs | Public & Private Cloud
    Solutions | Content Delivery Network
    Create your virtual datacenter in seconds!

  12. #62
    Good news, Julian has informed us that they are working on a mechanism to distribute the storage bandwidth better so that it helps reduce the TCP Incast issue that people are experiencing.

  13. #63
    Join Date
    May 2003
    Location
    San Francisco, CA
    Posts
    1,506

    OnApp Bonding issue

    Thanks for the update.
    * GeekStorage.com - Offering awesome website hosting for over 13 years!
    * Shared Hosting * Reseller Hosting * Virtual Private Servers * Dedicated Servers
    * Have questions? Send us an e-mail, we'd love to hear from you!

  14. #64
    Join Date
    Oct 2001
    Location
    Ohio
    Posts
    8,535
    Quote Originally Posted by CloudVZ View Post
    Good news, Julian has informed us that they are working on a mechanism to distribute the storage bandwidth better so that it helps reduce the TCP Incast issue that people are experiencing.
    Yep - the storage team asked that I tell everyone, they're working on a queue rate limiting mechanism to try and control the burst size for aggregate flows.

  15. #65
    Join Date
    Jan 2004
    Location
    Pennsylvania
    Posts
    942
    Some quick research shows using the RED queuing algorithm is enough to fix TCP incast problem. Have you tried adding it to your nodes?

    Code:
    tc qdisc add dev eth0 parent 1:1 handle 10: red
            limit 4000000 ecn bandwidth 10000Mbit
    ^ that might work, limit variable likely has to be tuned.
    Matt Ayres - togglebox.com
    Linux and Windows Cloud Virtual Datacenters powered by Onapp / Xen
    Instant Setup, Instant Scalability, Full Lifecycle Hosting Solutions

    www.togglebox.com

  16. #66
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Quote Originally Posted by IGobyTerry View Post
    Yep - the storage team asked that I tell everyone, they're working on a queue rate limiting mechanism to try and control the burst size for aggregate flows.
    Do they have an ETA? We will be deploying a onapp storage deployment soon.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  17. #67
    Join Date
    Oct 2001
    Location
    Ohio
    Posts
    8,535
    Quote Originally Posted by Steven View Post
    Do they have an ETA? We will be deploying a onapp storage deployment soon.
    Sorry, I don't have an ETA. Best way might be to open communication with your TAM and make them aware that you're following the issue and would like to be updated.

  18. #68
    Join Date
    Jun 2013
    Posts
    60
    We're trying to work out the best way of doing our storage network. It seems we're getting a different answer depending on who we talk to on OnApp support.

    We've 2 HV's and 1 Backup server. All of them have 4 Intel Pro NIC's @ 1Gbps on storage network. Storage switch is a HP2920G. Initially we tried them in LACP, but didn't get huge data rates accross HV's. Then moved to RR (on advice of OnApp support) and had to put each nic in separate VLAN and this gave us pretty much 4Gbps between nodes... Once we'd configured our storage with stripes then we managed around 500MB/s transfer rate...

    However, once we started using this in production we had backups failing on a regular basis and OnApp support (different person) have said that having each NIC in a separate VLAN is not a good idea if a NIC goes down then data could be lost between them causing backup to fail.

    If we have all the NIC's in the same VLAN on the HP then we get duplicate packet issues (on network tests) and I can imagine this would cause problems with OnApp backups and provisioning. LACP gives us the resiliency but not the performance (that I'd expect). Am I missing something simple here to get both resiliency and performance?

  19. #69
    Join Date
    Nov 2012
    Posts
    428
    What's the actual cause of the failing backups? You can rule out network issues by doing a ping test to see if you loose any packets over a period of time.

    Yes, there is always the risk of a NIC failing and it breaks the path. However, I think your switch will go long before your NIC craps out on you .

  20. #70
    Join Date
    Jun 2013
    Posts
    60
    Unable to pin that down yet... They were just failing on one HV, which led me to suspect network, but that looks fine as far as I can tell... No signs of NIC's going up and down that I can spot so far... I might try a good old re-seat of the NIC's (they're on PCI not onboard) just to be sure nothing dodgy there.

    I've run a ping test at the maximum MTU of the storage network for a decent amount of time and all looks good so far.

    When the backups fail we end up with orphaned disks in our store that then need to delete before the backup will run again.

  21. #71
    Join Date
    Aug 2011
    Location
    Dub,Lon,Dal,Chi,NY,LA
    Posts
    1,839
    Anything in the log suggesting the reason for failure?

  22. #72
    Join Date
    Jun 2013
    Posts
    60
    Support seems to think it's corrupt windows file systems, but running chkdsk /f on those reports as ok every time so far. The error is on ntfsclone and then the rollback seems to fail.

    I was suspecting the network as the backups seem to work fine on one of the HV's with no issues (so far), but can't see anything obvious there.

    Have any of you guys had problems with balance-RR and separate VLAN's on the switch?

  23. #73
    Join Date
    Aug 2011
    Location
    Dub,Lon,Dal,Chi,NY,LA
    Posts
    1,839
    We had a similar issue with some Windows VMs that several CHKDSK and offline / mounted fscks eventually resolved. The NTFSclone process seems very 'fussy'

  24. #74
    Join Date
    Jun 2013
    Posts
    60
    What was OnApp's response when asked about this?

  25. #75
    Join Date
    Aug 2011
    Location
    Dub,Lon,Dal,Chi,NY,LA
    Posts
    1,839
    To run a CHKDSK

Page 3 of 4 FirstFirst 1234 LastLast

Similar Threads

  1. HostBill/OnApp Issue
    By IshaKaarlo in forum Hosting Software and Control Panels
    Replies: 1
    Last Post: 03-08-2013, 10:36 AM
  2. Bonding Interfaces - CentOS
    By darkspace_co in forum Hosting Security and Technology
    Replies: 12
    Last Post: 05-14-2011, 08:40 AM
  3. MLPPP (ADSL Bonding)
    By TailoredVPS in forum Hosting Security and Technology
    Replies: 2
    Last Post: 10-25-2009, 08:15 PM
  4. Insurance and Bonding?
    By TubuNet in forum Web Hosting
    Replies: 2
    Last Post: 01-01-2007, 09:04 PM
  5. Hosting via ADSL Bonding
    By SME in forum Other Offers & Requests
    Replies: 21
    Last Post: 04-22-2002, 07:07 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •