Results 51 to 75 of 86
Thread: OnApp Bonding issue
-
09-27-2013, 11:16 AM #51Newbie
- Join Date
- Nov 2009
- Location
- Reading, UK
- Posts
- 17
@ditlev
We're running as many tests as we can at the moment, but one thing that stands out straight away is the testing of the raw drives vs the testing of them as a single drive in a Vdisk.
Firstly ran the tests on the raw drives :
Code:dd if=/dev/zero of=/dev/sd* bs=10M count=100 oflag=direct dd if=/dev/sd* of=/dev/null bs=10M count=100 iflag=direct
Code:dd if=/dev/zero of=/dev/nbd* bs=10M count=100 oflag=direct dd if=/dev/nbd* of=/dev/null bs=10M count=100 iflag=direct
Code:Vdisk Raw Differrence Hypervisor 1 W: 285 MB/s W: 320 MB/s W: -35 MB/sec R: 335 MB/s R: 457 MB/s R: -127 MB/sec Hypervisor 2 W: 268 MB/s W: 320 MB/s W: -52 MB/sec R: 349 MB/s R: 457 MB/s R: -108 MB/sec Hypervisor 3 W: 274 MB/s W: 320 MB/s W: -46 MB/sec R: 320 MB/s R: 457 MB/s R: -101 MB/sec
-
09-27-2013, 01:10 PM #52WHT Addict
- Join Date
- May 2013
- Posts
- 112
-
09-27-2013, 03:46 PM #53Newbie
- Join Date
- Nov 2009
- Location
- Reading, UK
- Posts
- 17
Same Hypervisor, all drives are local, running the test from onapp web interface.
ddwrapper BS=4096 Disk Size=10GB
1 Stripe : W: 246 MB/s R: 283 MB/s
2 Stripes: W: 379 MB/s R: 434 MB/s
4 Stripes: W: 426 MB/s R: 528 MB/s
-
09-27-2013, 04:00 PM #54Web Hosting Master
- Join Date
- Mar 2009
- Location
- NL
- Posts
- 594
YISP - High Bandwidth dedicated servers and colocation in YISP-AS(Amsterdam)!
Website: http://www.yisp.nl
Contact: info "(AT)" yisp.nl
-
09-27-2013, 04:01 PM #55WHT Addict
- Join Date
- May 2013
- Posts
- 112
Can you try doing a remote path? If those disks are on (EX) HV1, try testing it from HV2. That way you know what the actual performance will be.
When doing at least 1 replica, the data will be copied to a different server. So if the disks are local, when you do a Write test it has to write to the disks locally, and to the remote copy at the same time. The remote path is what is killing performance. Once you have 8 disks involved (4 local, 4 remote - 1 replica + 4 stripes) the write/read is very low.
-
09-27-2013, 04:42 PM #56Newbie
- Join Date
- Nov 2009
- Location
- Reading, UK
- Posts
- 17
ddwrapper BS=4096 Disk Size=10GB ( All Tested from HV 1 )
1 Stripe &1 Copy: W: 228 MB/s R: 336 MB/s ( 1x HV1 - 1x HV2 )
2 Stripes &1 Copy: W: 336 MB/s R: 484 MB/s ( 2x HV1 - 2x HV2 )
2 Stripes &1 Copy: W: 342 MB/s R: 119 MB/s ( 1x HV1 - 1x HV2 - 2x HV3 )
4 Stripes &1 Copy: W: 371 MB/s R: 564 MB/s ( 4x HV1 - 4x HV2 )
4 Stripes &1 Copy: W: 338 MB/s R: 180 MB/s ( 3x HV1 - 2x HV2 - 3x HV3)
118MB/sec or 180MB/sec obviously isn't anywhere near good enough for production
Also, why are we losing 100MB/sec of read performance on local drives even when they are just on their own ?
-
09-27-2013, 06:53 PM #57WHT Addict
- Join Date
- May 2013
- Posts
- 112
The last test you did, 4 stripes & 1 Replica with all your HV is a more realistic test since you will have more than 2 HV's eventually in production. I do think there is an issue with the dm-multipath. With 4 stripes, and 1 copy, you should be getting awesome performance. We brought up this performance issue 3 months ago.
-
10-04-2013, 03:36 PM #58New Member
- Join Date
- Sep 2013
- Posts
- 3
Michail's been testing a duplicate environment I setup to our production configuration and believes the issue is related to TCP Incast:
h t t p : //bradhedlund.com/2011/05/01/tcp-incast-and-cloud-application-performance/
Lowering the MTU helped slightly, but we're still seeing terrible remote read performance. With 4 stripes, 2 copies on SAS 15K disks and a MTU of 9000, we were getting 60MB/s write and 15MB/s read remotely. Dropping the MTU to 1500 got us to about 30MB/s read remotely (write stayed the same). Switching that to 2 stripes and 2 copies gets me to 67MB/s read remotely.
We're using a stack of two 2960X switches which each have a pair of ASIC's with 2MB buffers from what I read. I am seeing output drops which seems to confirm the TCP Incast issue (though I'd expect to see input drops since the nature of the issue is that the receiving system cannot process traffic fast enough).
Things I've tried:
- Add additional storage nic to only the remote node. Tried no etherchannel, etherchannel and lacp configurations on the switch. Performance results were ~5% of a single nic.
- Increase the hold-queues on the IS switchports. However, from what I read this only applies to process switched packets so that didn't help either.
- QoS is disabled on our switches, so I tried enabling it and tweaking the buffers, but that just caused more drops. Performance was roughly the same.
- Various MTU settings of 800, 500, and 284. Performance was roughly the same as 1500.
- Bonding on local and remote nodes. The only configuration that increased performance was a split VLAN (two nics on each system, different VLANs per nic). Split VLAN doubled my numbers, but requires that ALL systems have nics in those VLANs (ie. your backup and provisioning servers need two nics too).
What are others using for switches? I've heard the 4948's have larger buffers which can help.
-
10-04-2013, 03:58 PM #59WHT Addict
- Join Date
- May 2013
- Posts
- 112
We were told the same thing today (same link) and have tried many other settings like you. We tried dropping the Round Robin and went the ol-reliable LACP route, and also tried TLB and ALB. We can't get it to improve performance. Host to Host bandwidth throughput is great, a solid 20Gbps (2+ threads). Throughput to the storage controllers are awful, maybe 500Mbps. Only thing we haven't tried is the MTU changes. Just seems like we are running in circles here. Seems a bit odd that we would have line rate capacity issues on a bonded 2x 10Gig port, when only doing 500Mbps. Also tried going a single 10Gig to the SAN network on all servers (no bonding) didn't help much either.
We are going to spend a bit of time investigating the buffer issue theory to see if we can improve it. Nothing is off the table at this point, and I hope Julian (OnApp Storage Dev) keeps investigating this issue.
-
10-07-2013, 03:38 PM #60New Member
- Join Date
- Sep 2013
- Posts
- 3
Based on additional testing, it seems that only the number of replicas affects this issue. A datastore with 2 copies and 4 stripes has the same read performance characteristics as a datastore with 2 copies and 2 stripes.
There is a command that can affinitize traffic to a single node on the remote server: /usr/pythoncontroller/setPrio. When I used that command to direct a vdisk to a single hosted, I was able to get line rate read speed.
From the remote node:
Get the remote node's uuid:
onappstore getid
Find the vdisk id and members:
onappstore list
Find the hostid of a member:
onappstore nodeinfo uuid=<member id>
Online the disk on the remote node:
onappstore online uuid=<vdisk id> frontend_uuid=<node id>
Affinitize vdisk to specific node:
setPrio uuid=<vdisk id> hostid=<host id>
Run the test:
/.rw/ddwrapper device=/dev/mapper/<vdisk id> bs=4096K
Hoping OnApp can add a work-around leveraging this affinity to a single node.
-
10-07-2013, 10:58 PM #61WHT Addict
- Join Date
- May 2013
- Posts
- 112
The issue is mainly that TCP Incast issue, we can confirm it being an issue. It explains that when you have more replicas, it kills the performance since the packets can't get transmitted. The issue is the many-to-one (READS) that is killing the performance. So that SetPrio script helps limit the amount of servers responding to the request. We saw a huge performance increase on READS when using that. It doesn't explain why the WRITES are slow (one-to-many). Must be some more TCP Incast issues.
OnApp storage team will have to get very creative to over come this limitation, as there is no switch that can fix this...not even the expensive EX4550's we are using. The SetPrio helped with the reads, but to get anything faster a regular RAID Controller would be preferred as it could limit the TCP Incast issues (fewer disks responding to a query).█ Cloud IaaS Solutions Provider - www.CloudVZ.com
█ SSD SANs | High IOPs | Public & Private Cloud Solutions | Content Delivery Network
█ Create your virtual datacenter in seconds!
-
10-10-2013, 11:10 PM #62WHT Addict
- Join Date
- May 2013
- Posts
- 112
Good news, Julian has informed us that they are working on a mechanism to distribute the storage bandwidth better so that it helps reduce the TCP Incast issue that people are experiencing.
-
10-11-2013, 12:49 AM #63Web Hosting Master
- Join Date
- May 2003
- Location
- San Francisco, CA
- Posts
- 1,506
OnApp Bonding issue
Thanks for the update.
* GeekStorage.com - Offering awesome website hosting for over 13 years!
* Shared Hosting * Reseller Hosting * Virtual Private Servers * Dedicated Servers
* Have questions? Send us an e-mail, we'd love to hear from you!
-
10-11-2013, 08:37 AM #64Web Hosting Master
- Join Date
- Oct 2001
- Location
- Ohio
- Posts
- 8,535
-
10-11-2013, 09:34 AM #65Web Hosting Master
- Join Date
- Jan 2004
- Location
- Pennsylvania
- Posts
- 942
Some quick research shows using the RED queuing algorithm is enough to fix TCP incast problem. Have you tried adding it to your nodes?
Code:tc qdisc add dev eth0 parent 1:1 handle 10: red limit 4000000 ecn bandwidth 10000Mbit
Matt Ayres - togglebox.com
Linux and Windows Cloud Virtual Datacenters powered by Onapp / Xen
Instant Setup, Instant Scalability, Full Lifecycle Hosting Solutions
www.togglebox.com
-
10-11-2013, 11:16 AM #66Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance
-
10-11-2013, 11:26 AM #67Web Hosting Master
- Join Date
- Oct 2001
- Location
- Ohio
- Posts
- 8,535
-
10-14-2013, 10:48 AM #68Junior Guru Wannabe
- Join Date
- Jun 2013
- Posts
- 60
We're trying to work out the best way of doing our storage network. It seems we're getting a different answer depending on who we talk to on OnApp support.
We've 2 HV's and 1 Backup server. All of them have 4 Intel Pro NIC's @ 1Gbps on storage network. Storage switch is a HP2920G. Initially we tried them in LACP, but didn't get huge data rates accross HV's. Then moved to RR (on advice of OnApp support) and had to put each nic in separate VLAN and this gave us pretty much 4Gbps between nodes... Once we'd configured our storage with stripes then we managed around 500MB/s transfer rate...
However, once we started using this in production we had backups failing on a regular basis and OnApp support (different person) have said that having each NIC in a separate VLAN is not a good idea if a NIC goes down then data could be lost between them causing backup to fail.
If we have all the NIC's in the same VLAN on the HP then we get duplicate packet issues (on network tests) and I can imagine this would cause problems with OnApp backups and provisioning. LACP gives us the resiliency but not the performance (that I'd expect). Am I missing something simple here to get both resiliency and performance?
-
10-14-2013, 11:01 AM #69Aspiring Evangelist
- Join Date
- Nov 2012
- Posts
- 428
What's the actual cause of the failing backups? You can rule out network issues by doing a ping test to see if you loose any packets over a period of time.
Yes, there is always the risk of a NIC failing and it breaks the path. However, I think your switch will go long before your NIC craps out on you .
-
10-14-2013, 11:06 AM #70Junior Guru Wannabe
- Join Date
- Jun 2013
- Posts
- 60
Unable to pin that down yet... They were just failing on one HV, which led me to suspect network, but that looks fine as far as I can tell... No signs of NIC's going up and down that I can spot so far... I might try a good old re-seat of the NIC's (they're on PCI not onboard) just to be sure nothing dodgy there.
I've run a ping test at the maximum MTU of the storage network for a decent amount of time and all looks good so far.
When the backups fail we end up with orphaned disks in our store that then need to delete before the backup will run again.
-
10-14-2013, 11:30 AM #71Disabled
- Join Date
- Aug 2011
- Location
- Dub,Lon,Dal,Chi,NY,LA
- Posts
- 1,839
Anything in the log suggesting the reason for failure?
-
10-14-2013, 11:38 AM #72Junior Guru Wannabe
- Join Date
- Jun 2013
- Posts
- 60
Support seems to think it's corrupt windows file systems, but running chkdsk /f on those reports as ok every time so far. The error is on ntfsclone and then the rollback seems to fail.
I was suspecting the network as the backups seem to work fine on one of the HV's with no issues (so far), but can't see anything obvious there.
Have any of you guys had problems with balance-RR and separate VLAN's on the switch?
-
10-14-2013, 11:54 AM #73Disabled
- Join Date
- Aug 2011
- Location
- Dub,Lon,Dal,Chi,NY,LA
- Posts
- 1,839
We had a similar issue with some Windows VMs that several CHKDSK and offline / mounted fscks eventually resolved. The NTFSclone process seems very 'fussy'
-
10-14-2013, 11:56 AM #74Junior Guru Wannabe
- Join Date
- Jun 2013
- Posts
- 60
What was OnApp's response when asked about this?
-
10-14-2013, 12:01 PM #75Disabled
- Join Date
- Aug 2011
- Location
- Dub,Lon,Dal,Chi,NY,LA
- Posts
- 1,839
To run a CHKDSK
Similar Threads
-
HostBill/OnApp Issue
By IshaKaarlo in forum Hosting Software and Control PanelsReplies: 1Last Post: 03-08-2013, 10:36 AM -
Bonding Interfaces - CentOS
By darkspace_co in forum Hosting Security and TechnologyReplies: 12Last Post: 05-14-2011, 08:40 AM -
MLPPP (ADSL Bonding)
By TailoredVPS in forum Hosting Security and TechnologyReplies: 2Last Post: 10-25-2009, 08:15 PM -
Insurance and Bonding?
By TubuNet in forum Web HostingReplies: 2Last Post: 01-01-2007, 09:04 PM -
Hosting via ADSL Bonding
By SME in forum Other Offers & RequestsReplies: 21Last Post: 04-22-2002, 07:07 PM