Web Hosting Talk : Web Hosting Main Forums : Dedicated Server : SolusVM Xen Linux slow I/O performance.

[Cookiesowns]

Hey Guys,

I have a Dual E5620 24GB Dell R410 with two WD 7.200K drives in raid-0.

I'm running Latest SolusVM using Xen HVM, and well, the linux dd performance is pretty much garbage. 15MB/s.

As for the Windows VM under the same node can get about 260/110 Sequential, and 4K QD32 is 9.5/2.5

The linux machine.


root@vps [~]# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 80.5182 seconds, 13.3 MB/s


Same test on a E3-1240 16GB, Xen HVM VM, but this volume is on a Intel 80GB G2 SSD

dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 24.7843 seconds, 43.3 MB/s

Same test on the Dual E5620 24GB host node.

dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync

16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 7.26415 seconds, 148 MB/s

Well... I'm confused.

[Flapadar]

Out of curiosity, why are you using RAID-0? A single failure and everything is gone. Do you need faster write access compared to RAID1?

This is from a RAID1 2x3TB 7200 rpm.

# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 20.7501 seconds, 51.7 MB/s

I can't see why your speeds are that bad though. Really should be at least 30-40MB/s, preferably higher on those specs.

[PCS-Chris]

What version of Xen?

Xen 3.1.2 is garbage for HVM..

[Dougy]

Originally Posted by PCS-Chris

What version of Xen?

Xen 3.1.2 is garbage for HVM..

3.1.2 has a lot of backported fixes, it's not that bad, I used it for a good while.

3.4 and 4.0 and 4.1 are obviously a better choice, though.

[Cookiesowns]

Originally Posted by Flapadar

Out of curiosity, why are you using RAID-0? A single failure and everything is gone. Do you need faster write access compared to RAID1?

This is from a RAID1 2x3TB 7200 rpm.

# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 20.7501 seconds, 51.7 MB/s

I can't see why your speeds are that bad though. Really should be at least 30-40MB/s, preferably higher on those specs.

We're using Xen HVM 3.1.2, as that's the only version SolusVM officially supports, I tried upgrading to 4.1, but that just screwed up the bridges.

As far as data protection goes, we were running 1 drive + 1 ssd on our other nodes, we're only running 3-4 VM's on these, and all of them are replicated to an offsite FTP. It's either slow protection with vm's load balanced between two hdd's or just one raid-0 on the built in dell 1068E sas controller.

Anyone have ideas on how to upgrade Xen without bricking our previous VM's as well as solusvm integration?

[PCS-Chris]

Xen 3.1.2 does have lots of fixes backported but it doesn't change the fact its very old software at it's core. Newer versions of Xen work fine with SolusVM and always have, the only thing you need to change is the Xen Toolstack version under Node > Node Properties to XL if you are using Xen 4.

How Xend sets up the bridges depends entirley on the network-bridge and vif-bridge scripts in /etc/xen/scripts. If you are compiling from source or using someone elses RPM's chances are they will be different to CentOS's defaults.

You can keep your old files, modify them or if you are compiling from source you should be able to take a .spec file from a Fedora srpm and modify from there. Of course do this on a test box and test it well first..

[Cookiesowns]

Originally Posted by PCS-Chris

Xen 3.1.2 does have lots of fixes backported but it doesn't change the fact its very old software at it's core. Newer versions of Xen work fine with SolusVM and always have, the only thing you need to change is the Xen Toolstack version under Node > Node Properties to XL if you are using Xen 4.

How Xend sets up the bridges depends entirley on the network-bridge and vif-bridge scripts in /etc/xen/scripts. If you are compiling from source or using someone elses RPM's chances are they will be different to CentOS's defaults.

You can keep your old files, modify them or if you are compiling from source you should be able to take a .spec file from a Fedora srpm and modify from there. Of course do this on a test box and test it well first..

Gotcha.

Do solusvm write the scripts or no?

[Dougy]

Originally Posted by Cookiesowns

We're using Xen HVM 3.1.2, as that's the only version SolusVM officially supports, I tried upgrading to 4.1, but that just screwed up the bridges.

As far as data protection goes, we were running 1 drive + 1 ssd on our other nodes, we're only running 3-4 VM's on these, and all of them are replicated to an offsite FTP. It's either slow protection with vm's load balanced between two hdd's or just one raid-0 on the built in dell 1068E sas controller.

Anyone have ideas on how to upgrade Xen without bricking our previous VM's as well as solusvm integration?

It's a whopping five minutes max per server to fix the bridges.

[Cookiesowns]

Originally Posted by Dougy

It's a whopping five minutes max per server to fix the bridges.

Any advice?

[Dougy]

Originally Posted by Cookiesowns

Any advice?

Sadly I don't recall offhand, but even if I did, I don't remember how to make solusvm happy.

I only ever did it via CLI. I beleive setting bridge name in xend config helped me.

[Cookiesowns]

Originally Posted by Dougy

Sadly I don't recall offhand, but even if I did, I don't remember how to make solusvm happy.

I only ever did it via CLI. I beleive setting bridge name in xend config helped me.

Got it.

Anyways, I tested this on our old non production E3-1240 node, things seems to be much better after upgrading to 3.4.3. I used the gitco repo's to update and just re-added the solusVM links, infact the VM's didn't even have to go down, until I needed to reboot the machine.

Awesome right!

[Dougy]

Originally Posted by Cookiesowns

Got it.

Anyways, I tested this on our old non production E3-1240 node, things seems to be much better after upgrading to 3.4.3. I used the gitco repo's to update and just re-added the solusVM links, infact the VM's didn't even have to go down, until I needed to reboot the machine.

Awesome right!

3.4.3 is good.

[Cookiesowns]

Originally Posted by Dougy

3.4.3 is good.

Yup, though for network performance wise using the Intel E1000 emulation drivers, I'm only able to push around 300mbit downstream, and 150-200 up.

Servers are on gigabit, and testing are done from two nodes within the same switch under 1gbit, node - node can push 900mbit, vps to host or secondary node, only about 300mbits

[PCS-Chris]

Are you running CentOS 5.7? If not just a friendly bit of advice.. there are things in CentOS 5.7 which can and probably will break Gitco's Xen 3.4.3 build sooner or later.

[viGeek]

Can you post the output from the following...

shell#> cat /proc/diskstats
shell#> iostat
shell#> vmstat
shell#> blockdev --getra /dev/drive#
shell#> cat /sys/block/drive#/queue/nr_requests
shell#> cat /sys/block/drive#/queue/scheduler
shell#> cat /sys/block/drive#/queue/read_ahead_kb

Replace drive# with the drive in question (i.e xvda, sda, hda etc)

Also run this in you can (in place of diskstats).

#!/bin/bash
# Part of sysinfo (https://github.com/vigeek/sysinfo)
for ios in `cat /proc/diskstats |  awk '{$1="" ; $2="" ; print}' | egrep '[shxv]d[a-z][^1-9]' | awk '{print $1}'` ; do

  echo -e "Reads Complted: $(cat /proc/diskstats | grep $ios | egrep '[shxv]d[a-z][^1-9]' | awk '{print $4}')"

  echo -e "Time Reading: $(cat /proc/diskstats | grep $ios | egrep '[shxv]d[a-z][^1-9]' | awk '{print $7}' | awk '{ s += $1 } END { print s/1000}') [sec"

  echo -e "Writes Completed: $(cat /proc/diskstats | grep $ios | egrep '[shxv]d[a-z][^1-9]' | awk '{print $8}')"

  echo -e "Time Writing: $(cat /proc/diskstats | grep $ios | egrep '[shxv]d[a-z][^1-9]' | awk '{print $11}' | awk '{ s += $1 } END { print s/1000}') [sec]"

  echo -e "Current IO Requests: $(cat /proc/diskstats | grep $ios | egrep '[shxv]d[a-z][^1-9]' | awk '{print $12}')"

  echo -e "Time doing IO: $(cat /proc/diskstats | grep $ios | grep '[shxv]d[a-z][^1-9]' | awk '{print $13}') ms"

done