Web Hosting Talk : Web Hosting Main Forums : Specialty Hosting and Markets : R1SOFT v3 brings down servers (I/O issue)

[lifespeakers]

Hi,

Is there anyone else who had an issue with R1SOFT v3 entreprise? Each week it brings a server down because of an I/O issue. The servers are in RAID10, we have to reboot them in order for them to be back online (we see many "CDP I/O" in the process manager...).

Thank you WHT,

[funkywizard]

You might have full block scan turned on for the backups. You should only be doing a full block scan for the initial backup, or, potentially, for a backup that occurs where the r1soft server isn't sure it has a good record of all the file deltas (like if you had to reinstall the cdp agent after a kernel upgrade). Full block scan is an option you can turn on / off in the backup policy, so you want to make sure it's off. It's also worthwhile to schedule the r1soft backup for an off peak time of day. For all of these reasons (and more) we started doing daily backups instead of hourly, which also helps here.

[lifespeakers]

Hi,

Thank you for your answer. Full block scan is not active, I'll contact R1SOFT directly.

If anyone else had the same issue, please let me know

Best Regards,

[net]

Moved > Specialty Hosting and Markets.

[nibb]

Its normal. They will crash servers frequently, and sometimes cause file corruption on the drives as well which you need to manually fix. Welcome to the world of CDP.

[Mark Muyskens]

Originally Posted by nibb

Its normal. They will crash servers frequently, and sometimes cause file corruption on the drives as well which you need to manually fix. Welcome to the world of CDP.

nicely done.

[FHDave]

We rarely have issue with CDP. It we do, then my admins have not told me of the issues.

What kind of drives and how many of them do you have on your RAID10? How many servers are you backing up?

[AdmoNet]

* Upgrade your version to the latest available version
* Build a new kernel module (r1soft-setup --get-module) and then restart your CDP agent (/etc/init.d/cdp-agent restart)

There were lots of older versions of their kernel module that created IO issues. Please verify you are on the latest greatest versions. We back up quite a few systems without issues.

[Steven]

Are you using mdadm raid and cloudlinux?

[ethical]

i've been having this issue since January with no end in sight, did 4.0 fix it? nope.

the error related is this:

An exception occurred during the request. Unable to stop snapshot for device '/dev/xvda#' with id 1: Operation not permitted

and then when the next scheduled backup starts, it cant tell that something is still running and causes the CPU to surge and this kills the server.

they keep saying it will be fixed but nothing.

also any attempt to stop the cdp process if you can catch after the first bad backup, fails with any attempts i've tried so you STILL have to reboot to clear the issue (although at least you can turn off the backup and your server wont hang so you can do it at a good time)

yep im using cloudlinux

[chennaihomie]

Originally Posted by ethical

yep im using cloudlinux

Are you using Cloudlinux 6?
Check this thread out there seem to be people reporting performance issues here
http://www.webhostingtalk.com/showth...1155043&page=2

[prickett233]

Originally Posted by nibb

Its normal. They will crash servers frequently, and sometimes cause file corruption on the drives as well which you need to manually fix. Welcome to the world of CDP.

In our CDP world we use box backup and have never had a single stability issue with it. We also have rolled our own CDP like backup system using some custom server side scripts called by bacula.

[ethical]

Originally Posted by chennaihomie

Are you using Cloudlinux 6?
Check this thread out there seem to be people reporting performance issues here
http://www.webhostingtalk.com/showth...1155043&page=2

nope, Im using CL5 but i will read through that link thanks.

[nibb]

The issue is back like never before in version 5. Just had 2 crash in 1 week since I upgraded to Idera Server Backup version 5. In all of them R1Soft was doing a backup and not only crashed the VM like it was normal in v3, but crashed the whole dom0 node !!! The whole hardware went crazy because of high I/O load.

When rebooting the node, the agent was still doing the backup, it never failed, even while the hardware was being rebooted, 5 minutes after it was online, it hung again because r1soft server was still hitting the server, cancelling the backup task immediately made the node respond again. This is not bad. This is AWFUL !!!

Xenserver 6 will give all type of errors under load like Input/ouput errors, without letting you enter any command at all. Stopping the backup task solves the problem.

[ethical]

i've found mine to still be pretty stable so far with 5, what did support say about it I dont want to see this happening again??

thanks