hosted by liquidweb


Go Back   Web Hosting Talk : Web Hosting Main Forums : Specialty Hosting and Markets : R1SOFT v3 brings down servers (I/O issue)
Reply

Forum Jump

R1SOFT v3 brings down servers (I/O issue)

Reply Post New Thread In Specialty Hosting and Markets Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old 04-14-2012, 07:32 AM
lifespeakers lifespeakers is offline
WHT Addict
 
Join Date: Jan 2008
Location: Montreal, Canada
Posts: 133

R1SOFT v3 brings down servers (I/O issue)


Hi,

Is there anyone else who had an issue with R1SOFT v3 entreprise? Each week it brings a server down because of an I/O issue. The servers are in RAID10, we have to reboot them in order for them to be back online (we see many "CDP I/O" in the process manager...).

Thank you WHT,



Sponsored Links
  #2  
Old 04-14-2012, 11:28 AM
funkywizard funkywizard is offline
unghhh... Baaandwidth....
 
Join Date: Jan 2005
Posts: 8,631
You might have full block scan turned on for the backups. You should only be doing a full block scan for the initial backup, or, potentially, for a backup that occurs where the r1soft server isn't sure it has a good record of all the file deltas (like if you had to reinstall the cdp agent after a kernel upgrade). Full block scan is an option you can turn on / off in the backup policy, so you want to make sure it's off. It's also worthwhile to schedule the r1soft backup for an off peak time of day. For all of these reasons (and more) we started doing daily backups instead of hourly, which also helps here.

__________________
IOFLOOD.com -- We Love Servers
Need More Ram? 12 core 72gb ram servers are here!
Email (sales [at] ioflood . com) for details.

  #3  
Old 04-15-2012, 12:34 AM
lifespeakers lifespeakers is offline
WHT Addict
 
Join Date: Jan 2008
Location: Montreal, Canada
Posts: 133
Hi,

Thank you for your answer. Full block scan is not active, I'll contact R1SOFT directly.

If anyone else had the same issue, please let me know

Best Regards,

Sponsored Links
  #4  
Old 04-15-2012, 01:01 AM
net net is offline
Community Liaison
 
Join Date: Mar 2003
Posts: 10,881
Moved > Specialty Hosting and Markets.

  #5  
Old 04-15-2012, 06:00 PM
nibb nibb is offline
Web Hosting Master
 
Join Date: Jun 2005
Posts: 2,720
Its normal. They will crash servers frequently, and sometimes cause file corruption on the drives as well which you need to manually fix. Welcome to the world of CDP.

__________________
PingHosters - Expert community for hosters - NEW: Post Reviews

  #6  
Old 04-15-2012, 06:41 PM
Mark Muyskens Mark Muyskens is offline
Rebooting is a hack, not a fix
 
Join Date: May 2008
Location: Citrus Heights, CA
Posts: 1,638
Quote:
Originally Posted by nibb View Post
Its normal. They will crash servers frequently, and sometimes cause file corruption on the drives as well which you need to manually fix. Welcome to the world of CDP.
nicely done.

__________________
Best Regards,

Mark

  #7  
Old 04-15-2012, 08:40 PM
FHDave FHDave is offline
Web Hosting Master
 
Join Date: Jun 2002
Location: PA, USA
Posts: 5,122
We rarely have issue with CDP. It we do, then my admins have not told me of the issues.

What kind of drives and how many of them do you have on your RAID10? How many servers are you backing up?

  #8  
Old 04-15-2012, 10:17 PM
AdmoNet AdmoNet is offline
Web Hosting Evangelist
 
Join Date: Mar 2003
Location: Kansas City, Missouri
Posts: 461
* Upgrade your version to the latest available version
* Build a new kernel module (r1soft-setup --get-module) and then restart your CDP agent (/etc/init.d/cdp-agent restart)

There were lots of older versions of their kernel module that created IO issues. Please verify you are on the latest greatest versions. We back up quite a few systems without issues.

__________________
=>Admo.net Managed Hosting
=> Managed Hosting • Dedicated Servers • SolusVM VPS • vmware ESX VM's
=> Dark Fiber Access to 1102 Grand, Multiple Public Providers
=> Over •Sixteen• Years of Service

  #9  
Old 04-15-2012, 10:23 PM
Steven Steven is offline
Problem Solver
 
Join Date: Mar 2003
Location: California USA
Posts: 12,917
Are you using mdadm raid and cloudlinux?

__________________
Steven Ciaburri | Proactive Linux Server Management - Rack911.com
System Administration Extraordinaire | Follow us on twitter:@Rack911Labs
Managed Servers (AS62710), Server Management, and Security Auditing.
www.HostingSecList.com - Security notices for the hosting community.

  #10  
Old 06-05-2012, 09:25 AM
ethical ethical is offline
Junior Guru Wannabe
 
Join Date: Jul 2006
Posts: 88
i've been having this issue since January with no end in sight, did 4.0 fix it? nope.

the error related is this:

An exception occurred during the request. Unable to stop snapshot for device '/dev/xvda#' with id 1: Operation not permitted

and then when the next scheduled backup starts, it cant tell that something is still running and causes the CPU to surge and this kills the server.

they keep saying it will be fixed but nothing.

also any attempt to stop the cdp process if you can catch after the first bad backup, fails with any attempts i've tried so you STILL have to reboot to clear the issue (although at least you can turn off the backup and your server wont hang so you can do it at a good time)

yep im using cloudlinux

__________________
Best Web Hosting Rebates - Coupons & Rebates off top hosts including Green Geeks, Lunarpages & BlueHost.
Hostgator Coupons - 25% Coupon - Save up to $599

  #11  
Old 06-05-2012, 10:14 AM
chennaihomie chennaihomie is offline
The Guru!
 
Join Date: Nov 2007
Location: Chennai, India
Posts: 2,355
Quote:
Originally Posted by ethical View Post
yep im using cloudlinux
Are you using Cloudlinux 6?
Check this thread out there seem to be people reporting performance issues here
http://www.webhostingtalk.com/showth...1155043&page=2

  #12  
Old 06-05-2012, 03:16 PM
prickett233 prickett233 is offline
WHT Addict
 
Join Date: Jan 2007
Posts: 158
Quote:
Originally Posted by nibb View Post
Its normal. They will crash servers frequently, and sometimes cause file corruption on the drives as well which you need to manually fix. Welcome to the world of CDP.
In our CDP world we use box backup and have never had a single stability issue with it. We also have rolled our own CDP like backup system using some custom server side scripts called by bacula.

  #13  
Old 06-05-2012, 03:37 PM
ethical ethical is offline
Junior Guru Wannabe
 
Join Date: Jul 2006
Posts: 88
Quote:
Originally Posted by chennaihomie View Post
Are you using Cloudlinux 6?
Check this thread out there seem to be people reporting performance issues here
http://www.webhostingtalk.com/showth...1155043&page=2
nope, Im using CL5 but i will read through that link thanks.

__________________
Best Web Hosting Rebates - Coupons & Rebates off top hosts including Green Geeks, Lunarpages & BlueHost.
Hostgator Coupons - 25% Coupon - Save up to $599

  #14  
Old 12-11-2012, 01:39 PM
nibb nibb is offline
Web Hosting Master
 
Join Date: Jun 2005
Posts: 2,720
The issue is back like never before in version 5. Just had 2 crash in 1 week since I upgraded to Idera Server Backup version 5. In all of them R1Soft was doing a backup and not only crashed the VM like it was normal in v3, but crashed the whole dom0 node !!! The whole hardware went crazy because of high I/O load.

When rebooting the node, the agent was still doing the backup, it never failed, even while the hardware was being rebooted, 5 minutes after it was online, it hung again because r1soft server was still hitting the server, cancelling the backup task immediately made the node respond again. This is not bad. This is AWFUL !!!

Xenserver 6 will give all type of errors under load like Input/ouput errors, without letting you enter any command at all. Stopping the backup task solves the problem.

__________________
PingHosters - Expert community for hosters - NEW: Post Reviews

  #15  
Old 12-17-2012, 11:43 PM
ethical ethical is offline
Junior Guru Wannabe
 
Join Date: Jul 2006
Posts: 88
i've found mine to still be pretty stable so far with 5, what did support say about it I dont want to see this happening again??

thanks

__________________
Best Web Hosting Rebates - Coupons & Rebates off top hosts including Green Geeks, Lunarpages & BlueHost.
Hostgator Coupons - 25% Coupon - Save up to $599

Reply

Similar Threads
Thread Thread Starter Forum Replies Last Post
R1Soft issue warlock-m VPS Hosting 6 04-06-2011 07:28 AM
R1Soft issue on Xen VM HarrySX Hosting Software and Control Panels 2 04-02-2011 08:47 AM
Secret Santa Brings Servers @ $15.00 Over Cost!! Win a Nintendo Wii or Ipod iTouch RobertMaltby Dedicated Hosting Offers 0 12-16-2007 11:14 PM
Secret Santa Brings Servers @ $15.00 Over Cost - Win a Nintendo Wii or Ipod iTouch!! RobertMaltby Dedicated Hosting Offers 9 12-12-2007 05:38 PM
MegaNetServe brings back $49 /mo. Dedicated Servers MegaNetServe Dedicated Hosting Offers 7 03-17-2005 05:13 PM

Related posts from TheWhir.com
Title Type Date Posted
More Downtime for HostGator and BlueHost Customers as Router Issues Plague Utah Data Center Web Hosting News 2014-04-16 15:46:56
eTail West 2014 Web Hosting Events 2014-02-06 18:52:42
Windows Azure, Xbox Live Experience Problems as Xbox One Launches Web Hosting News 2013-11-22 13:05:40
HostGator Continues to Restore Hosting Services After Data Center Outage Web Hosting News 2013-11-20 16:54:54
WHMCS Security Issue Allows for Information Disclosure Web Hosting News 2013-10-25 09:30:46


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
Advertisement:
Web Hosting News:



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?