Web Hosting Talk : Web Hosting Main Forums : Web Hosting : R1soft Backup

[mheaton]

R1Soft is a backup “solution” aimed at companies that want to backup only the changes that have been made since the last time a backup was created. Its a good idea, but it went horribly wrong. I am writing about it today because it was such a poor experience working with them and we had so many server problems and REPEATED DATA LOSS that I felt compelled to warn other hosting companies and customers about this extremely poor product.

R1soft caused enormous I/O load that literally killed our servers off and on multiple occasions simply would not restore our data and caused 100% data loss on the servers we had “backed up”.

MAJOR TECHNICAL DEFICIENCIES AND OPERATIONAL ISSUES WITH R1SOFT -

1) Buagent (Their software that records what has changed) is a HUGE i/o pig. It constantly blocks on I/O when there are changes causing every other drive or array in our system to lag while it catches up. R1soft told me it “works great” and to send them over details and they would “Look into it”.
2) Buserver (Their software that writes what has changed) is INCREDIBLY slow in every way that it is virtually unusable when you have a lot of data to back up with tens of thousands block changes every day per server. Clearly it hasn’t been tested well under sustained and real heavy load - at least in my opinion. There are multiple kernel and userspace methods to speed this up, but again it falls flat on its face when trying to keep up with our load.
3) Restores are buggy and VERY unreliable. We have attempted more than 20 restores (whole partitions) and had more than 30% be a complete failure, meaning TOTAL data loss. Doing user level restores were the same story - incomplete backups, unreliable and SLOW restores.
4) R1soft support is lackluster to say the least. When I complained they said, “You only opened 17 tickets in the last 60 days.” 17 tickets? Often when we had issues doing a restore or other problems we would call and get no one to answer the phone. Its comforting to know that when you can’t do a restore for 500 clients that we can possibly talk to someone on Monday who will blow us off and tell us that the newest version of the software MIGHT fix our issue. In the meantime they apologized for losing all our data over and over again. Wonderful.
5) The software is based on filesystem block changes. The problem is that the kernel and ext3 and reiser are changing all the time. As we made updates it constantly broke R1soft. When they would finally fix the problem it usually required us to reseed the entire backup again losing everything that we had previously backed up and putting enormous load on the server again while it started over from scratch

In all my years of hosting I have never had such a bad experience with a “partner” company as R1soft.

Just thought I would let other potential customers and hosting companies know my experience.

[CodyRo]

This is particularly interesting to me because we recently started using R1Soft Backups.

Most of what you say we haven't noticed (at least compared to our old solution) - or if we have it hasn't affected day-to-day operations. The one thing that you said that rings true is point #5. We've had to reseed numerous times because it randomly breaks when there is an update or something silly.. mildly irritating.

I find the software great for the most part - but the whole reseed issue is becoming tedious and annoying.. hopefully they clean up their act and get a more stable release rolling.

I'm still a fan of R1Soft for the time being - the restores have been wonderful so far .

Sort of relevant blog post: http://www.hawkhost.com/blog/2008/07...-a-difference/

It doesn't take into account IO though, which is what you mentioned.

**EDIT** Do you have any experience with any other backup solutions similar to R1Soft? If so what's your experience / opinion?

[Cirtex]

Sorry to hear about your issues Matt, I've heard many good things about r1soft in the past but it seems like it does not work for larger more cpu intensive servers like in your case it seems.

Anyhow hopefully we'll hear an update from r1soft here on this issue.

Cheers

[eming]

We did extensive testing of R1 as well, and while it from a business perspective really seemed like the right thing - my US techs put down their foot with a big "no way". It did work out better in UK2 than in Midphase though.

I can still see strong points to their approach, and might give it a try in a year or so if they have matured the platform by then.

But like Cody sais, what are the alternatives?

[turbovps]

With all due respect and regards to r1soft. I would like to share our experience. First of all no doubt they have wonderful product but when we needed it the most it failed on us.
We are using them for over 1 year now for backing up our servers.
Recently one of our server crashed. We tried bare metal restore it failed, we tried restoring files from backup server it failed because CDP server was not able to write the data.
The CDP server start giving error "Error adding block to buffer"

Internal error: Error restoring file (vz/x/x/x/). ( Error adding block to Buffer )
CDT (tid:2315) AGENT: (Righteous Backup Linux Agent) 1.47.0 build 4343

We contacted r1soft.com immediately. They told us to try upgrading CDP Server to 2.9.0 and running the restore from there. 2.9.0 will write 0's to where it sees bad blocks and restore the file which may solve it.
We upgraded the CDP server to 2.9.0 but it never worked.

Later they said there is something wrong with the integrity of data (md5 checksum) so try backing up folder by folder and file by file.
So we had no option left but follow there instructions. We started restoring the data folder by folder. Whenever we get's the error like "Error adding block to Buffer" we had to start the process again and uncheck the folder/files which were aborting the process.
The biggest problem we found was that there is no option to skip the corrupted file. If we have to copy 10gig of folder and if single file is corrupted it abort the operation. We had horrible time restoring the data. We eventually spent over 24 hours restoring 100gigs of data file by file/ folder by folder. We were able to restore 100% data of the clients but it took over 24 hours.
If every backup was verified on completion why it did not gave any error to us that there is something wrong with disk/safe disk etc and when we started restoring data it was giving us errors left and right.
Eventually R1soft start blaming us there is hardware problem that's why its not restoring. We gave them the proof that if we restore same folder over and over again it does restore. And they finally closed our ticket as resolved.
But the time when we needed it the most it never worked for us the way we expected.

Originally Posted by mheaton

R1Soft is a backup “solution” aimed at companies that want to backup only the changes that have been made since the last time a backup was created. Its a good idea, but it went horribly wrong. I am writing about it today because it was such a poor experience working with them and we had so many server problems and REPEATED DATA LOSS that I felt compelled to warn other hosting companies and customers about this extremely poor product.

R1soft caused enormous I/O load that literally killed our servers off and on multiple occasions simply would not restore our data and caused 100% data loss on the servers we had “backed up”.

MAJOR TECHNICAL DEFICIENCIES AND OPERATIONAL ISSUES WITH R1SOFT -

1) Buagent (Their software that records what has changed) is a HUGE i/o pig. It constantly blocks on I/O when there are changes causing every other drive or array in our system to lag while it catches up. R1soft told me it “works great” and to send them over details and they would “Look into it”.
2) Buserver (Their software that writes what has changed) is INCREDIBLY slow in every way that it is virtually unusable when you have a lot of data to back up with tens of thousands block changes every day per server. Clearly it hasn’t been tested well under sustained and real heavy load - at least in my opinion. There are multiple kernel and userspace methods to speed this up, but again it falls flat on its face when trying to keep up with our load.
3) Restores are buggy and VERY unreliable. We have attempted more than 20 restores (whole partitions) and had more than 30% be a complete failure, meaning TOTAL data loss. Doing user level restores were the same story - incomplete backups, unreliable and SLOW restores.
4) R1soft support is lackluster to say the least. When I complained they said, “You only opened 17 tickets in the last 60 days.” 17 tickets? Often when we had issues doing a restore or other problems we would call and get no one to answer the phone. Its comforting to know that when you can’t do a restore for 500 clients that we can possibly talk to someone on Monday who will blow us off and tell us that the newest version of the software MIGHT fix our issue. In the meantime they apologized for losing all our data over and over again. Wonderful.
5) The software is based on filesystem block changes. The problem is that the kernel and ext3 and reiser are changing all the time. As we made updates it constantly broke R1soft. When they would finally fix the problem it usually required us to reseed the entire backup again losing everything that we had previously backed up and putting enormous load on the server again while it started over from scratch

In all my years of hosting I have never had such a bad experience with a “partner” company as R1soft.

Just thought I would let other potential customers and hosting companies know my experience.

[Mxhub]

Originally Posted by mheaton

3) Restores are buggy and VERY unreliable. We have attempted more than 20 restores (whole partitions) and had more than 30% be a complete failure, meaning TOTAL data loss. Doing user level restores were the same story - incomplete backups, unreliable and SLOW restores.

Our last & only restore failed.. it is a baremetal restore via liveboot.
the problem is the new server is using SAS hardware which r1soft do not support at this time. Yucks!


So, beware if you want to restore.. double check what kind of new hardware you are using .. it is best you use the same type of hardware like the old server..

[TonyB]

We have not experienced any super high i/o while doing the backups. The highest load point was when doing the first seed after that the backups were not terribly noticable or at least not as bad as other methods we've tried in the past. It could be the specifications of our boxes though

The biggest problem we've ran into is we get this:

Backup failed to cleanup from previous interruption: Error reading next free IDXEntry. IDXentry found did not have free flag set. Error reading next free IDXEntry. IDXentry found did not have free flag set.

Then we've also seen:

Internal error: Page number (3678129227) invalid. Max page number allowed is (2147483647).

And finally the best one:
CDT Data File (0) for partition (/dev/sda7) Disk Safe (0) is dirty, repair required.
Backup failed to cleanup from previous interruption: Data File (0) for partition (/dev/sda7) Disk Safe (0) is dirty, repair required.
Backup failed to cleanup from previous interruption: Data File (0) for partition (/dev/sda7) Disk Safe (0) is dirty, repair required.
Previous backup for device (/dev/sda7) is incomplete.

Running a repair did not fix this problem.

Having to reseed does not discriminate. We've had it happen on every configuration:

Intel 2.4ghz Quad CPU | Single and Dual 2.66ghz Harpertowns
CentOS4 32bit CentOS5 32bit CentOS5 64bit with varying kernels
Raid-10, Raid-5
Low load machines high load machines

I have yet to get an answer on the best settings to not get this constantly. Should backups have compression, should they be verified or not I do not know. We've tried several agent versions and CDP server versions with no fix.

If they could just fix this one issue we'd be set. Sad thing is it's still better than the alternatives we've tried.

[gate2vn]

Originally Posted by TonyB

Sad thing is it's still better than the alternatives we've tried.

Can you share what you have tried?

[TonyB]

Originally Posted by gate2vn

Can you share what you have tried?

cPanel's various systems
BMU doing incremental over 7 days to make a full set of account backups
Rsync

Then you have stuff like Evault which do not do MySQL and then you have your super expensive systems.


Each has their own flaws some being no tight integration with others being slow.

With R1Soft at least we're able to back up 200GB of data on a box without it taking 3 days to generate the backup and there being a tiny bit of integration.

[MarioR]

http://forum.r1soft.com/showthread.php?p=4240#post4240

We have officially responded within our forums to this much anticipated thread. We were threatened that this would haapen so it comes as no surprise to us that Matt chose to do this.

We recently attended Hostingcon where we had the great opportunity to meet with Matt and some of his team. nothing about this was discussed as I presume he may not have had all the outcome updates of what all triggered this. We also met with many other R1Soft clients as well and just like I mentioned to them if I said our solutions work 100% of the time I would be lying that number to me personally is more like 90% with that 10% using or expecting our solutions to do something they were not designed to do.

I dont believe all the facts were presented and if Matt would like to discuss this openly we can, so what will it be Matt? Would you like us to talk openly here about it all?

[CodyRo]

R1Soft’s sales team and its executive management (me) told Matt Heaton directly numerous times that categorically without exception the software will not function correctly when the agent (buagent) is installed on the same computer as the backup repository (buserver), especially in high performance Linux servers like yours that have already highly loaded disk I/O demands.

Well that little nugget was emitted from the OP's post. Thanks for the reply Mario / David - it is appreciated .

[bqinternet]

We offered a trial R1Soft service to some of our customers, and decided a few months ago to discontinue it. R1Soft has the potential to be a great product, but our experience and that of our customers showed that it's not quite ready.

There was a 67-post thread about various problems back in April:
http://www.webhostingtalk.com/showthread.php?t=688770

I've met some of the R1Soft guys in person at Hostingcon a few years ago, and they seem like nice guys. I'd like to see them get the kinks out and have a polished product.

[CrazyPenguin]

Originally Posted by MarioR

We have officially responded within our forums to this much anticipated thread. We were threatened that this would haapen so it comes as no surprise to us that Matt chose to do this.

Isn't the official response from R1soft found here at
http://forum.r1soft.com/showthread.php?t=862?

[FS - Mike]

Personally, I use cPanel's built in back up feature. For the small amount of accounts we have per server, it works perfectly for us. Backups are done every day, with cron jobs operating on the receiving server to ensure that backups are archived and kept.

Mike

[dynamicnet]

Greetings Matt:

We've been using r1soft.com's software solution for about a year now with no medium to major problems with minor problems being promptly resolved by their support department.

We currently backup 19 CentOS, RedHat, and Windows 2003 servers every day with close to zero load on each server let along the backup server.

We've done on demand backups and restores in prime time without any server load or customer impact.

All restores over the past year have been 100% successful, 100% of the time. Thank God we've only had to restore directories and files.

The company and product have done such a tremendous job that when our managed service clients ask us for backup recommendations, we recommend http://www.r1soft.com/ as the very 1st and best choice.

Thank you.