Results 1 to 10 of 10
  1. #1
    Join Date
    Aug 2005
    Posts
    421

    Fast backup solution for large amounts of data ?

    Hello,

    We are looking for a solution to backup and restore large amounts of data. However all the solutions we have tested seem to use single thread for compression and are just too slow.

    We're looking for a solution capable of using at least 8 cores for compression and capable of using fast connections (10Gbps) with fast disks on both sides (10 disk raid 10).

    We used to use R1Soft but abandonned it, a few years ago, in favor of bacula4hosts beacuse of serveur crashes, however while we would like to use gzip compression, it doesn't output more than 100Mbps on our 10Gbps network between the two servers.

    I'm sure we are not the only ones with large amounts of data to backup.

    I'm now thinking of something like rsync and installing ZFS on the backup server to have multiple resore points.

    Any suggestions about fast and reliable backup systems for large amounts of data ?

    Is Idera R1Soft single threaded ? Last time I asked their support they didn't know, that was when I was about to go back to their product but their staff not knowing their product specifications worried be a bit !

    Thanks !

  2. #2
    Quote Originally Posted by wonker View Post
    however while we would like to use gzip compression, it doesn't output more than 100Mbps on our 10Gbps network between the two servers.
    Hello wonker,

    you are correct. in our implementation of bacula, our only current compression option is gzip. gzip is fantastic in that it offers a very high level of compression. however, gzip is limited to a single thread, and since compression is done on the server side before data is transferred, it can sometimes become the limiting factor in backup speed.

    we are looking to integrate additional compression options to accommodate users like yourself, however, those simply arent available on our platform yet.

    a couple of comments/recommendations based on your requirements and our current capabilities/limitations

    1) we have implemented a virtual full module. so essentially, you are only taking an original full off of your server and over your network. after that, they will always be incrementals and virtual fulls will be created directly on your backup disk after that. So, if you want to stick with compression (to save on disk space and network traffic), then once you get past your original full, the single thread limitations of gzip may not be as large of a concern for you

    2) you could opt to remove the compression option. this will remove the single thread limitation with gzip and will push your backups out much faster. likely your disk IO on either end would become the natural limitation (and I assume this would be MUCH higher then the single thread limitation of gzip). of course this option will utilize more disk space on your backup device and of course more network traffic - so, it may not be ideal for you


    Is Idera R1Soft single threaded ? Last time I asked their support they didn't know, that was when I was about to go back to their product but their staff not knowing their product specifications worried be a bit !
    last time we used Idera, compression was done on the CDP server, after the data transfer (ie not on the server you are backing up) and it was not limited to a single thread. The benefit of this is that in theory you could compress faster, and since the compression is done on the CDP side, you can in theory pull data off of your server you are backing up faster. the problem with this approach is that you are pushing uncompressed data over the network (so higher network usage for similar amounts of raw data being transferred). the other issue is that you create a bottleneck on the CDP server as many servers are being compressed at the same time on the CDP server. so, although not a single thread bottleneck on the source server, you have to be really careful to stagger your jobs on each CDP server, as otherwise you create a bottleneck on the destination end

    Thanks for your input and I hope this information above is useful in helping you find a solution for your use case

  3. #3
    Join Date
    Apr 2005
    Location
    Cochin
    Posts
    2,446
    softaculous was preparing something in their backyard and was supposed to released . You should email them to try a beta ..
    Sam
    Supportlobby.com - Expertise in Windows/Linux server support, IaaS and PaaS Management
    Spiralbean.com - Custom Software Development
    Email: exec @ activelobby.net | Skype: unni_active

  4. #4
    Join Date
    Dec 2011
    Location
    Netherlands
    Posts
    831
    When using R1Soft (idera), the compression is performed at the hard Disk sector level on the Host during a Backup by the CDP Agent, this means it's sent compressed from the client to the CDP Server, minimizing the amount of data needed to be transferred.

  5. #5
    Join Date
    Oct 2013
    Location
    Everywhere
    Posts
    696
    Quote Originally Posted by activelobby4u View Post
    softaculous was preparing something in their backyard and was supposed to released . You should email them to try a beta ..
    Any information on this? I'd love to give it a try!

  6. #6
    Join Date
    Apr 2005
    Location
    Cochin
    Posts
    2,446
    Just try their email , its not listed on their website yet
    Sam
    Supportlobby.com - Expertise in Windows/Linux server support, IaaS and PaaS Management
    Spiralbean.com - Custom Software Development
    Email: exec @ activelobby.net | Skype: unni_active

  7. #7
    Join Date
    Aug 2005
    Posts
    421
    Thanks Andrew. Sticking with Bacula but without compression for the moment. We were not worried about network and cpu on backup server is always idle and actually has dual cpus. But if idera's compression is also single threaded, I would likely hit the same issue. For softaculus, we didn't use their product because their support was too slow few years ago ( almost non existant). I presume it's got better but it did put me off as it seemed that support wasn't their priority.

  8. #8
    Join Date
    Dec 2011
    Location
    Netherlands
    Posts
    831
    Quote Originally Posted by wonker View Post
    Thanks Andrew. Sticking with Bacula but without compression for the moment. We were not worried about network and cpu on backup server is always idle and actually has dual cpus. But if idera's compression is also single threaded, I would likely hit the same issue. For softaculus, we didn't use their product because their support was too slow few years ago ( almost non existant). I presume it's got better but it did put me off as it seemed that support wasn't their priority.
    idera is using QuickLZ by default, giving you compression, but since it's very fast, it should not really be a problem if it's single threaded, sure you would get more on multiple cores, but quickLZ is way faster than zlib in general.

    You can probably utilize a lot more bandwidth with no compression, but if you also need to send the double amount of data, I don't really know if that's beneficial, and will require more storage on the backup server itself.

  9. #9
    Quote Originally Posted by AcclaimedHost Alan View Post
    Any information on this? I'd love to give it a try!
    They are posting access to the beta on their forum: http://www.softaculous.com/board/ind...0.9.3_Launched

    Or jump straight to their wiki: http://www.nucdp.com/wiki/Category:Install

  10. #10
    Idera(formerly R1Soft) is a very stable and mature product now. You just need to get your setup right and it is probably the best CDP backup solution available on the market for hosters.
    OSHS Ltd
    OSHS Services - DNS Clusters | R1Soft Licenses | Remote Backup Storage | R1Soft CDP Storage | Cheap Dedicated Servers
    EconDC.com - Enterprise UK Server Colo & Rack Space at Lowest Prices

Similar Threads

  1. Paid backup solution for large MySQL database
    By sh4ka in forum Hosting Security and Technology
    Replies: 15
    Last Post: 10-15-2011, 07:59 AM
  2. Moving large amounts of data from one datacenter to another...
    By anonymous718 in forum Dedicated Server
    Replies: 20
    Last Post: 07-26-2008, 10:28 AM
  3. Transferring large amounts of data from server to server
    By Edman in forum Hosting Security and Technology
    Replies: 9
    Last Post: 01-12-2007, 08:52 AM
  4. Idea for copying large amounts of data quickly
    By Russ Foster in forum Hosting Security and Technology
    Replies: 3
    Last Post: 11-28-2004, 04:22 AM
  5. Uploading large amounts of data to server
    By alexkay in forum Web Hosting
    Replies: 5
    Last Post: 09-01-2004, 09:31 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •