Results 1 to 23 of 23
  1. #1

    Our company and Cloud VPS dead in the water

    We are currently dealing with a very popular hosting company (name will be revealed later) that has not been able to restore our cloud VPS for over 24 hours now. What started as SANS maintenance turned into a total disaster and as we are typing this we are still dead in the water.

    We are deeply concerned about this but want to make sure we are not expecting too much. Is this a normal occurrence? In all my years of hosting I have never seen such a long down time.

    Why wouldn't this data center have a reliable fail safe in place? What I really need to understand is how to choose a reliable data center that can handle things like this.

    We have not decided if we are leaving this hosting company yet but as mentioned we really need to learn which companies have better reparation and restoration processes.

    Any advice is greatly appreciated.

    Daniel

  2. #2
    Join Date
    Mar 2003
    Location
    /root
    Posts
    23,990
    Hardware do fail sometimes no matter what the quality of it.

    This is why, it is wise that you always have a remote backup. This is required when you are doing hosting a site especially if it is critical.

    Specially 4 U
    Reseller Hosting: Boost Your Websites | Fully Managed KVM VPS: 3.20 - 5.00 Ghz, Pure Dedicated Power
    JoneSolutions.Com is on the net 24/7 providing stable and reliable web hosting solutions, server management and services since 2001
    Debian|Ubuntu|cPanel|DirectAdmin|Enhance|Webuzo|Acronis|Estela|BitNinja|Nginx

  3. #3
    Join Date
    Jan 2008
    Posts
    519
    Does your company have a backup plan? Has it been enacted?

    Issues come up for all hosting companies, no matter their size or reputation.

    Of course, you can decide to continue with that company based on your findings post the issue, but... were you prepared for your host to have an outage?
    Mike

  4. #4
    Quote Originally Posted by redmeds View Post
    Does your company have a backup plan? Has it been enacted?

    Issues come up for all hosting companies, no matter their size or reputation.

    Of course, you can decide to continue with that company based on your findings post the issue, but... were you prepared for your host to have an outage?
    I have been running this hosting company for over 5 years and never really had to deal with our own backups, I guess we always were with a company that managed our server and had backups in place.

    I'm not saying this company doesnt have backups for this incident - but to take this long to restore is not right.

    That's more my angle on this - who are the solid companies out there that have redundancy at the node level (or whatever) so no matter the hardware issue they can swap and get the server back online in minutes.

  5. #5
    Join Date
    Nov 2007
    Location
    New Jersey, USA
    Posts
    4,740
    We really do not know the full story here, what went wrong, what they are doing to restore access etc. Depending on what they are restoring and how much data there is it can sometimes take days.

    Are they saying that they will get everything back online? It's most likely just a waiting game now.

    Good Luck.

    - Daniel

  6. #6
    Quote Originally Posted by TmzHosting View Post
    We really do not know the full story here, what went wrong, what they are doing to restore access etc. Depending on what they are restoring and how much data there is it can sometimes take days.

    Are they saying that they will get everything back online? It's most likely just a waiting game now.

    Good Luck.

    - Daniel
    To your point, they claim: "As you can tell by the length of the maintenance, what initially had been emergency maintenance to replace a failing member disk has expanded into an outage that we are working diligently to address."

    I just checked and our sites seem to be back online. I still really need to find out which companies have better preventive measures in place. I talked to another company and they claim to have 30 minute or less complete restore on hard drive failures.

    We simply cannot afford to have this happen again.

  7. #7
    Join Date
    Feb 2005
    Location
    Norway
    Posts
    1,651
    Quote Originally Posted by steadysystems View Post
    I have been running this hosting company for over 5 years and never really had to deal with our own backups, I guess we always were with a company that managed our server and had backups in place.
    Wow, 5 years without number 1.backup rule: Always keep your own backups. Do not trust your provider fully out with this. Always have a updated backup on hand. (of course not stored on any server from your current provider).

    Then you could had been up and running within minutes after a thing like this, on a new server or on a new provider.
    My Top 20 benchmark list (and review site)
    Powered by: Kimsufi, backed up by: Hetzner, DigitalOcean and Vultr.com
    Also using
    SolaDrive.com (56+ months), KnownHost.com (56+ months)

  8. #8
    Join Date
    Nov 2003
    Location
    Texas, United States
    Posts
    115
    Based on what has been stated, this issue does not seem like a simple hard drive replacement". While you may want to, don't rate your current hosting company based on this single SAN outage event. SAN issues can affect not just a single hard drive but can affect the entire SAN Frame resulting in an outage to all SAN disks within that specific SAN Frame. That could be many Terabytes of data and if it was all somehow corrupted (in my ~20yrs of experience I have seen it happen once with an earlier EMC SAN Array model) the typical restore time for that much data will be hours/days, not 30 minutes or less (depending if they are using some type of BCV/Timefinder/ShadowImage copy technology, which I would say most hosting providers are not as that would more than double the cost of storage).
    Tim Benoit

  9. #9
    Join Date
    Nov 2011
    Location
    Datacenters
    Posts
    1,401
    Data and Life has no guarantee mate .Always take backup before packup !
    Fastest . Reliable . Affordable Web Hosting by MNCHOST.COM

    █ Shared | Business | Reseller : 1.Litespeed 2.Cpanel 3.Pure SSDs 4.US Support 5.Hourly Backups
    Fully Managed Dedicated Servers | Premium Bandwidth | New SSD-US Servers

  10. #10
    Join Date
    Mar 2003
    Posts
    470
    You are not in the worst situation. There are other incidents posted here in WHT where disk restoration took several weeks. As others have already emphasized, you should always have your own remote backups, at all times, even if your provider is considered "reliable". Also, hardware issues isn't the only thing you should be concerned about, I remember a popular company back then, management gone AWOL, customers dead in the water.

  11. #11
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    14,135
    Quote Originally Posted by myhken View Post
    Wow, 5 years without number 1.backup rule: Always keep your own backups.
    RIGHT? You learn this day 1.
    It's amazing that people still don't grasp this very, very simple concept... Do your own backups. Check them

  12. #12
    Join Date
    Dec 2013
    Location
    Internet ;)
    Posts
    36

    Thumbs up Hi

    Quote Originally Posted by MNCHOST - Chris View Post
    Data and Life has no guarantee mate .Always take backup before packup !
    Hello,
    totally agree with you my friend

  13. #13
    Join Date
    Jan 2003
    Location
    Canada
    Posts
    5,073
    Sounds like RAID5

    If they lost a 2nd disk then they're going to have some issues. If the drives aren't dead they can actually use DDRESCUE from onboard intel SATA sockets to mirror the drive over. It should get automatically accepted into the RAID and they'll be out whatever bad sectors there was instead of a potential total loss

    Francisco
    BuyVM - OpenVZ & KVM Based VPS Servers - Chat with us
    - All popular VPN methods supported
    - Affordable offloaded MySQL & DDoS protection
    - 5GB backup space, unmetered private LAN bandwidth & native IPv6 included. All with a strong serving of pony

  14. #14
    thanks for the great feedback guys. Regarding taking our own backups, do you mean simply running a manual backup from WHM and save it elsewhere? That file could be pretty large.

    or is there a more automated approach for this?

  15. #15
    Join Date
    Nov 2003
    Location
    Texas, United States
    Posts
    115
    WHM can perform backups automatically and even transfer those backups to another server. While that will get your web site data it won't backup many OS-related items.

    I would recommend using 3rd party software to perform server backups off-host. Idera ServerBackup (formerly R1Soft) is what I would recommend to perform those backups.
    Tim Benoit

  16. #16
    Join Date
    Nov 2007
    Location
    New Jersey, USA
    Posts
    4,740
    Quote Originally Posted by steadysystems View Post
    thanks for the great feedback guys. Regarding taking our own backups, do you mean simply running a manual backup from WHM and save it elsewhere? That file could be pretty large.

    or is there a more automated approach for this?
    WHM has an automatic backup feature. Buy a backup VPS in a different location and automatically backup your data every day or every week. If something like this ever happens again you can provision a new VPS quickly and restore the data.

    - Daniel

  17. #17
    Join Date
    Jan 2008
    Posts
    519
    Quote Originally Posted by TmzHosting View Post
    WHM has an automatic backup feature. Buy a backup VPS in a different location and automatically backup your data every day or every week. If something like this ever happens again you can provision a new VPS quickly and restore the data.

    - Daniel


    especially if your company holds value!
    Mike

  18. #18
    Join Date
    Nov 2007
    Location
    New Jersey, USA
    Posts
    4,740
    Quote Originally Posted by redmeds View Post


    especially if your company holds value!
    And it definitely seems like his did.

    - Daniel

  19. #19
    Quote Originally Posted by tbenoit View Post
    WHM can perform backups automatically and even transfer those backups to another server. While that will get your web site data it won't backup many OS-related items.

    I would recommend using 3rd party software to perform server backups off-host. Idera ServerBackup (formerly R1Soft) is what I would recommend to perform those backups.
    Is that online backup?

  20. #20
    Join Date
    Nov 2003
    Location
    Texas, United States
    Posts
    115
    Quote Originally Posted by steadysystems View Post
    Is that online backup?
    Using Idera ServerBackup, yes it is. It will also backup your MySQL databases online (no outage required).
    Tim Benoit

  21. #21
    Join Date
    Apr 2002
    Location
    England, UK
    Posts
    734
    If the data is valuable to you and I say this from experience don't rely on the host always keep a daily copy offsite.

  22. #22
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    14,135
    DON'T just rely on WHM's backups either.
    Create your own system.
    Backup MySQL frequently. As in 3-4 times a day!!!
    Send it all offsite.

    Get yourself a server, not a VPS. Somewhere like hetzner, where storage is dirt cheap, and you have unmetered data
    learn about rsync, it's your friend... Or take a look at r1soft or bacula

  23. #23
    Like imitech I can also based on experience recommend NOT to rely on the host for backup (I have done that and I have lost customer data)

Similar Threads

  1. Dedimove - Dead in the Water?
    By TBoneS in forum Dedicated Server
    Replies: 5
    Last Post: 06-14-2008, 07:57 AM
  2. Replies: 1
    Last Post: 05-29-2008, 10:24 PM
  3. My Email to the Water Company
    By Webdude in forum Web Hosting Lounge
    Replies: 14
    Last Post: 08-31-2007, 04:41 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •