Results 1 to 23 of 23
-
01-04-2014, 11:44 PM #1Newbie
- Join Date
- Jan 2014
- Posts
- 16
Our company and Cloud VPS dead in the water
We are currently dealing with a very popular hosting company (name will be revealed later) that has not been able to restore our cloud VPS for over 24 hours now. What started as SANS maintenance turned into a total disaster and as we are typing this we are still dead in the water.
We are deeply concerned about this but want to make sure we are not expecting too much. Is this a normal occurrence? In all my years of hosting I have never seen such a long down time.
Why wouldn't this data center have a reliable fail safe in place? What I really need to understand is how to choose a reliable data center that can handle things like this.
We have not decided if we are leaving this hosting company yet but as mentioned we really need to learn which companies have better reparation and restoration processes.
Any advice is greatly appreciated.
Daniel
-
01-04-2014, 11:47 PM #2The Linux Specialist
- Join Date
- Mar 2003
- Location
- /root
- Posts
- 23,990
Hardware do fail sometimes no matter what the quality of it.
This is why, it is wise that you always have a remote backup. This is required when you are doing hosting a site especially if it is critical.
Specially 4 U
Reseller Hosting: Boost Your Websites | Fully Managed KVM VPS: 3.20 - 5.00 Ghz, Pure Dedicated Power
JoneSolutions.Com is on the net 24/7 providing stable and reliable web hosting solutions, server management and services since 2001
Debian|Ubuntu|cPanel|DirectAdmin|Enhance|Webuzo|Acronis|Estela|BitNinja|Nginx
-
01-04-2014, 11:48 PM #3Web Hosting Evangelist
- Join Date
- Jan 2008
- Posts
- 519
Does your company have a backup plan? Has it been enacted?
Issues come up for all hosting companies, no matter their size or reputation.
Of course, you can decide to continue with that company based on your findings post the issue, but... were you prepared for your host to have an outage?Mike
-
01-05-2014, 12:00 AM #4Newbie
- Join Date
- Jan 2014
- Posts
- 16
I have been running this hosting company for over 5 years and never really had to deal with our own backups, I guess we always were with a company that managed our server and had backups in place.
I'm not saying this company doesnt have backups for this incident - but to take this long to restore is not right.
That's more my angle on this - who are the solid companies out there that have redundancy at the node level (or whatever) so no matter the hardware issue they can swap and get the server back online in minutes.
-
01-05-2014, 12:26 AM #5Managed VPS Experts
- Join Date
- Nov 2007
- Location
- New Jersey, USA
- Posts
- 4,740
We really do not know the full story here, what went wrong, what they are doing to restore access etc. Depending on what they are restoring and how much data there is it can sometimes take days.
Are they saying that they will get everything back online? It's most likely just a waiting game now.
Good Luck.
- Daniel
-
01-05-2014, 01:36 AM #6Newbie
- Join Date
- Jan 2014
- Posts
- 16
To your point, they claim: "As you can tell by the length of the maintenance, what initially had been emergency maintenance to replace a failing member disk has expanded into an outage that we are working diligently to address."
I just checked and our sites seem to be back online. I still really need to find out which companies have better preventive measures in place. I talked to another company and they claim to have 30 minute or less complete restore on hard drive failures.
We simply cannot afford to have this happen again.
-
01-05-2014, 01:40 AM #7Norwegian polar bear hunter
- Join Date
- Feb 2005
- Location
- Norway
- Posts
- 1,651
Wow, 5 years without number 1.backup rule: Always keep your own backups. Do not trust your provider fully out with this. Always have a updated backup on hand. (of course not stored on any server from your current provider).
Then you could had been up and running within minutes after a thing like this, on a new server or on a new provider.My Top 20 benchmark list (and review site)
Powered by: Kimsufi, backed up by: Hetzner, DigitalOcean and Vultr.com
Also using SolaDrive.com (56+ months), KnownHost.com (56+ months)
-
01-05-2014, 01:43 AM #8WHT Addict
- Join Date
- Nov 2003
- Location
- Texas, United States
- Posts
- 115
Based on what has been stated, this issue does not seem like a simple hard drive replacement". While you may want to, don't rate your current hosting company based on this single SAN outage event. SAN issues can affect not just a single hard drive but can affect the entire SAN Frame resulting in an outage to all SAN disks within that specific SAN Frame. That could be many Terabytes of data and if it was all somehow corrupted (in my ~20yrs of experience I have seen it happen once with an earlier EMC SAN Array model) the typical restore time for that much data will be hours/days, not 30 minutes or less (depending if they are using some type of BCV/Timefinder/ShadowImage copy technology, which I would say most hosting providers are not as that would more than double the cost of storage).
► Tim Benoit
-
01-05-2014, 01:46 AM #9LiteSpeed Server Expert
- Join Date
- Nov 2011
- Location
- Datacenters
- Posts
- 1,401
Data and Life has no guarantee mate .Always take backup before packup !
█ Fastest . Reliable . Affordable Web Hosting by MNCHOST.COM
█ Shared | Business | Reseller : 1.Litespeed 2.Cpanel 3.Pure SSDs 4.US Support 5.Hourly Backups
█ Fully Managed Dedicated Servers | Premium Bandwidth | New SSD-US Servers
-
01-05-2014, 04:14 AM #10Web Hosting Evangelist
- Join Date
- Mar 2003
- Posts
- 470
You are not in the worst situation. There are other incidents posted here in WHT where disk restoration took several weeks. As others have already emphasized, you should always have your own remote backups, at all times, even if your provider is considered "reliable". Also, hardware issues isn't the only thing you should be concerned about, I remember a popular company back then, management gone AWOL, customers dead in the water.
-
01-05-2014, 04:25 AM #11
-
01-05-2014, 04:47 AM #12Temporarily Suspended
- Join Date
- Dec 2013
- Location
- Internet ;)
- Posts
- 36
-
01-05-2014, 05:04 AM #13Poooooonnyyy :*
- Join Date
- Jan 2003
- Location
- Canada
- Posts
- 5,073
Sounds like RAID5
If they lost a 2nd disk then they're going to have some issues. If the drives aren't dead they can actually use DDRESCUE from onboard intel SATA sockets to mirror the drive over. It should get automatically accepted into the RAID and they'll be out whatever bad sectors there was instead of a potential total loss
FranciscoBuyVM - OpenVZ & KVM Based VPS Servers - Chat with us
- All popular VPN methods supported
- Affordable offloaded MySQL & DDoS protection
- 5GB backup space, unmetered private LAN bandwidth & native IPv6 included. All with a strong serving of pony
-
01-05-2014, 10:57 AM #14Newbie
- Join Date
- Jan 2014
- Posts
- 16
thanks for the great feedback guys. Regarding taking our own backups, do you mean simply running a manual backup from WHM and save it elsewhere? That file could be pretty large.
or is there a more automated approach for this?
-
01-05-2014, 11:01 AM #15WHT Addict
- Join Date
- Nov 2003
- Location
- Texas, United States
- Posts
- 115
WHM can perform backups automatically and even transfer those backups to another server. While that will get your web site data it won't backup many OS-related items.
I would recommend using 3rd party software to perform server backups off-host. Idera ServerBackup (formerly R1Soft) is what I would recommend to perform those backups.► Tim Benoit
-
01-05-2014, 11:08 AM #16Managed VPS Experts
- Join Date
- Nov 2007
- Location
- New Jersey, USA
- Posts
- 4,740
-
01-05-2014, 01:19 PM #17Web Hosting Evangelist
- Join Date
- Jan 2008
- Posts
- 519
-
01-05-2014, 01:21 PM #18Managed VPS Experts
- Join Date
- Nov 2007
- Location
- New Jersey, USA
- Posts
- 4,740
-
01-05-2014, 01:44 PM #19Newbie
- Join Date
- Jan 2014
- Posts
- 16
-
01-05-2014, 03:10 PM #20WHT Addict
- Join Date
- Nov 2003
- Location
- Texas, United States
- Posts
- 115
-
01-05-2014, 07:53 PM #21Web Hosting Master
- Join Date
- Apr 2002
- Location
- England, UK
- Posts
- 734
If the data is valuable to you and I say this from experience don't rely on the host always keep a daily copy offsite.
-
01-05-2014, 07:55 PM #22
DON'T just rely on WHM's backups either.
Create your own system.
Backup MySQL frequently. As in 3-4 times a day!!!
Send it all offsite.
Get yourself a server, not a VPS. Somewhere like hetzner, where storage is dirt cheap, and you have unmetered data
learn about rsync, it's your friend... Or take a look at r1soft or bacula
-
01-06-2014, 05:47 AM #23New Member
- Join Date
- Jun 2013
- Posts
- 4
Like imitech I can also based on experience recommend NOT to rely on the host for backup (I have done that and I have lost customer data)
Similar Threads
-
Dedimove - Dead in the Water?
By TBoneS in forum Dedicated ServerReplies: 5Last Post: 06-14-2008, 07:57 AM -
Need help: HD died, OS reload partially broken, dead in the water
By theNonsuch in forum Dedicated ServerReplies: 1Last Post: 05-29-2008, 10:24 PM -
My Email to the Water Company
By Webdude in forum Web Hosting LoungeReplies: 14Last Post: 08-31-2007, 04:41 PM