Results 1 to 22 of 22
-
10-11-2013, 05:17 PM #1Junior Guru
- Join Date
- Feb 2007
- Posts
- 205
What To Do When Data Center has Issues?
You can have RAID, redundant PSUs, a stash of RAM on standby ... but what do you do when the data center has network problems?
Tonight my site goes offline for 2hrs (and counting ...) A network fault is identified and a fix is in place. There has been small outages in the past but nothing as long as this.
So what can be done as a fallback? I don't mean for you high rollers with replicated servers all over the place. What about us small time sites?
Another server in a different DC is one solution. But how do you switch over DNS settings so that visitors connect with the back up server instantly?
-
10-11-2013, 05:44 PM #2Web Hosting Master
- Join Date
- Nov 2005
- Posts
- 3,944
The easiest solution is to run an external DNS server with a low TTL and update the records upon failure. However, this will not work 100% because current/recent visitors may have DNS information cached.
-
10-11-2013, 05:57 PM #3WHT Addict
- Join Date
- Sep 2013
- Posts
- 105
You can try Distributed domain name server service and CDN-s
█ CLOUDYHOST ™ | UNIQUE: ⚡ Super Fast NGINX Hosting | WordPress Manager | Auto-Updater | Headless Mode | Elementor Pro & DIVI Pro Included
-
10-11-2013, 08:21 PM #4Junior Guru
- Join Date
- Feb 2007
- Posts
- 205
OK, will take a look at those.
---
Site was down for nearly 4 hours in total. In 14 years of running a site (dedicated and co-location) I have never experienced downtime like this. It's unacceptable.
-
10-11-2013, 10:16 PM #5Disabled
- Join Date
- Mar 2007
- Posts
- 365
The best solution is first - to outsource the DNS and second - to have a 2nd server on a different DC for failover.
Some DNS providers offer TTL as low as 1 or 5 minutes, so propagating the "new" records will be quite fast.
The 2nd solution is as @webhostal suggested - using a CDN to serve static content of your web sites (but won't work with dynamic content) - something similar to CF's "always on".
-
10-11-2013, 11:43 PM #6Web Hosting Master
- Join Date
- Feb 2011
- Posts
- 607
Use datacenters with redundant routers and redundant networks and have redundant paths from your network to those datacenter routers via redundant NICs and switches. Or run your own network, but that would cost much more.
-
10-12-2013, 04:42 AM #7Junior Guru
- Join Date
- Feb 2007
- Posts
- 205
I thought the datacentre I had was reliable; did have all that resilience. It seems that last night this happened:
"The underlying root cause of tonight's issues has been identified as a fibre break. This caused excessive flapping on the E1 transit ring which resulted in several of the ring switches rebooting. Unfortunately one of the switches did not recover - this then led to severe network instability which proved difficult to pinpoint."
---
Why isn't TTL set to 1 minute for all sites already? Excessive lookups?
-
10-12-2013, 04:47 AM #8Web Hosting Master
- Join Date
- Feb 2011
- Posts
- 607
Just because there is some redundancy does not mean that it is good enough. Single fibre break should not lead to any issues in a properly designed system. Neither it should lead to any issues in a properly tested system.
-
10-12-2013, 07:11 PM #9Web Hosting Guru
- Join Date
- Apr 2009
- Location
- Toronto
- Posts
- 251
I find a lot of customers will have a cloud instance ready for fail-over. Low TTL for manual switch over or intelligent DNS load balancing (detects when server 1 goes dead, directs to server2/cloud). This can be done fairly cheap depending on what the website is doing.
Its always good to have a fail over plan in place. We've always found its not if your data center will go down, but when and being prepared for that. It might be perfect for 1 year, 5 years or more, but sooner or later something will occur even in the best setups.
-
10-13-2013, 04:28 AM #10Cable Director
- Join Date
- Aug 2007
- Location
- Datacenter
- Posts
- 4,414
You can also have a talk to the datacenter to ask why on earth they don't have a redundant setup.
This doesn't cost you a thing
Unless they are victim of cyber crime which is off course another story.» www.InstantDedicated.com - Online in no time
» Dedicated Servers in [EU] Netherlands + Belgium with DAILY support, also on weekends
» 3.2 Tbit/s Network AS49453 with only 100 Gbit/s uplink backbone
» 1G/10G/40G/100 Gbit ports available | 99,99% Network Uptime goal
-
10-13-2013, 11:01 AM #11Newbie
- Join Date
- Oct 2013
- Posts
- 5
-
10-13-2013, 11:37 AM #12Web Hosting Master
- Join Date
- Oct 2009
- Posts
- 590
You cannot rely on anything any provider says about their data center unless you see it for yourself and you know what questions to ask.
No matter how redundant they say they are very few truly are. I am saying this from experience with a lot of them. The economics of being truly redundant make it almost impossible to compete so they almost never are. Yes even some of the supposedly best of the best who charge top dollar. Also, some truly redundant data centers have still gone down for other reasons. Like when Hurricane Sandy flooded Manhattan.
As someone else said, always assume that it's not a matter of if but when a data center goes down. You should do everything you can to make things data center redundant. DNS load-balancing/failover is probably the easiest and most economical.Last edited by UnfinishedSentenc; 10-13-2013 at 11:42 AM.
-
10-13-2013, 12:08 PM #13Junior Guru
- Join Date
- Feb 2007
- Posts
- 205
econdc. Do you think it is to do with the recent take-over of PH by P?
-
10-14-2013, 03:05 PM #14Web Hosting Guru
- Join Date
- Oct 2012
- Location
- Arlington, VA
- Posts
- 332
What about a failover cross-connect to another provider in the same facility?
InfoRelay Online Systems, Inc.
Colocation | Cloud | Data Centers | Bandwidth | Managed Services
New York / Northern Virginia / Chicago / Dallas / Miami / Los Angeles / San Jose
www.InfoRelay.com/Services/Data-Centers
-
10-14-2013, 04:01 PM #15Web Hosting Guru
- Join Date
- Feb 2005
- Location
- localhost
- Posts
- 5,473
A provider with multiple locations with a failover option might be a good idea. cloud service??
Respectfully,
Mr. Terrence
-
10-14-2013, 06:16 PM #16Junior Guru Wannabe
- Join Date
- Jan 2006
- Posts
- 72
Many large ISP's have a minimum threshold for TTL's (measured in hours), so setting yours to 1 minute would probably be futile depending upon your customer base.
Another possible solution would be load balancing, though at a higer price depending on the amount of bandwidth your using. rackspace has load balancing services (you don't have to be hosting with them), but if your pushing many TB's of data that's probably going to be more than you're looking to spend. Some smaller hosting companies might provide this service if you are hosting in 2 of their sites. If your purchasing bandwidth from them your not paying for it twice as you would be with rackspace. The company you're with now may be able to work with you on such a solution, or possibly recommend somebody who has done this for their customers already.Last edited by MetroAce; 10-14-2013 at 06:21 PM.
-
10-21-2013, 05:18 PM #17Web Hosting Evangelist
- Join Date
- Nov 2012
- Location
- Los Angeles, CA
- Posts
- 499
Look for another data center that has the security and reliability to handle all your needs.
¦¦ COLOCATIONAMERICA.COM™ It’s All About Connections...™
¦¦ Colocation, Dedicated Servers, Premium Bandwidth, 24/7 Support!
¦¦ Los Angeles , New York , New Jersey , San Francisco , Chicago
¦¦ Email: Sales@ColocationAmerica.com for a quote or call 888-505-COLO!
-
10-22-2013, 08:54 AM #18Web Hosting Guru
- Join Date
- Oct 2012
- Location
- Arlington, VA
- Posts
- 332
If it were that simple, we'd all use that one "data center that has the security and reliability". The bottom line is that all data centers have issues now and then - the only variable is how the service team handles them. Do they ignore tickets and fight back on WHT, like some providers we've seen on here? Or do they suck it up and accept that it was their own fault and deal with it accordingly?
InfoRelay Online Systems, Inc.
Colocation | Cloud | Data Centers | Bandwidth | Managed Services
New York / Northern Virginia / Chicago / Dallas / Miami / Los Angeles / San Jose
www.InfoRelay.com/Services/Data-Centers
-
10-22-2013, 10:16 AM #19Junior Guru
- Join Date
- Feb 2007
- Posts
- 205
I am putting a new server in a different DC with the old server staying where it is. I now just need to work out a good strategy to use if the new DC is unavailable for some period of time.
An hour or two of downtime would just be about acceptable (as long as it is a rare occassion). Anything longer than that and I would like the site users to be pointed to the old backup server.
Updating the DNS record is fine but there is the problem of waiting for it to ripple through to users.
I am not bothered about new visitors or search engine bots not picking up that the backup server is in use but I do need to ensure that my regular customers can access their data. Thus I would need them to access the back server ASAP.
One way to do that I guess is for them to modify their hosts file. If they add an entry for
example.com 222.222.222.222 (the backup server)
then issue
ipconfig /flushdns
and restart their browser they should instantly connect to the backup server, correct?
-
10-22-2013, 01:36 PM #20Web Hosting Guru
- Join Date
- Jan 2010
- Posts
- 308
/flushdns doesn't control the cache of their upstream DNS resolver. That just flushes the host's DNS resolver cache. Setting a low'ish TTL usually works well enough. From what we've seen after changing DNS records, the amount of people still hitting the old IP immediately after TTL expiring is very low. Usually in the 1-3% range.
Keeping a low TTL works a lot better than the fear mongerers would suggest. Are there some ISP's that pin cache TTL's to their own liking? Sure. Do they have a ton of eyeballs? No.
-
10-23-2013, 09:49 AM #21Web Hosting Master
- Join Date
- Aug 2010
- Posts
- 1,976
what datacenter is this?
Superb Houston/Los Angeles Colocation: LAYERHOST.COM https://www.layerhost.com/colocation
*not affiliated, just recommendation*
-
10-23-2013, 01:59 PM #22Junior Guru Wannabe
- Join Date
- Jan 2006
- Posts
- 35
Amazon's Route 53 DNS service also offers DNS failover with low TTL. I haven't used it myself, but reading the documentation this may help you with your setup Larry and is quite low cost (it seems).
Similar Threads
-
Any one have issues with Softlayer Netherland Data center ?
By public_html in forum Dedicated ServerReplies: 8Last Post: 05-16-2012, 02:56 PM -
Europe - Romania Data Center - Dedicated Servers at Ch-center.com
By ClaudiuPopescu in forum Dedicated Hosting OffersReplies: 2Last Post: 03-04-2011, 05:49 AM -
Data Center Issues?
By crockett5 in forum Web HostingReplies: 3Last Post: 01-27-2005, 11:57 PM -
Pegasus Data Center: Network Issues
By JenniH in forum Providers and Network Outages and UpdatesReplies: 2Last Post: 01-18-2005, 06:41 PM -
Does the data center and the ownership of the data center matter.
By coneil in forum Dedicated ServerReplies: 16Last Post: 02-08-2003, 11:37 PM