Results 1 to 17 of 17
  1. #1
    Join Date
    Oct 2006
    Posts
    39

    Exclamation Liquidweb Michigan datacenter down?

    Our site has been down all morning. We talked to Liquidweb and they say it's something to do with the ISPs and that the site is up in parts of the country but down in others. We are not in an affected area ourselves, but we cannot access the site.

    Liquidweb posting: http://www.liquidweb.com/support/

    We asked if they would switch our server to one of their other backbones then to fix the problem to which they replied they would not be able to. Time is of the essence as this is an ecommerce site. Should they not be rerouting the traffic instead of just saying hey it's not our problem?

  2. #2
    Join Date
    Mar 2011
    Location
    Lansing, MI (USA)
    Posts
    111
    It isn't an outage at our data center. Nor is it a single provider, which is making this difficult for us as we would generally just reroute our traffic.

    Our network team is still working to see if we can shift traffic over other providers, but many have been affected by this as you can see on the maps tracking these large ISP & Network providers:

    http://downdetector.com/status/comcast-xfinity/map/
    http://downdetector.com/status/att/map/
    http://downdetector.com/status/time-warner-cable/map/
    http://downdetector.com/status/verizon-wireless/map/

    We're currently working to get a timeframe and additional information as everyone else is.

  3. #3
    Join Date
    Jan 2008
    Location
    Europe
    Posts
    779
    It's been 4-5 hours at least now. We had to find out for ourselves, still haven't had a customer wide email updating everyone about the situation, so who knows if it's been longer than that.

    You say that it's affecting many other ISPs and providers, but so far it seems no other ISP has reported any issues, there's nothing on http://status.cogentco.com . Those links you provided are for the general public too, there's always issues reported on there regarding cable internet access etc.

    Liquidweb.com is also online, even though its being routed through Cogent.

  4. #4
    Join Date
    Jun 2014
    Posts
    57
    Having worked for ISPs previously, and in a data center, ISPs almost never admit to outages. I (sadly) worked as a contracted technician for comcast for several years... they once re-wired an entire city pole-to-pole without pre-warning the customers; they even sent us on outage calls all day even though they knew their main lines were being replaced. Someone could be in their house looking at a bucket truck taking down the hardline at the pole, call comcast and say "my internet is out" and comcast would say "no known outages at this time, we'll send a tech out."

    Those outage maps are startling to say the least. Clearly there is something huge going on in Detroit, which is out of Liquid Webs control.

    Looking on the status.cogentco.com link, last update was over 12 hours ago (Cogent Network Status Report Last Updated Tues Aug 12 01:23:45 2014 GMT)
    Last edited by quizknows; 08-12-2014 at 10:44 AM.

  5. #5
    Join Date
    Oct 2006
    Posts
    39
    For my part, I'm not saying it's Liquidweb's data centers. I'm just saying the should reroute their customer's traffic temporarily until the issue is fixed.

    Unless something catastrophic happened like all the backbone wires being cut out of both sides of their data center I would think they would be able to reroute the traffic for at least the dedicated servers who are paying a premium for their services. We are paying over $600 a month in hosting and have called twice, explained that we are a time sensitive site, yet we are told just to wait. Not a good customer experience regardless of if the issue is directly in their data center or not, they could do something to reroute traffic. That after all is what we are paying a premium for, a redundant fail safe data-center with multiple backbones coming into the data-center like they advertise.

  6. #6
    Join Date
    Mar 2011
    Location
    Lansing, MI (USA)
    Posts
    111
    We are rerouting as much of our traffic as possible. As mentioned, this is hindered by the prevalence of the multiple providers experiencing outages in our US Central zone.

  7. #7
    Join Date
    Oct 2006
    Posts
    39
    Thanks LiquidwebCale but this doesn't help us much as our site is down. We cannot process customers orders, receive customer messages, or take new customer orders. Our business is dead in the water. Is there anything you can do to get our site back up?

  8. #8
    Join Date
    Mar 2004
    Location
    Chicago IL
    Posts
    573
    We use a dedicated server in in the Liquid Web DC to house our main web site only. If we try to access it through our Comcast Business Package we can not reach the site. But switching over to Verizon Hot Spots we have no issues in connecting to the site.

    You can't blame Liquid Web they are doing the best they can given the circumstances.

  9. #9
    Join Date
    Aug 2010
    Location
    Sorting Office
    Posts
    9,530
    Quote Originally Posted by LiquidWebCale View Post
    We are rerouting as much of our traffic as possible.
    Cale, Allstream have a fibre cut between Chicago and Canada which might be at least partly responsible for this. It's also affecting Level3 and OVH.

    http://downdetector.com/status/level3/map/

    http://status.ovh.net/?do=details&id=7483

    http://weathermap.ovh.net/usa (Scroll right and down a bit on this last one).

  10. #10
    Join Date
    Jan 2008
    Location
    Europe
    Posts
    779
    Quote Originally Posted by quizknows View Post
    Those outage maps are startling to say the least. Clearly there is something huge going on in Detroit, which is out of Liquid Webs control.
    It's weird that no other provider seems to be as badly affected as LW (if at all). Looking on datacentermap.com I can't find any other provider in MI who is offline, and the only other provider who has downtime reported on WHT is related to DDoS.

    If this is really outside of LW's network then why is no other provider offline? They seem to have no idea what is going on.

    Over 6 hours now, still no information.

  11. #11
    Join Date
    Jun 2014
    Posts
    57
    Liquid Web is really the only big data center in MI that I'm aware of (at least web hosting wise).

    It's not just them: http://www.reddit.com/r/sysadmin/com...king_problems/

    As others have stated they can get to their sites at LW from one ISP but not another (namely comcast).

  12. #12
    Join Date
    Mar 2004
    Location
    Chicago IL
    Posts
    573
    Quote Originally Posted by sam0 View Post
    It's weird that no other provider seems to be as badly affected as LW (if at all). Looking on datacentermap.com I can't find any other provider in MI who is offline, and the only other provider who has downtime reported on WHT is related to DDoS.

    If this is really outside of LW's network then why is no other provider offline? They seem to have no idea what is going on.
    LiquidWeb is not off line, I can connect to them through Verizon and AT&T with no issues, just not Comcast.

  13. #13
    Join Date
    Jan 2008
    Location
    Europe
    Posts
    779
    Quote Originally Posted by quizknows View Post
    Liquid Web is really the only big data center in MI that I'm aware of (at least web hosting wise).

    It's not just them: http://www.reddit.com/r/sysadmin/com...king_problems/

    As others have stated they can get to their sites at LW from one ISP but not another (namely comcast).
    Thanks for that link. It's nice to see it's not just us. And there's quite a few colo providers in MI and Detroit.

    Quote Originally Posted by WebzPro View Post
    LiquidWeb is not off line, I can connect to them through Verizon and AT&T with no issues, just not Comcast.
    Liquidweb.com is up for me via Cogent, but WHT and our servers are down via Cogent . I'm having to use a proxy to post.
    Last edited by sam0; 08-12-2014 at 12:52 PM. Reason: clarification

  14. #14
    Join Date
    Nov 2005
    Posts
    3,944
    Quote Originally Posted by sam0 View Post
    but WHT and our servers are down via Cogent . I'm having to use a proxy to post.
    Well I'm in Michigan and not having any issues with WHT or Liquidweb. What is your URL? (You can PM me)

    It doesn't look like this is a widespread outage, only affecting a few ISPs. Liquidweb probably can't control the inbound routing as much as you think, it is dependent on your ISP to correctly route you to them.

  15. #15
    Join Date
    Jun 2014
    Posts
    57
    This could prove interesting...

    http://www.nux.ro/archive/2014/08/51..._everyone.html

    tl;dr, a lot of routers can't handle all the BGP routes as the number is approaching or may have exceeded 512k. I expect a lot of ISPs to be having problems because of this.
    Last edited by quizknows; 08-12-2014 at 02:34 PM.

  16. #16
    Join Date
    Jan 2008
    Location
    Europe
    Posts
    779
    Quote Originally Posted by devonblzx View Post
    Well I'm in Michigan and not having any issues with WHT or Liquidweb. What is your URL? (You can PM me)
    Thanks, but our servers are back online now. Looks like our uniques dropped by about 25-33% (usually pretty stable) but it's now back to normalish levels. It was about 9 hours in the end I think, WHT is still down for me though.

  17. #17
    Join Date
    Oct 2013
    Location
    Germany
    Posts
    84
    This is so funny

    I live in Germany.

    I can access WHT from my cable but not from my DSL line.

    I see no problem with other websites.

    Regards,
    Iosif

Similar Threads

  1. Michigan datacenter for a Southamerica and Spain oriented web any good?
    By daantaelon in forum Colocation, Data Centers, IP Space and Networks
    Replies: 10
    Last Post: 03-01-2010, 12:08 PM
  2. Replies: 0
    Last Post: 05-05-2009, 07:06 PM
  3. Site5 or Liquidweb? Same datacenter...
    By joriki in forum Reseller Hosting
    Replies: 7
    Last Post: 04-07-2005, 09:08 PM
  4. Replies: 1
    Last Post: 02-03-2005, 12:53 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •