Page 1 of 2 12 LastLast
Results 1 to 25 of 46
  1. #1
    Join Date
    Aug 2005
    Location
    patiala, punjab, india
    Posts
    264

    Thumbs down More than 5 hour downtime for Softlayer Cloud Computing

    Today, The Cloudlayer Instance we are having with Softlayer has faced a major downtime. At the time of writing this post, the service is down (5.30 hour already).

    The support staff hasn't provided much information about the reason for downtime.

    They say, the power failed in server room #5 for Dallas. The power was restored soon, but i don't see the CloudLayer Instance coming up even after 5 hours.

    Having account with Softlayer for more than 2.5 years, i have been very happy with their team and their network, but more than 5 hour downtime and still the support not offering any ETA, is making me think of shifting to some other provider.

    I am not much aware of Cloud Computing Instances and how much time they would need to come back online. But, i don't think it should take so much of time.

    I am still waiting for the service to be restored. I guess some other members on WHT may have also faced the trouble today.

    Please feel free to add comments.
    Get Best Prices for Domains - Secure payments via paypal using the Resellerclub interface. Sub Resellers invited. Hosting Companies i DON'T recommend anymore: MDDHosting, Hostgator, DownTownHost.

  2. #2
    Join Date
    Feb 2008
    Location
    Houston, Texas, USA
    Posts
    2,878
    The power outage could have uncovered almost-dead hardware nodes, corrupted file stores, and erased non-persistently saved configuration settings. I find it interesting, but not surprising, how much overhead the "cloud" setup can bring about.

    I'm sure SL is working around the clock on this. Hopefully all will be restored soon.

    Regards
    UNIXy - Fully Managed Servers and Clusters - Established in 2006
    [ cPanel Varnish Nginx Plugin ] - Enhance LiteSpeed and Apache Performance
    www.unixy.net - Los Angeles | Houston | Atlanta | Rotterdam
    Love to help pro bono (time permitting). joe > unixy.net

  3. #3
    Join Date
    Oct 2008
    Location
    Singapore
    Posts
    4,485
    Don't they have backup generators?
    LIMENEX WEB HOSTING
    Affordable High Performance Web Hosting in United States, United Kingdom & Singapore
    Web Hosting | Reseller Hosting | Managed VPS | Managed Dedicated Servers | Cheap SSL Certificates
    Check out our Managed Dedicated Server Special Deals!

  4. #4
    Join Date
    Jun 2008
    Posts
    204
    From this thread in the "Providers and Network Outages and Updates" section, the outage was only about 20 minutes. You didn't mention if you put in a trouble ticket or not. You should call them.

    I've seen servers that power up but don't restart properly, and I've seen services like httpd not start after a power up.

    http://www.webhostingtalk.com/showthread.php?t=894444

  5. #5
    Join Date
    Aug 2005
    Location
    patiala, punjab, india
    Posts
    264

    Angry

    Quote Originally Posted by web-1 View Post
    From this thread in the "Providers and Network Outages and Updates" section, the outage was only about 20 minutes. You didn't mention if you put in a trouble ticket or not. You should call them.

    I've seen servers that power up but don't restart properly, and I've seen services like httpd not start after a power up.

    http://www.webhostingtalk.com/showthread.php?t=894444
    I was probably the first person to spot the trouble as i was working on the server at that time. I created a ticket within 2-3 minutes of this trouble with their Cloud Computing Instance.

    After that, i called them as well.

    On the ticket there have been around 5 updates from LT Staff, but none of those provide any real information about what is happening and in how much time the things will be up to normal.

    Simply, they are telling...

    "We are waiting on our Information Systems team to bring the Cloud computing instances back online. Unfortunately we do not have a solid ETA on when they will be complete. We will update you when we have more information."
    Get Best Prices for Domains - Secure payments via paypal using the Resellerclub interface. Sub Resellers invited. Hosting Companies i DON'T recommend anymore: MDDHosting, Hostgator, DownTownHost.

  6. #6
    i also faced this outage
    Top10DedicatedHosting.com Dedicated Hosting

    Superior Hosting Solutions, Exclusive Specials on Dedicated Web Server

  7. #7
    Join Date
    Aug 2005
    Location
    patiala, punjab, india
    Posts
    264
    The instance is up after 10 hour of Downtime.
    And, this happened first time in more than 2 years. And i can tell that Softlayer Support doesn't work good at all.

    They are good if we need some hardware upgrade or we need to pay them additionally for any services.

    Duing outage, the support is TERRIBLE.
    Get Best Prices for Domains - Secure payments via paypal using the Resellerclub interface. Sub Resellers invited. Hosting Companies i DON'T recommend anymore: MDDHosting, Hostgator, DownTownHost.

  8. #8
    Join Date
    Jul 2009
    Posts
    126
    Maybe i have misunderstood something about the whole cloud hype but shouldn't it be possible for SoftLayer to just move your instance to other nodes that are up? I can't think that SoftLayer just have one server room where all their cloud nodes is..

  9. #9
    Join Date
    Jun 2008
    Posts
    204
    Quote Originally Posted by gdtechind View Post
    The instance is up after 10 hour of Downtime.
    And, this happened first time in more than 2 years. And i can tell that Softlayer Support doesn't work good at all.

    They are good if we need some hardware upgrade or we need to pay them additionally for any services.

    Duing outage, the support is TERRIBLE.
    2 years = 17520 hours

    10 hours of that makes it close to 99.95% uptime.

    Your math may vary depending on your mood

  10. #10
    Join Date
    Aug 2005
    Location
    patiala, punjab, india
    Posts
    264

    Angry

    Quote Originally Posted by web-1 View Post
    2 years = 17520 hours

    10 hours of that makes it close to 99.95% uptime.

    Your math may vary depending on your mood
    If it was 5 min downtime every month, i would find it acceptable.

    If it is 10 hour in a day, I guess no one will be happy with it. And, i continued with them for this time, because they were really offering good services.

    I am amazed, What SOFTLAYER MEANS BY CLOUD ???? If their team is seeing this thread, they should explain it. If cloud can have such downtime, i don't think they can ever be successful.

    Are they testing the cloud services with their clients ?
    Get Best Prices for Domains - Secure payments via paypal using the Resellerclub interface. Sub Resellers invited. Hosting Companies i DON'T recommend anymore: MDDHosting, Hostgator, DownTownHost.

  11. #11
    Join Date
    May 2006
    Location
    EU & USA
    Posts
    3,683
    As far as i know SL isn't even delivering cloud for over 2 years; more close to about 5 months (announced at 5-11-2009)

  12. #12
    Join Date
    May 2002
    Location
    Russia, Moscow
    Posts
    1,490
    And also TS should be understand when he said - "10 hours of downtime unacceptable" he don't knew what happen. Yes,i can agree that when server died and need restart and i need wait upto 10 hours to get it back because tech sleep - it is unacceptable amount of downtime when in TOS provider promise me round of clock support.
    But when cluster hard disk massive was died and amount of failed data big enough - tech need time to isolate issue, find solution and run restoration process, And it is only one scenario - solution may work or not and they need find another and so on.
    It is hard to decide are they do the best or not without any real information but usually any host need save their client base and of course most of host value customer and especially within outage time. So, it is mostly trust question because not all host would like provide end-user with detailed information about issue.
    Rustelekom LLC Dedicated server since 2002, RIPE NCC member, LIR, AS51168

  13. #13
    Join Date
    May 2006
    Location
    EU & USA
    Posts
    3,683
    Quote Originally Posted by rustelekom View Post
    And also TS should be understand when he said - "10 hours of downtime unacceptable" he don't knew what happen. Yes,i can agree that when server died and need restart and i need wait upto 10 hours to get it back because tech sleep - it is unacceptable amount of downtime when in TOS provider promise me round of clock support.
    But when cluster hard disk massive was died and amount of failed data big enough - tech need time to isolate issue, find solution and run restoration process, And it is only one scenario - solution may work or not and they need find another and so on.
    It is hard to decide are they do the best or not without any real information but usually any host need save their client base and of course most of host value customer and especially within outage time. So, it is mostly trust question because not all host would like provide end-user with detailed information about issue.
    That is a bit the problem with cloud computing; the larger your environment get; the larger the problems get when you are faced one.

    While a cloud has many protections for problems; they will be rather serious when you really get one. Clouds are just as a VPS for me; something really fun to work with; but not yet for my production systems.

    As you may have seen in other threads about this outage; most got back after about 20minutes, they are likely using dedicated servers.

  14. #14
    Join Date
    Aug 2005
    Location
    patiala, punjab, india
    Posts
    264
    I would say that they are not ready for Cloud Computing yet.

    While i was having the downtime, i started digging the forums on Softlayer (some of them are restricted for hosting clients only, i guess.).

    I found that one client on forum was telling that everyday there is around 5 - 10 minutes of accessibility problem with the Softlayer Cloud. The server doesn't go down, but isn't work fine during this period. After some time, things come back to normal.

    I would say Softlayer is still at Amateur level for Cloud Computing solutions.

    I prefer them as they can offer easy internal connectivity between my Dedicated Server and Cloud Computing Instance.
    Get Best Prices for Domains - Secure payments via paypal using the Resellerclub interface. Sub Resellers invited. Hosting Companies i DON'T recommend anymore: MDDHosting, Hostgator, DownTownHost.

  15. #15
    Join Date
    Oct 2002
    Location
    Georgia, USA
    Posts
    109

    *

    Our CCI was down for almost 12 hours.
    --
    Speckz

  16. #16
    Join Date
    Sep 2005
    Location
    USA
    Posts
    428
    Quote Originally Posted by gdtechind View Post
    I would say that they are not ready for Cloud Computing yet.

    While i was having the downtime, i started digging the forums on Softlayer (some of them are restricted for hosting clients only, i guess.).

    I found that one client on forum was telling that everyday there is around 5 - 10 minutes of accessibility problem with the Softlayer Cloud. The server doesn't go down, but isn't work fine during this period. After some time, things come back to normal.

    I would say Softlayer is still at Amateur level for Cloud Computing solutions.

    I prefer them as they can offer easy internal connectivity between my Dedicated Server and Cloud Computing Instance.



    Been great for me and what I use it for. 100% registered UPtime for three months now tells me they know what they're doing.

    I'm on the 4gb Ram Cloud instance, and I have been pushing roughly 3gb-4gb for three months now, Constantly. No issues whatsoever.
    Brandon Williams - 7 Years and counting in the hosting industry.

  17. #17
    Join Date
    Aug 2005
    Location
    patiala, punjab, india
    Posts
    264
    Quote Originally Posted by B r a n d o n View Post
    Been great for me and what I use it for. 100% registered UPtime for three months now tells me they know what they're doing.

    I'm on the 4gb Ram Cloud instance, and I have been pushing roughly 3gb-4gb for three months now, Constantly. No issues whatsoever.
    Were you having downtime today ?
    Get Best Prices for Domains - Secure payments via paypal using the Resellerclub interface. Sub Resellers invited. Hosting Companies i DON'T recommend anymore: MDDHosting, Hostgator, DownTownHost.

  18. #18
    Join Date
    Sep 2005
    Location
    USA
    Posts
    428
    Quote Originally Posted by gdtechind View Post
    Were you having downtime today ?
    I stated 100% UP time, so No I haven't, Maybe I just been lucky?
    Brandon Williams - 7 Years and counting in the hosting industry.

  19. #19
    Quote Originally Posted by B r a n d o n View Post
    I stated 100% UP time, so No I haven't, Maybe I just been lucky?
    Well, it looks like they had a power outage in Server Room 5, so if you're not there, I'd imagine you are fine.

    Like another user said though, I'm surprised the UPS systems, or the backup generators didn't prevent it, if it was indeed a power outage. I also thought they based their cloud on a SAN for failover and high availability.
    :: Bobby - PreciselyManaged.com - Precision Hosting Solutions
    :: Introducing real fully managed VPS servers!
    :: Check it out: www.PreciselyManaged.com/vps-hosting.html
    :: Security Mailing List for Hosting Providers

  20. #20
    Join Date
    Aug 2005
    Location
    patiala, punjab, india
    Posts
    264
    Yes, telling Power Outage as a reason is funny.

    Why these companies make big claims about their "State of the Art Data centers" ?
    Get Best Prices for Domains - Secure payments via paypal using the Resellerclub interface. Sub Resellers invited. Hosting Companies i DON'T recommend anymore: MDDHosting, Hostgator, DownTownHost.

  21. #21
    Join Date
    May 2006
    Location
    EU & USA
    Posts
    3,683
    Quote Originally Posted by gdtechind View Post
    Yes, telling Power Outage as a reason is funny.

    Why these companies make big claims about their "State of the Art Data centers" ?
    If you payed attention to the industry you will see that power outages are the #1 problem for any datacenter; even the largest companies have to deal with this problem (i.e. Google, Rackspace, The Planet had recent serious issues).

    Should it happen ? No of course not, no datacenter would like this to happen, but the fact is nearly all datacenters (if not all) do face power outages from time to time.

    SL solved the power issue within 20 minutes; which is really fast, trust me.

  22. #22
    Join Date
    Aug 2009
    Posts
    33
    If it's just one server room within a facility, chances are that this was some kind of internal power infrastructure failure, one that backup generators may not have been able to prevent because they are farther up the line and designed to handle facility wide power failures caused by an external event.

    Also sounds like their cloud is contained within specific "server rooms" which probably mean that while they have failover capibility, if the whole system including SANs and nodes goes offline there isnt a whole lot to fail over to eh? Maybe a redesign should be considered.
    ---
    Liquid Web VPS Customer for years --- Worth every extra penny spent.
    ---

  23. #23
    Join Date
    Oct 2002
    Location
    Georgia, USA
    Posts
    109
    Quote Originally Posted by 040Hosting View Post

    SL solved the power issue within 20 minutes; which is really fast, trust me.
    20 minutes? My cloud computing instance was down for almost 12 hours.
    --
    Speckz

  24. #24
    Join Date
    Aug 2005
    Location
    patiala, punjab, india
    Posts
    264
    Quote Originally Posted by 040Hosting View Post
    SL solved the power issue within 20 minutes; which is really fast, trust me.
    Power Issue was solved in 20-30 minutes. But, after that, there were the issues, which took around 10 hours.

    The questions which need answers are.

    They were not having enough technical staff ?
    Their staff wasn't aware what to do ?
    Was the process so long that it was normal to take this much of time?
    Why Softlayer support didn't inform about the ETA (i asked them twice on phone and 5 times in ticket)?


    Their staff has finally said that they will soon provide the ROF to me in the ticket. Lets see what they have to say.
    Get Best Prices for Domains - Secure payments via paypal using the Resellerclub interface. Sub Resellers invited. Hosting Companies i DON'T recommend anymore: MDDHosting, Hostgator, DownTownHost.

  25. #25
    Join Date
    May 2006
    Location
    EU & USA
    Posts
    3,683
    Quote Originally Posted by gdtechind View Post
    Power Issue was solved in 20-30 minutes. But, after that, there were the issues, which took around 10 hours.

    The questions which need answers are.

    They were not having enough technical staff ?
    Their staff wasn't aware what to do ?
    Was the process so long that it was normal to take this much of time?
    Why Softlayer support didn't inform about the ETA (i asked them twice on phone and 5 times in ticket)?


    Their staff has finally said that they will soon provide the ROF to me in the ticket. Lets see what they have to say.
    You where blaming the power issue took so long, but this did seem to take only a short while (with some exception according to the information in their portal); but it seems there where other problems caused by the power issue; which caused the cloud to be down much longer.

    Anyways i feel for you 10 hours is a long time; but many users asking the same question and asking for updates 5 times in tickets and twice on the phone doesnt speed up the process either.

    Did you know that a single linux machine with large disks could already take hours to do a FSCK when booting ? And no technical person who can stop this process or make it go faster when its busy with it ? An ETA can not even be given on this. Can you imagine if a whole cloud goes down ?

    Again; it is sad you feel you did not get the notifications you liked; and its sad they couldn't bring your systems up any faster; but if your data is so important to you, you should look into the technical implications of the solution you choose. This includes and is not limited to the worst case scenario.

Page 1 of 2 12 LastLast

Similar Threads

  1. What is cloud computing?
    By Kevin in forum Cloud Hosting
    Replies: 29
    Last Post: 09-16-2009, 04:00 AM
  2. NEWS: Softlayer offers Cloud Computing
    By ryan14 in forum Cloud Hosting
    Replies: 35
    Last Post: 06-16-2009, 09:20 AM
  3. Cloud Computing
    By eTalkup in forum WHT Announcements, Feedback and Questions
    Replies: 20
    Last Post: 07-20-2008, 10:24 PM
  4. Cloud Computing
    By douglasdcin in forum Web Hosting
    Replies: 17
    Last Post: 05-07-2008, 01:44 PM

Related Posts from theWHIR.com

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •