Results 1 to 17 of 17
  1. #1
    Join Date
    Mar 2002
    Location
    London & Kent, UK
    Posts
    372

    HostEurope London power outage - any better providers?

    HostEurope (UK) are sending out emails to announce downtime to the London data centre of up to 6 hours on a Tuesday in a couple of weeks time. This is allegedly for maintainance to the power system, although I've not been able to speak to anyone who actually knows why they need downtime in order to do this. Given that they now have multiple power providers, (they learnt that lesson after the DC had hours of downtime on a couple of occasions when they had only one provider), and one would hope that they have onsite power generators too, I'm at a loss for why they would need to shutdown all power in order to perform maintainance. Certainly it should be part of the design of the DC that they can shut down providers individually in order that work can be completed, and also to be able to reroute power around components in order that critical pieces can be replaced.

    This all sounds like another case of incompetence and bad planning on their part, something that they've demonstrated an aptitude for on several occasions in the past (physically losing a hard drive for over a day when trying to move it from one machine to another, for example),. To cap it all, making a change in the middle of the week (sure 1am may be night time here, but it's not in australia or for many of the places where our visitors come from) is ludicous as well, and it seems that they cannot power on machines remotely, and have to visit each machine in turn to restore power. Mickey mouse outfit? Well we know that they're not quite that, but it sure sounds like it, and certainly not "World Class" as they claim on the telephone. Thankfully we have servers in multiple DC's so that we can survive downtime on the main server without losing web presence, but that's not really the point.

    So I'm wondering whether this is this par for the course, or if there any DC's that are designed to allow power maintaince to be performed safely without having to pull out the plug on the customers too?

  2. #2
    Join Date
    Sep 2002
    Location
    London/Edinburgh
    Posts
    1,131
    Hello.

    If anyone has a copy of this email, i would be interested in reading it.
    Posting it in this thread would be great

    Regards,
    Thomas Currie
    Europhase UK Limited - XEN/OpenVZ UK/US Virtual Private Servers | R1Soft Backups for Web Hosts
    Custom R1Soft Backups | UK/US Web Hosting | UK/US Virtual Private Servers
    99.9% Uptime SLA | Non Overloaded Service | Guaranteed Ram / Resources
    Company: SC345142 VAT: GB-947754180 Tel: 0870 4710 895

  3. #3
    Join Date
    Mar 2002
    Location
    London & Kent, UK
    Posts
    372
    See below.

    Checking the SLA, it interestingly doesn't make exceptions for so-called planned maintainance, and claims 99.9% up time. So if the network is down for more than the amount allowed in the sla (about 5 hours based on 0.1% of May) then there should be grounds for compensation.

    Generally HE are reliable, and support staff are helpful, but their problems always seem to be in how the manage occasional issues, and lack of planning to minimise potential downtime is the big failing. The "we apologise for *any* inconvenience" is also a bit cheeky, as it's hard to see how this isn't going to be a major inconvenience for any business that's serious about it's internet presence.

    --

    Planned Maintenance - London Datacentre - 18th May 2004

    Dear Valued Customer,

    We would like to advise you that we will be performing essential power maintenance
    work in the LONDON datacentre between 1 AM and 3AM on TUESDAY 18TH MAY. This work is
    vital to ensure we continue to maintain the best possible datacentre environment and
    the highest quality network to support your service.

    We anticipate that downtime will last no longer than 90 minutes for the data centre
    work, but would advise that some customers may experience unavoidable downtime of up
    to 6 hours whilst all servers are restarted.

    To ensure this process runs as smoothly as possible, it would be beneficial if
    customers could power down their own servers prior to the 1 AM start. Host Europe
    engineers will personally power on all servers immediately all necessary work is
    completed, and will aim to keep downtime to an absolute minimum.

    We apologise for any inconvenience caused by this work. Maintaining an efficient,
    resilient infrastructure is part of our commitment to delivering a strong and
    efficient service for all of our customers going forward.

    Yours sincerely,

    Host Europe

  4. #4
    Join Date
    Aug 2002
    Location
    UK
    Posts
    1,040
    It's also odd that they'd give less than 2 weeks notice for something as huge as this... they must've known before now that they would have to take the power down, so why didn't they give people more advance notice?

    essential power maintenance work

    That is so incredibly vague that it could really mean anything at all. It's strange that they can't be more specific.

    Something just seems a little fishy
    Robin Balen
    Gyron Internet Ltd - http://gyron.net/
    UK colocation, managed hosting and connectivity services with 100% uptime SLAs

  5. #5
    Join Date
    Aug 2002
    Location
    Trouble will find me!
    Posts
    1,470
    Would this have anything to-do with the fact they have been bought out by PIPEX?

    Maybe there power system was not built on a n+1 redundant system and PIPEX are making changes to stabilise these issues ...
    ^^ IM WITH STUPID!! ^^

    "The only way to overcome fear, is to challenge it head on"
    "The quickest way to get over a woman, is to get under another"

  6. #6
    Join Date
    Sep 2002
    Location
    London/Edinburgh
    Posts
    1,131
    Shazzy,

    Thats what I was thinking...
    6 hours is extremely major, wonder if we could fish for more details!

    Regards,
    Thomas Currie
    Europhase UK Limited - XEN/OpenVZ UK/US Virtual Private Servers | R1Soft Backups for Web Hosts
    Custom R1Soft Backups | UK/US Web Hosting | UK/US Virtual Private Servers
    99.9% Uptime SLA | Non Overloaded Service | Guaranteed Ram / Resources
    Company: SC345142 VAT: GB-947754180 Tel: 0870 4710 895

  7. #7
    Join Date
    Jul 2003
    Posts
    40
    6 hours is nothing compared to FDC's UNANNOUNCED downtime of more than 12 hours.

  8. #8

    Thumbs down

    at least they give their customer announcement asap.
    fdc is worst

  9. #9
    Join Date
    Sep 2002
    Location
    London/Edinburgh
    Posts
    1,131
    Hello.

    FDC and HostEurope are on completely levels...
    Dont even try to compare them

    Regards,
    Thomas Currie
    Europhase UK Limited - XEN/OpenVZ UK/US Virtual Private Servers | R1Soft Backups for Web Hosts
    Custom R1Soft Backups | UK/US Web Hosting | UK/US Virtual Private Servers
    99.9% Uptime SLA | Non Overloaded Service | Guaranteed Ram / Resources
    Company: SC345142 VAT: GB-947754180 Tel: 0870 4710 895

  10. #10
    Join Date
    Mar 2002
    Location
    London & Kent, UK
    Posts
    372
    "6 hours is extremely major, wonder if we could fish for more details!"

    It is, although they say 90 minutes for the work (although they've always underestimated these figures in the past), and 6 hours to flick the switch on everyones machines. Aside from the fact that there simply shouldn't need to be any power outage ever, what troubles me is that they aren't able to bring up machines rather quicker than that. They only have 3000 machines in the DC, and whilst they might want to avoid possible network storms from all machines coming up around the same time, (although such things shouldn't happen anyway), it shouldn't take 6 hours to get things restarted.

  11. #11
    Join Date
    Aug 2002
    Location
    Trouble will find me!
    Posts
    1,470
    Originally posted by GetYourHitS
    6 hours is nothing compared to FDC's UNANNOUNCED downtime of more than 12 hours.
    These are totally different fish’s, Host Europe is classed as a premium provider along side the Rackspaces of the world, albeit they may not be as good (network wise) they have a reputation to live up to and many corporations and large business depend on them in UK/Europe.

    Could you imagine Rackspace announcing such maintenance?

    Something is fishy and I do think we are all being told the truth.
    ^^ IM WITH STUPID!! ^^

    "The only way to overcome fear, is to challenge it head on"
    "The quickest way to get over a woman, is to get under another"

  12. #12
    Join Date
    Mar 2002
    Location
    London & Kent, UK
    Posts
    372
    Something is fishy and I do think we are all being told the truth.
    I'm sure we're being told the truth, (or did you mean not being told), but what we're being told isn't why, and that's the crux of the matter. 3 main reasons to use a data centre are a guarantee of power, security and connectivity. There should never be a need for downtime such as this, and so something must be majorly wrong, either in terms of the managment of the work, or in the datacentre design itself so that it was impossible to avoid having to turn off the power. The reallity is that if they could avoid power outage but it would take 3 solid days to do the work instead of 2 hours, then it should take 3 solid days, all hands to the pump, overtime paid if necessary etc. No if's buts or maybes on that one. So, are they just slacking and not willing to put in the effort to maintain power through whatever work needs to be done, or is the datacentre badly designed?
    Real-time intrusion protection and error reporting for PHP sites ioncube24.com
    Software protection for website owners and PHP developers ionCube PHP Encoder

  13. #13
    Join Date
    May 2004
    Location
    Manchester, UK
    Posts
    30
    I personally think it's safe to assume that Pipex wanted to sort out the data centre as quickly as possible so they can move onto other issues.

    I imagine the team at Pipex won't stand for anything less than excellence in the data centre. It is surprising to hear that this kind of data centre design bungling can happen...

    Dave

  14. #14
    Join Date
    Aug 2002
    Location
    Seattle
    Posts
    5,512
    Co-locate a 16kVA Symmetra LX, give it some time to charge, and you should be able to weather the outage

  15. #15
    Join Date
    Aug 2002
    Location
    DC
    Posts
    3,635
    Originally posted by s.h.a.zz.y
    Would this have anything to-do with the fact they have been bought out by PIPEX?

    Maybe there power system was not built on a n+1 redundant system and PIPEX are making changes to stabilise these issues ...
    That sounds like it could be the reason to me. Let's wait for an official word on this, though.

    - Matt

  16. #16

    HE aka Pipex

    I personally think that Pipex are just beefing themselves up to sell on the data centre to someone else.

    Therefore they don't care about their clients in the HE centre.

    Power Outages, DOS attacks, really how can that happen if you're well equipped, I mean that is what they sell to you don't they.

    We have been with them for more than 1.5 years now, and they've now cut themselves back to a 9-5pm data centre, come on how can you really offer a co-lo 9-5pm, so the lady on the phone says to me, can't you predict when your server will crash.... hmmm let me see what are the lottery numbers for tonights draw???

    And if you want out, as they have now changed the rules, well there is no way out, leave if you want but no refund. So we're stuck.

    I say if you're even thinking about co-lo or any type of hosting, stay well away from Pipex.

  17. #17
    Join Date
    Mar 2002
    Location
    London & Kent, UK
    Posts
    372

    LOL they lose hard drives too

    can't you predict when your server will crash
    LOL

    Worse things also happened for us after this, and serve as a further warning about the HE/Pipex outfit.

    The summary is that the drive started to fail on one of our servers, and they were asked to replace it. We checked to ensure that they could mount the old drive read only in addition to the new drive, allowing data to be copied across data easily, and get going quicker than using backups. No problem they said, and estimated 30 minutes to do it. I was optimistic at this point.

    After 24 hours of downtime, the status was that they had physically lost the hard drive that they were replacing, and the server was still down. This wasn't good, and any faith in them had long since evaporated by this point.

    Sometime later, support claimed to have found the drive, but said that the partition table was damaged and no data could be found. This I knew to be rubbish, so they were asked to mount it anyway so that we could examine the raw drive. Whether they believed that they had found the correct drive, or whether they thought that we knew even less than them I don't know, but the drive they had put in was someone elses, from a Windows machine, and so not even someone elses from a Linux machine. The drive hadn't been reformatted, and we were quickly able to tell them whose drive they had just installed in our server, and that they were claiming to be ours. So much for data security on drives taken out of customers machines!

    A while later, our original drive was finally found, althuogh by then of course the machine was mostly going from backups.

    This whole experience was pretty shocking, and I've since learnt from our ex-account manager that they were really running on a shoestring, and only just keeping their heads above water. I can't confirm that, but certainly the experiences when there have been problems would tend to bear that out.

    I'm not sure that things are better now, and when just a week or so ago there were major network problems in the London DC that affected connectivity to servers from within the UK, although thankfully not from elsewhere, it took many hours before the problems were correctly resolved.

    We no longer use HE/Pipex exclusively, and will be phasing out servers with them in due course. As with anything, buyer beware.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •