Page 1 of 2 12 LastLast
Results 1 to 40 of 63
  1. #1
    Join Date
    Apr 2001
    Location
    Austin, Texas
    Posts
    508

    HE.net Fremont entire data center power loss

    We had servers in opposite ends of the room (row4 near the NOC, row11 near the kitchen) at HE.net Fremont go down hard... power loss.

    Anyone know details?

    1:30pm Pacific Time

  2. #2
    Join Date
    Apr 2001
    Location
    St. Louis, MO
    Posts
    2,508
    Yeah, heard it could be related to UPS maintenance (heard that from 3 people)?

  3. #3
    Join Date
    Jun 2001
    Location
    San Diego, CA
    Posts
    283

    BS

    I just got off the phone with Tiffany at HE.

    She said it was a "hiccup" during routine UPS work.

    I call bullsh*t since we have 8 cabinets there more or less offline and another cabinet in a totally different part of the datacenter also offline.

    -davidu

  4. #4
    Join Date
    Mar 2004
    Posts
    185
    Hello,


    I am thinking to contract cabinet full with line 100mbits in I HE, but finish to me scaring, you who you are with them who you can say to me? that uptime has? they are good? recommendable? excellent? bad?


    Please Help for HE

  5. #5
    Join Date
    Jun 2001
    Location
    San Diego, CA
    Posts
    283
    Originally posted by comers
    Please Help for HE
    Please help for davidu

  6. #6
    Join Date
    Dec 2000
    Location
    Sunny Florida (Orlando)
    Posts
    397
    I would like to see an explanation from HE.Net as my sales rep is unresponsive to this situation. We have 1 cabinet and around 50-75mbps from them out there and it was down.

  7. #7
    Join Date
    Aug 2002
    Location
    California
    Posts
    247

    sharing our bs

    No luck on the phone (all circuits busy), emailing the noc yielded a "glitch during routine UPS maintenance". We had a guy over there who got "accidental power down of main UPS" from one of the techs. Apparently, there are a lot of angry people over there.

    It's not the end of the world, but very disappointing indeed.

  8. #8
    Join Date
    Jul 2003
    Location
    Saint Louis, MO
    Posts
    85
    all of our cabinets in fremont,ca went down hard as well. not very happy about this situation.

  9. #9
    Join Date
    Aug 2002
    Location
    California
    Posts
    247
    Hey webby--where did you disappear to?
    Originally posted by webtech
    I would like to see an explanation from HE.Net as my sales rep is unresponsive to this situation. We have 1 cabinet and around 50-75mbps from them out there and it was down.

  10. #10
    Join Date
    May 2004
    Location
    New York
    Posts
    21
    Been down since 4:30 pm EST (now 7:30pm EST). Anyone else still down?

  11. #11
    Join Date
    Apr 2004
    Posts
    55
    Same here; I had a cabinet go down, and my reseller's cabinets appear to have as well according to the router reboot stats.

    The timing isn't such to annoy me at this time assuming everything comes up OK, just strikes me as kind of funny. Will be interesting to see what happens as a result.

  12. #12
    Join Date
    Apr 2001
    Posts
    1,044
    Damn! sounds like when exchangecolo had their problem last year.

  13. #13
    Join Date
    Apr 2004
    Posts
    55
    Originally posted by ottenhoffs
    Been down since 4:30 pm EST (now 7:30pm EST). Anyone else still down?
    According to my router graphs, it looks like it came back up around 3:30PST, but I'm not certain if it may have actually come up earlier or later than that. At any rate, everything seems to be working now.

  14. #14
    Join Date
    Jun 2001
    Location
    San Diego, CA
    Posts
    283
    We still have a ton of people offline and HE's phones are totally swamped.

    They have sucked, continue to suck and will probably suck forever.

    Even when they are online and working, they suck. we get so much arp traffic and broadcast storms from other subnets it's shocking.

    They are also listed all over SPEWS and whatnot. (not that I care about SPEWS)

    And has anyone noticed how everyone who works there seems to be related?

    -davidu

  15. #15
    Join Date
    Jul 2003
    Location
    Saint Louis, MO
    Posts
    85
    looks like they need ups's for their ups's.

    i sure hope someone got fired for this mess...

    :/

  16. #16
    Join Date
    Jun 2001
    Location
    San Diego, CA
    Posts
    283
    AHHH

    We just had boxes that WERE up go back DOWN

    and they still aren't picking up the phone....when it rings....mostly we get fast-busy's

    So crappy.

    Level3 in San Diego and Equinix in IAD have never been like this....HE is like this often.

    To their credit, they hook us up on ACL entries...but that's not worth it. We pay cheap prices and we get cheap service...I've been dying to leave, maybe now we will.

    -davidu

  17. #17
    Join Date
    Nov 2002
    Posts
    2,780
    You have like 9 cabinets and now u're complaining they suck and want to leave? why didnt u figure this out when u ordered your second one...

  18. #18
    Join Date
    Apr 2001
    Location
    Austin, Texas
    Posts
    508
    I want to be clear that I've been at HE.net Fremont for 2 years, and this is the first time we have had a power loss.

    It happens. No pattern of problem here.

    The actual outage was pretty short, for us only 30 minutes. If you servers are switches were in good order and set to come on after power up...

    We have a couple servers that require manual power on... took them about 90 minutes to get through on phone and have them powered up. They are obviously very busy (there are thousands of servers in that room).
    Last edited by StephenRS; 05-03-2004 at 08:05 PM.

  19. #19
    Join Date
    May 2004
    Location
    New York
    Posts
    21
    Originally posted by StephenRS
    I want to be clear that I've been at HE.net Fremont for 2 years, and this is the first time we have had a power loss.

    It happens. No pattern of problem here.
    Same here. This is the first power loss for us.

    I guess the more disturbing part of this incident is the way it supposedly happened. This is not the August 2003 blackout... Word is that it happened while performing maintenance on the UPS...

    So, agreed that power loss is definitely not a patter, just a bit disturbed at how it occurred.

  20. #20
    Join Date
    Jul 2003
    Location
    Saint Louis, MO
    Posts
    85
    maybe that overloaded PDU on row 3 thats been beeping constantly for the last 6 months finally blew-up.

  21. #21
    Join Date
    Nov 2002
    Posts
    2,780
    Is it all up yet?

  22. #22
    Join Date
    Mar 2004
    Posts
    185
    they say that this in HE.NET is very frequent, others say that it is the next time


    in which we were? ? help please for mi decision

  23. #23
    Join Date
    May 2004
    Location
    New York
    Posts
    21
    Originally posted by Mfjp
    Is it all up yet?
    i'm not up yet.

  24. #24
    Join Date
    Apr 2001
    Location
    Austin, Texas
    Posts
    508
    i'm not up yet.
    Are you sure it isn't just a problem with your server? I mean the hard drive could have corrupted, it could have a bad battery and BIOS waiting for a prompt...

    Are you using a reseller who has their own problems?

  25. #25
    Fremont HE racks down from 1:35 to 2:30 PST. Couldn't get through to tech support. Some boxes still not up, hoping the power spike wasnt too bad. A marketing rep mentioned the same UPS "small glitch", nice way to deal with customers that need real answers. Apparently no redundancy?

  26. #26
    Join Date
    Mar 2004
    Posts
    185
    they say that this in HE.NET is very frequent, others say that it is the next time


    in which we were? ? help please for mi decision

  27. #27
    Join Date
    Jul 2003
    Location
    Venice Beach, Ca.
    Posts
    83
    "Hmmmmm... I wonder what this here little red button does if I push it..." he says to himself curiously as he looks at the sign in front of him with the very clear warning... "Do not press this red button!".

  28. #28
    Join Date
    Dec 2002
    Location
    Vancouver, Canada
    Posts
    541
    Everything should be up now, all my servers are up. And this is the first time i've experienced this, other than this there has been no problems with he.net in fremont.

  29. #29
    Join Date
    Jun 2001
    Location
    San Diego, CA
    Posts
    283
    This is not our first power problem with them although it is the largest.

    Yes, we should have known to move a long time ago, but it was easy to say "this is pretty good."

    They once emailed us saying we were using too much power (but we weren't tripping our amp) and that we had like a couple days to fix it.

    Their PDU's are *ALWAYS* blowing alarm beeps, at all times. I've never heard of a time they weren't.

    -davidu

  30. #30
    Join Date
    Apr 2004
    Posts
    55
    Anyone here know DavidU? He seems kind of....trollish.

    I'd say an email warning that you're getting close to your power limit is good; gives you a chance to take care of things before normal variations trip the circuit.

    I agree that 15AMP is not as good as 20, but it's still more than I use. hrug: And cabinets are pretty cheap in Fremont.

  31. #31
    Join Date
    Mar 2004
    Posts
    185
    then they guarantee that this it is the first serious problem that to had HE.NET?


    they recommend HE.NET to contract fullrack to me with 100Mbits with 40AMP? I have thought to mount 40 server 1U for my clients. I will not have problems with the line or the energy?

  32. #32
    Join Date
    Nov 2002
    Posts
    2,780
    DavidU, mind me to ask if you have any transport capacity to outside He.net Fremont?

  33. #33
    Join Date
    Apr 2004
    Posts
    55
    40AMP? I believe HE.net Fremont provides power in 15AMP increments only.

    Also, HE.net cabinets are very shallow. I don't remember the exact depth, but I'm very close to the end even though I don't have any particularly deep gear.

  34. #34
    I have thought to mount 40 server 1U for my clients. I will not have problems with the line or the energy?
    Yes and no. Depending on the hardware you are planning on putting in there(dual CPU's, multiple harddrives per machine), you could easily burn through those 40AMP's with 40 1U's. Judging by how HE is dealing with their power right now I would be a little scared putting that many 1U's on a 40AMP @ HE, but thats just me.

    Peter

  35. #35
    Join Date
    Mar 2004
    Posts
    185
    they would not be dual, they would be amd, they celeron, pentium 4 and xeon.

  36. #36
    Join Date
    Nov 2002
    Posts
    2,780
    HE.net cabinet's are all 15 amps. Upgrading the power cost the same as getting a new cabinet.

  37. #37
    Join Date
    Nov 2002
    Posts
    2,780
    15Amps trips at 12Amps, You'll be lucky getting 20 servers in there (probably 15 would be the MAX i would do...).

  38. #38
    Join Date
    Mar 2004
    Posts
    185
    there is no problem, I will pay the increase of AMP

  39. #39
    Join Date
    Aug 2002
    Location
    California
    Posts
    247
    We've been in Fremont for years and we've never seen a power issue like this. Each 15AMP circuit is individually breakered and I've had customer's cabinets' trip, but never a row or the entire datacenter.

    They have really slipped recently, a few "router reboots" over the past couple months have really made up for the otherwise "above average" service we've come to know.

  40. #40
    Join Date
    Mar 2004
    Posts
    185
    they guarantee to me that it is a very good line? as it goes of latency? of uptime?

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •