Page 1 of 2 12 LastLast
Results 1 to 40 of 59
  1. #1
    Join Date
    Mar 2004
    Location
    Edmonton, Canada
    Posts
    339

    Wholesale Internet Down?

    Are any other users experiencing trouble getting to servers with Wholesale Internet? Their site, as well as my server, seem to be down.

    http://www.wholesaleinternet.com/ is them
    http://www.lalso.ca is my server

    Thanks in advance
      0 Not allowed!

  2. #2
    Join Date
    Aug 2001
    Location
    Bay Area - Ca
    Posts
    289
    I am not able to access there site from California. so yep the seem down.
      0 Not allowed!

  3. #3
    Join Date
    Mar 2004
    Location
    Edmonton, Canada
    Posts
    339
    *sigh*

    Thanks
      0 Not allowed!

  4. #4
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    there is probably an issue with power in the building.

    another provider is down as well ...
      0 Not allowed!

  5. #5
    Join Date
    Aug 2004
    Posts
    83
    They told me over an hour ago it was their UPS. Not sure how a UPS can take down a DC. Not sure if he meant power was out and their UPS failed.. Anyway they told me to e-mail support.

    Funny considering their server is down - don't think they will get the e-mail.
      0 Not allowed!

  6. #6
    Join Date
    Mar 2004
    Location
    Edmonton, Canada
    Posts
    339
    Anyone know more about this other provider who is down?
      0 Not allowed!

  7. #7
    Join Date
    Oct 2004
    Location
    Rockwall, Texas
    Posts
    11
    WSI's down for me in Dallas. I'm actually slightly relieved, though, cause I was a bit late in paying, so I was hoping that I didn't have it shut off on me and lose all my sites or something. I need to make a new backup ;-).

    Rick
      0 Not allowed!

  8. #8
    Join Date
    Nov 2004
    Location
    Toronto, Canada
    Posts
    163
    I'm in Toronto and I am down as well with WSI, I thought it was just me, glad to see I am not the only one.
    I'm sure Aaron and James are on it.
      0 Not allowed!

  9. #9
    Join Date
    Aug 2003
    Location
    PA
    Posts
    1,899
    WHI down for me also. Both my server and also their website is unaccessable from PA. Server also not accessable from FL through another of my servers. Seems their entire network has lost connectivity.

    Hopefully WHI gets this issue sorted.

    -Justin

    EDIT: Just noticed outage is now already at just over 2 hours. Not looking good.
      0 Not allowed!

  10. #10
    Join Date
    Aug 2004
    Posts
    83
    Anyone have an updated word? Aaron, you out there?
      0 Not allowed!

  11. #11
    Join Date
    Oct 2003
    Posts
    64
    I have also a server at WSI and its down...
      0 Not allowed!

  12. #12
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    ok people ...

    the WHOLE building is having power issues. so it's not only james and aaron working on this; it's MCI, cogent, WilTel, kcnap, list goes on ...

    from what I heard, it could be awhile before things are back up...

    damn storm season in missouri!!!
      0 Not allowed!

  13. #13
    Join Date
    Aug 2003
    Location
    PA
    Posts
    1,899
    I just got off the phone with one of the WHI support personnel and they are working on getting secondary power switched over. There are engineers on site as well as nearly every network carrier also. I have been told they will be power cycling everyone to be sure everybody is back online once the power is restored.

    -Justin
      0 Not allowed!

  14. #14
    Join Date
    Apr 2002
    Location
    North Kansas City, MO
    Posts
    2,565
    Hello All,

    Here is a brief synopsis of the issue so far.

    The building we are in is undergoing some renovations. One of the contractors didn't plug a storm drain they had cut from the roof. It's been raining here for several days. The storm drain has been emptying into the basement and sub-basement.

    At this point the sub basement in the building is completely under water, to the point where the basement above it has an inch of standing water on the floor. Power comes into the building in the sub basement. All the transformers and power distribution hardware is under water. Our generator is also in the sub basement along with several other providers.

    We are waiting for some backup generators to arrive at 6am (central). This will restore limited service. The power systems in the building are trashed and may take days or weeks to fully restore.

    We are somewhat fortunate in that our transformers are on the 9th floor so as soon as the fire department gets the basement pumped out it should just be a matter of KCP&L flipping our cutover switch and we'll be back up.

    At this point I don't have an ETA. Williams, Cogent, Qwest, Expedius, Broadwing, MCI, SBC.... etc etc etc are all in the same boat and thier NOC techs are all on site as well.

    I will update you again when I am able.

    Aaron
      0 Not allowed!

  15. #15
    Sounds like a huge problem. I hope it all gets worked out. Would be a shame for clients to lose business to a problem with dc building structure.
      0 Not allowed!

  16. #16
    Join Date
    Jun 2004
    Posts
    34
    mo powah to zee bryant
      0 Not allowed!

  17. #17
    Join Date
    Oct 2003
    Posts
    64
    Wholesaleinternet.com is UP. my server is still down. What is the situation status?
      0 Not allowed!

  18. #18
    Join Date
    Mar 2004
    Location
    Seattle, WA
    Posts
    2,561
    Wow, I didn't know the storm yesterday was that bad. Now I feel kinda better that I was not able to pick a Kansas City datacenter for colocation, although I regret I had to colo in NYC DANGIT! I hope things get worked out Aaron. Is Switch and Data down? Also, you guys got the UPS in? I saw someone post saying something about a UPS, and you said you are getting one this month.
      0 Not allowed!

  19. #19
    Join Date
    Aug 2003
    Location
    PA
    Posts
    1,899
    TheNetWay,

    according to what I was told over the phone when I had called them, the technician/support person I spoke to did state the UPS's seized/drained and that is when the systems all went down obviously. So, yes I would presume they now have UPS backups.

    Aaron is there any ETA on when the client systems will be back online?

    -Justin
      0 Not allowed!

  20. #20
    Join Date
    Oct 2003
    Posts
    64
    I am back up.
      0 Not allowed!

  21. #21
    my server is still down?
      0 Not allowed!

  22. #22
    Join Date
    Mar 2004
    Location
    Bristol, UK
    Posts
    231
    So is mine.
    I wouldn't trust it too much if it did come back just now.
    It sounds like they're going through power outage hell at their data centre at the moment and who knows if there will be more outages until the power company sort out supply?
    I really feel for the guys at WII... good luck and here's hoping for a happy ending.
      0 Not allowed!

  23. #23
    Join Date
    Aug 2003
    Location
    PA
    Posts
    1,899
    server still down here as well. I wonder what ever happened to the backup generators. Hopefully this mess is fixed shortly. Would really be nice if power companies wouldn't drag their feet just because it is early morning.

    EDIT: Just got a reply from Aaron that they are restoring power and servers should be coming back up in blocks. More then likely being done to prevent any chance of overloads.
      0 Not allowed!

  24. #24
    Join Date
    Mar 2004
    Location
    Seattle, WA
    Posts
    2,561
    They do not have backup generators. The building does, but it's not theirs, it's MCI's.
      0 Not allowed!

  25. #25
    aaron ? any news?
      0 Not allowed!

  26. #26
    Join Date
    Aug 2002
    Location
    Kansas City. MO
    Posts
    1
    Originally posted by WII-Aaron
    ... One of the contractors didn't plug a storm drain they had cut from the roof. It's been raining here for several days. The storm drain has been emptying into the basement and sub-basement.

    At this point the sub basement in the building is completely under water, to the point where the basement above it has an inch of standing water on the floor. Power comes into the building in the sub basement. All the transformers and power distribution hardware is under water. Our generator is also in the sub basement along with several other providers.

    ...
    Man, I'd hate to be that contractor this morning.
      0 Not allowed!

  27. #27
    Join Date
    Mar 2004
    Location
    Bristol, UK
    Posts
    231
    My server has just come back online and looking at uptime it has been powered on for over 2 hours so it looks like the switches are being brought up after the servers are getting power.
    Good effort to all involved at Wholesale Internet.
      0 Not allowed!

  28. #28
    Join Date
    Nov 2004
    Location
    Toronto, Canada
    Posts
    163
    My server at WSI says it been up 6 hours, but the site was still not working checked the dns report sure enough lame nameservers. Configured the DNS Administration again and "Bingo" she's lit up.

    Good job Aaron and James.
      0 Not allowed!

  29. #29
    Join Date
    Mar 2004
    Location
    Edmonton, Canada
    Posts
    339
    Thanks for the update Aaron, I for one as a Customer totally understand. You guys have been good to us, and this I can live with
      0 Not allowed!

  30. #30
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    ha. ****** day to be in the bryant building ...

    even bigger providers suffered outages due to their generators being under water from what I've been told.

    somebody's going to be getting sued. heh
      0 Not allowed!

  31. #31
    Join Date
    Aug 2003
    Location
    PA
    Posts
    1,899
    even bigger providers suffered outages due to their generators being under water from what I've been told.

    somebody's going to be getting sued. heh
    Same thing I heard off the WSI person I spoke to on the phone. Wouldn't be suprised if someone gets into serious **** over this mess. Hopefully all the mains can be restored soon and there wasn't too much damage caused.

    Good job WSI at restoring this mess back to working condition again.



    -Justin
      0 Not allowed!

  32. #32
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    heh.

    gotta love all the *'s in the last two posts.
      0 Not allowed!

  33. #33
    Join Date
    May 2005
    Location
    Kansas City
    Posts
    9
    Originally posted by WII-Aaron
    Hello All,

    Here is a brief synopsis of the issue so far.

    The building we are in is undergoing some renovations. One of the contractors didn't plug a storm drain they had cut from the roof. It's been raining here for several days. The storm drain has been emptying into the basement and sub-basement.

    At this point the sub basement in the building is completely under water, to the point where the basement above it has an inch of standing water on the floor. Power comes into the building in the sub basement. All the transformers and power distribution hardware is under water. Our generator is also in the sub basement along with several other providers.

    We are waiting for some backup generators to arrive at 6am (central). This will restore limited service. The power systems in the building are trashed and may take days or weeks to fully restore.

    We are somewhat fortunate in that our transformers are on the 9th floor so as soon as the fire department gets the basement pumped out it should just be a matter of KCP&L flipping our cutover switch and we'll be back up.

    At this point I don't have an ETA. Williams, Cogent, Qwest, Expedius, Broadwing, MCI, SBC.... etc etc etc are all in the same boat and thier NOC techs are all on site as well.

    I will update you again when I am able.

    Aaron
    Ok, I have to say I find this quite funny, and quite the oppsite of what happen.

    Your 'generator' is in the sub-basement along with everyone elses? Excuse me, but NO ONE has a generator in the sub basement.

    Your stepdown transformers are in your space, they step-down 13,200 volts to 110/208 volts. So you have a generator putting out 13,200 volts in the sub-basement?

    I work for one of 5 companies that have perm generators installed. MCI, WilTel, Alltel, SBC, and KCNAP. (I'm with KCNAP)

    MNA, Broadwing, Cogent, and several others didn't need to have generators on site, because they use DC plants powered by several battary packs. Cogent has a small pack, and they last 12 hours.

    I personally helped K&W Underground hook up Cogent's generator, and talked with the site tech who lives out in Denver. I watched 2 other companies hook theirs in as well. The building dosn't have a generator, and won't have one for another 6~9 months.

    What did I see from you... Standing around for 3~4 hours with a dumb look on your face.

    You know what pisses me off the most (and the whole reason why I signed up here) is one simple fact.

    You haven't publicly thanked MCI for letting you run extention cords down the stairway from the 9th floor, to the 7th floor to gain access to some power. Instead your saying that you got some generator of yours up and running for some of your customers. Probably some of your customers that were comming to me asking if they could move their equipment to me because I had power.

    What I'm going to do tomorrow is print out this thread, and hand it to MCI so that they can see how thankful you were for them helping you at no cost.

    KCNAP had a diaster plan, and when it came to needing it, it worked flawlessly. KCNAP's network didn't blink, with the exception of some customers in the building that didn't have anything more than UPS power.

    The power went down in the building at 22:57 CDT, and returned at 05:05 CDT.

    The fire department told me they wouldn't pump out the water, and you just happen to be standing over in the corner evesdropping. I'm the one that took them to the sub-basement in hopes they could be of some help. Due to the high voltage, saftey wouldn't let them.

    One other thing for your customers, what's your current plan when KCPL turns off the old-substation in the sub-basement and switches to the new one they are currently building on the first floor? You know there will be a power outage of several hours. They are also only providing 480 volts up the building, and your transformers have a 13.2k volt input.

    KCNAP's generators are powered by natrual gas, so we don't have to worry about fuel.

    At least James had enough gumpsion to help out Duke, the building engineer, by carrying out buckets that water was falling into.
    Attached Images Attached Images
      0 Not allowed!

  34. #34
    Join Date
    May 2005
    Posts
    1
    I was down at the Bryant Building when this happened with Shawn. I am collocated in the KCNAP and I can testify that nothing in the NAP blinked when the power went out. I have pictures to PROVE that Wholesale Internet DOES NOT have any generator in the sub-basement let alone ANY generator that they own.

    I will post pictures of the SUB-Basement, and the generators that belong to WilTel, MCI, and Cogent.. not WII.

    Putting the smack down on BS..
      0 Not allowed!

  35. #35
    Join Date
    Dec 2003
    Location
    Boston, MA
    Posts
    603
    People really need to have disaster plans and invest this money to make them work and not take for granted and lie when someone else assists you. If I had space in that building and MCI was helping me out I would sure post a thread “Thanks to MCI Major PROPS”.


    Good Job KCNAP
      0 Not allowed!

  36. #36
    Join Date
    May 2005
    Posts
    1
    Here's the pipe that was cut:

    http://xero3.net/pics/IMG_1442.JPG
    http://xero3.net/pics/IMG_1443.JPG

    Hmm.. here's the sub basement where KCP&L comes into the building.. I see water but no generator.. I smell BS.

    http://xero3.net/pics/IMG_1444.JPG
    http://xero3.net/pics/IMG_1445.JPG

    Here's MCI, WilTel, and Cogent Generators showing up:

    http://xero3.net/pics/IMG_1449.JPG
    http://xero3.net/pics/IMG_1452.JPG
    http://xero3.net/pics/IMG_1453.JPG

    Anyone else feel they deserve the truth?
      0 Not allowed!

  37. #37
    Join Date
    May 2005
    Location
    Kansas City
    Posts
    9

    btw...

    edge-01.kc.kcnap.net# sh ip bgp sum
    BGP router identifier 66.39.192.241, local AS number 31985
    54878 BGP AS-PATH entries
    7 BGP community entries

    Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
    65.77.99.x 4 7911 3616525 126014 0 0 0 12w3d10h 158665
    66.39.192.x 4 31985 203623 5294898 0 0 0 10:35:37 97
    66.39.192.x 4 25994 124915 4337204 0 0 0 2d23h05m 18
    206.165.102.x 4 31932 4836335 126001 0 0 0 12w3d10h 158725


    This is from just one edge router...
      0 Not allowed!

  38. #38
    Join Date
    Aug 2003
    Posts
    32
    Having an outage of anytime is not a fun thing to deal with. It can dumbfound you for a while. The one thing that we know is that this outage could have been prevented if the demolition contractor would have used his head.

    Why did this outage take so long to resolve?

    Kansas City Power and light had over 1200 customers out in Kansas City. The Bryant building called KCPL right after their service was interrupted. For some reason KCPL decided that their residential customers needed their power on first. The first tech that arrived from KCPL was not sure about correctly operating the 1929 power substation in the subbasement of the Bryant building. So he decided to wait until a senior tech could arrive. Once the senior tech arrived the Bryant Building was reconnected to the KCPL Network.

    The Bryant building rides the power grid called the KCPL Network. This grid rarely fails as it houses Major Telecommunication’s companies and large banks. It also houses the main CO for SBC in Kansas City.


    Why doesn’t my host have a generator on site?

    Most of the companies in the Bryant building have offsite generators that are deployed within 2 to 6 hours. We store our generators in Harrisonville. Why because the Bryant Building is currently out of space for new generators. Also the NEW building owners have assured all the tenants that the building will have a new building wide generator by the year end. (I think they said that in 2004) We are still waiting for them to install the new generators and power substation.

    As for having generators on site they can still fail. Some months ago Switch & Data across the street from the Bryant Building had a 6 + hour outage because their generator would not start after their battery backup system failed.

    The one main thing that this outage did was bring this building together as a whole. Many large companies and small companies worked together to resolve power issues throughout the whole building.
      0 Not allowed!

  39. #39
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    wow. I'm almost at a loss of words ...

    guess things don't change.

    all I have to say to a few of you is GROW UP.

    didn't your mom ever teach you that if you didn't have anything nice to say not to say anything at all?

    it's almost sad that you feel the need to put other people/companies down to try and somehow benefit yourself. just a newsflash for you guys, it's not working. all I see is COMPLIMENTS from that entity that you're trying to put down or prove wrong's CUSTOMERS. that has to say something doesn't it?

    I'm sure they could fire back with just as much dirt as you *think* you have. but they don't. why? probably because they have a maturity level greater than a 12 year old ...

    just my two cents though.
      0 Not allowed!

  40. #40
    Join Date
    May 2005
    Location
    Kansas City
    Posts
    0
    Originally posted by omaha.stylee
    wow. I'm almost at a loss of words ...

    guess things don't change.

    all I have to say to a few of you is GROW UP.

    didn't your mom ever teach you that if you didn't have anything nice to say not to say anything at all?

    it's almost sad that you feel the need to put other people/companies down to try and somehow benefit yourself. just a newsflash for you guys, it's not working. all I see is COMPLIMENTS from that entity that you're trying to put down or prove wrong's CUSTOMERS. that has to say something doesn't it?

    I'm sure they could fire back with just as much dirt as you *think* you have. but they don't. why? probably because they have a maturity level greater than a 12 year old ...

    just my two cents though.
    I don't see anyone deliberateily "putting down" anyone else here. It looks like a straightforward fact checking session. On another note, I can too attest to the fact that KCNAP has a backup plan. I do contract work for a KCNAP downstream and aside from his power issues, KCNAP was not down at all.
    Last edited by mnicks; 05-16-2005 at 09:22 AM.
      0 Not allowed!

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •