Page 2 of 3 FirstFirst 123 LastLast
Results 26 to 50 of 59
  1. #26
    Join Date
    Aug 2002
    Location
    Kansas City. MO
    Posts
    1
    Originally posted by WII-Aaron
    ... One of the contractors didn't plug a storm drain they had cut from the roof. It's been raining here for several days. The storm drain has been emptying into the basement and sub-basement.

    At this point the sub basement in the building is completely under water, to the point where the basement above it has an inch of standing water on the floor. Power comes into the building in the sub basement. All the transformers and power distribution hardware is under water. Our generator is also in the sub basement along with several other providers.

    ...
    Man, I'd hate to be that contractor this morning.
      0 Not allowed!

  2. #27
    Join Date
    Mar 2004
    Location
    Bristol, UK
    Posts
    232
    My server has just come back online and looking at uptime it has been powered on for over 2 hours so it looks like the switches are being brought up after the servers are getting power.
    Good effort to all involved at Wholesale Internet.
      0 Not allowed!

  3. #28
    Join Date
    Nov 2004
    Location
    Toronto, Canada
    Posts
    164
    My server at WSI says it been up 6 hours, but the site was still not working checked the dns report sure enough lame nameservers. Configured the DNS Administration again and "Bingo" she's lit up.

    Good job Aaron and James.
      0 Not allowed!

  4. #29
    Join Date
    Mar 2004
    Location
    Edmonton, Canada
    Posts
    339
    Thanks for the update Aaron, I for one as a Customer totally understand. You guys have been good to us, and this I can live with
      0 Not allowed!

  5. #30
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    ha. ****** day to be in the bryant building ...

    even bigger providers suffered outages due to their generators being under water from what I've been told.

    somebody's going to be getting sued. heh
      0 Not allowed!

  6. #31
    Join Date
    Aug 2003
    Location
    PA
    Posts
    1,914
    even bigger providers suffered outages due to their generators being under water from what I've been told.

    somebody's going to be getting sued. heh
    Same thing I heard off the WSI person I spoke to on the phone. Wouldn't be suprised if someone gets into serious **** over this mess. Hopefully all the mains can be restored soon and there wasn't too much damage caused.

    Good job WSI at restoring this mess back to working condition again.



    -Justin
      0 Not allowed!

  7. #32
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    heh.

    gotta love all the *'s in the last two posts.
      0 Not allowed!

  8. #33
    Join Date
    May 2005
    Location
    Kansas City
    Posts
    9
    Originally posted by WII-Aaron
    Hello All,

    Here is a brief synopsis of the issue so far.

    The building we are in is undergoing some renovations. One of the contractors didn't plug a storm drain they had cut from the roof. It's been raining here for several days. The storm drain has been emptying into the basement and sub-basement.

    At this point the sub basement in the building is completely under water, to the point where the basement above it has an inch of standing water on the floor. Power comes into the building in the sub basement. All the transformers and power distribution hardware is under water. Our generator is also in the sub basement along with several other providers.

    We are waiting for some backup generators to arrive at 6am (central). This will restore limited service. The power systems in the building are trashed and may take days or weeks to fully restore.

    We are somewhat fortunate in that our transformers are on the 9th floor so as soon as the fire department gets the basement pumped out it should just be a matter of KCP&L flipping our cutover switch and we'll be back up.

    At this point I don't have an ETA. Williams, Cogent, Qwest, Expedius, Broadwing, MCI, SBC.... etc etc etc are all in the same boat and thier NOC techs are all on site as well.

    I will update you again when I am able.

    Aaron
    Ok, I have to say I find this quite funny, and quite the oppsite of what happen.

    Your 'generator' is in the sub-basement along with everyone elses? Excuse me, but NO ONE has a generator in the sub basement.

    Your stepdown transformers are in your space, they step-down 13,200 volts to 110/208 volts. So you have a generator putting out 13,200 volts in the sub-basement?

    I work for one of 5 companies that have perm generators installed. MCI, WilTel, Alltel, SBC, and KCNAP. (I'm with KCNAP)

    MNA, Broadwing, Cogent, and several others didn't need to have generators on site, because they use DC plants powered by several battary packs. Cogent has a small pack, and they last 12 hours.

    I personally helped K&W Underground hook up Cogent's generator, and talked with the site tech who lives out in Denver. I watched 2 other companies hook theirs in as well. The building dosn't have a generator, and won't have one for another 6~9 months.

    What did I see from you... Standing around for 3~4 hours with a dumb look on your face.

    You know what pisses me off the most (and the whole reason why I signed up here) is one simple fact.

    You haven't publicly thanked MCI for letting you run extention cords down the stairway from the 9th floor, to the 7th floor to gain access to some power. Instead your saying that you got some generator of yours up and running for some of your customers. Probably some of your customers that were comming to me asking if they could move their equipment to me because I had power.

    What I'm going to do tomorrow is print out this thread, and hand it to MCI so that they can see how thankful you were for them helping you at no cost.

    KCNAP had a diaster plan, and when it came to needing it, it worked flawlessly. KCNAP's network didn't blink, with the exception of some customers in the building that didn't have anything more than UPS power.

    The power went down in the building at 22:57 CDT, and returned at 05:05 CDT.

    The fire department told me they wouldn't pump out the water, and you just happen to be standing over in the corner evesdropping. I'm the one that took them to the sub-basement in hopes they could be of some help. Due to the high voltage, saftey wouldn't let them.

    One other thing for your customers, what's your current plan when KCPL turns off the old-substation in the sub-basement and switches to the new one they are currently building on the first floor? You know there will be a power outage of several hours. They are also only providing 480 volts up the building, and your transformers have a 13.2k volt input.

    KCNAP's generators are powered by natrual gas, so we don't have to worry about fuel.

    At least James had enough gumpsion to help out Duke, the building engineer, by carrying out buckets that water was falling into.
    Attached Images Attached Images
      0 Not allowed!

  9. #34
    Join Date
    May 2005
    Posts
    1
    I was down at the Bryant Building when this happened with Shawn. I am collocated in the KCNAP and I can testify that nothing in the NAP blinked when the power went out. I have pictures to PROVE that Wholesale Internet DOES NOT have any generator in the sub-basement let alone ANY generator that they own.

    I will post pictures of the SUB-Basement, and the generators that belong to WilTel, MCI, and Cogent.. not WII.

    Putting the smack down on BS..
      0 Not allowed!

  10. #35
    Join Date
    Dec 2003
    Location
    Boston, MA
    Posts
    608
    People really need to have disaster plans and invest this money to make them work and not take for granted and lie when someone else assists you. If I had space in that building and MCI was helping me out I would sure post a thread “Thanks to MCI Major PROPS”.


    Good Job KCNAP
      0 Not allowed!

  11. #36
    Join Date
    May 2005
    Posts
    1
    Here's the pipe that was cut:

    http://xero3.net/pics/IMG_1442.JPG
    http://xero3.net/pics/IMG_1443.JPG

    Hmm.. here's the sub basement where KCP&L comes into the building.. I see water but no generator.. I smell BS.

    http://xero3.net/pics/IMG_1444.JPG
    http://xero3.net/pics/IMG_1445.JPG

    Here's MCI, WilTel, and Cogent Generators showing up:

    http://xero3.net/pics/IMG_1449.JPG
    http://xero3.net/pics/IMG_1452.JPG
    http://xero3.net/pics/IMG_1453.JPG

    Anyone else feel they deserve the truth?
      0 Not allowed!

  12. #37
    Join Date
    May 2005
    Location
    Kansas City
    Posts
    9

    btw...

    edge-01.kc.kcnap.net# sh ip bgp sum
    BGP router identifier 66.39.192.241, local AS number 31985
    54878 BGP AS-PATH entries
    7 BGP community entries

    Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
    65.77.99.x 4 7911 3616525 126014 0 0 0 12w3d10h 158665
    66.39.192.x 4 31985 203623 5294898 0 0 0 10:35:37 97
    66.39.192.x 4 25994 124915 4337204 0 0 0 2d23h05m 18
    206.165.102.x 4 31932 4836335 126001 0 0 0 12w3d10h 158725


    This is from just one edge router...
      0 Not allowed!

  13. #38
    Join Date
    Aug 2003
    Posts
    32
    Having an outage of anytime is not a fun thing to deal with. It can dumbfound you for a while. The one thing that we know is that this outage could have been prevented if the demolition contractor would have used his head.

    Why did this outage take so long to resolve?

    Kansas City Power and light had over 1200 customers out in Kansas City. The Bryant building called KCPL right after their service was interrupted. For some reason KCPL decided that their residential customers needed their power on first. The first tech that arrived from KCPL was not sure about correctly operating the 1929 power substation in the subbasement of the Bryant building. So he decided to wait until a senior tech could arrive. Once the senior tech arrived the Bryant Building was reconnected to the KCPL Network.

    The Bryant building rides the power grid called the KCPL Network. This grid rarely fails as it houses Major Telecommunication’s companies and large banks. It also houses the main CO for SBC in Kansas City.


    Why doesn’t my host have a generator on site?

    Most of the companies in the Bryant building have offsite generators that are deployed within 2 to 6 hours. We store our generators in Harrisonville. Why because the Bryant Building is currently out of space for new generators. Also the NEW building owners have assured all the tenants that the building will have a new building wide generator by the year end. (I think they said that in 2004) We are still waiting for them to install the new generators and power substation.

    As for having generators on site they can still fail. Some months ago Switch & Data across the street from the Bryant Building had a 6 + hour outage because their generator would not start after their battery backup system failed.

    The one main thing that this outage did was bring this building together as a whole. Many large companies and small companies worked together to resolve power issues throughout the whole building.
      0 Not allowed!

  14. #39
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    wow. I'm almost at a loss of words ...

    guess things don't change.

    all I have to say to a few of you is GROW UP.

    didn't your mom ever teach you that if you didn't have anything nice to say not to say anything at all?

    it's almost sad that you feel the need to put other people/companies down to try and somehow benefit yourself. just a newsflash for you guys, it's not working. all I see is COMPLIMENTS from that entity that you're trying to put down or prove wrong's CUSTOMERS. that has to say something doesn't it?

    I'm sure they could fire back with just as much dirt as you *think* you have. but they don't. why? probably because they have a maturity level greater than a 12 year old ...

    just my two cents though.
      0 Not allowed!

  15. #40
    Join Date
    May 2005
    Location
    Kansas City
    Posts
    0
    Originally posted by omaha.stylee
    wow. I'm almost at a loss of words ...

    guess things don't change.

    all I have to say to a few of you is GROW UP.

    didn't your mom ever teach you that if you didn't have anything nice to say not to say anything at all?

    it's almost sad that you feel the need to put other people/companies down to try and somehow benefit yourself. just a newsflash for you guys, it's not working. all I see is COMPLIMENTS from that entity that you're trying to put down or prove wrong's CUSTOMERS. that has to say something doesn't it?

    I'm sure they could fire back with just as much dirt as you *think* you have. but they don't. why? probably because they have a maturity level greater than a 12 year old ...

    just my two cents though.
    I don't see anyone deliberateily "putting down" anyone else here. It looks like a straightforward fact checking session. On another note, I can too attest to the fact that KCNAP has a backup plan. I do contract work for a KCNAP downstream and aside from his power issues, KCNAP was not down at all.
    Last edited by mnicks; 05-16-2005 at 09:22 AM.
      0 Not allowed!

  16. #41
    Wow omaha.stylee, I was wondering last night if I was the only one thinking the guys from KCNAP needed to grow up. You can prove a company wrong, by not coming off like a completely arrogant person.

    Are we supposed to be impressed the KCNAP space didn’t lose power?
      0 Not allowed!

  17. #42
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    *gasp* shawnm ... arrogant? ... NEVER!!


    TRUST ME, there is nothing to be impressed by. AT ALL.

    On another note, I can too attest to the fact that KCNAP has a backup plan.
    what's their backup plan for their "router room" full of FreeBSD routers when it gets up to 90+ degrees?
      0 Not allowed!

  18. #43
    Join Date
    May 2005
    Location
    Kansas City
    Posts
    9
    I'm not trying to impress anyone, only bring attetion to the fact that wholesale internet dosn't have things that they claim to.

    What happens if the room get's up to 90+ is easy. Environmental alarms go off well before then and notify 4 diffrent people. If the HVAC units are down, there's large 42" fan's that move 20,000 CFM that are pulled in from storage on the same floor.

    Before you go knocking FreeBSD as a router, keep in mind Juniper networks uses FreeBSD on their routers.

    KCNAP was one of several companies that didn't loose power, Wholesale internet was the one begging MCI for some power, then didn't thank them, but instead spew out more lies.
      0 Not allowed!

  19. #44
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    attetion? not too sure what that is.

    juniper may use FreeBSD, but it's used on a TRUE router, not sure that you'd call p3 dell boxes TRUE routers.

    maybe you don't get it, but people really don't care that kcnap didn't lose power.

    and in all reality, who cares? because you all suck.


    grow up shawnm.
      0 Not allowed!

  20. #45
    Join Date
    Dec 2002
    Location
    acton, MASS
    Posts
    229
    Originally posted by omaha.stylee
    *gasp* shawnm ... arrogant? ... NEVER!!


    TRUST ME, there is nothing to be impressed by. AT ALL.



    what's their backup plan for their "router room" full of FreeBSD routers when it gets up to 90+ degrees?

    A router is nothing more than a device that performs longest match lookup and forward packet to egress interface.

    Before you bash on FreeBSD or Linux based routers, get a clue and realize that in this business, nothing matters but the Bottom Line. If you can maintain stability and uptime using FreeBSD routers and have sufficient clue to operate them (while more difficult to do so than say Cisco, et al), then you have a lower opex.

    -J
      0 Not allowed!

  21. #46
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    but on a p3?

    what's the bus speed on most p3's ...

    I know for a fact it's not fast enough to do any SUBSTANTIAL traffic without saturating the pci bus.

    I'm not going to sit here and tell you that I know everything that there is to know about this stuff. but I do have somewhat of a clue and lots of smart friends, and that seems to help my ignorance out a bit.
      0 Not allowed!

  22. #47
    Join Date
    Dec 2002
    Location
    acton, MASS
    Posts
    229
    Originally posted by omaha.stylee
    but on a p3?

    what's the bus speed on most p3's ...

    I know for a fact it's not fast enough to do any SUBSTANTIAL traffic without saturating the pci bus.

    I'm not going to sit here and tell you that I know everything that there is to know about this stuff. but I do have somewhat of a clue and lots of smart friends, and that seems to help my ignorance out a bit.
    Substantial traffic?

    If my edge router with 3x100Mb interfaces can route them fine, why would I care when the device is routing the circuits I'm throwing at?

    -J
      0 Not allowed!

  23. #48
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    sure.

    but you need minimum of 2ghz to barely do things proper with pc hardware. and what i mean by 'proper', is doing gig-e fully saturated with small 64 byte packets at 1.4 million pps

    for every packet, the router has to lookup forwarding table to decide where to route out, and freebsd/linux use patricia trie algorithm which is quite inefficient... so it takes a HUGE toll on CPU.

    but you I'm sure you know all this. right?
      0 Not allowed!

  24. #49
    Join Date
    Dec 2002
    Location
    acton, MASS
    Posts
    229
    Originally posted by omaha.stylee
    sure.

    but you need minimum of 2ghz to barely do things proper with pc hardware. and what i mean by 'proper', is doing gig-e fully saturated with small 64 byte packets at 1.4 million pps
    Last I heard from some people around here, KCNAP does not have gig-e circuits, but 100Mb interfaces. P3 CPUs are far enough to route 100Mb interfaces at linerate even with PATRICIA/RADIX trie lookups taxing at 140kpps, if you tune it properly.


    for every packet, the router has to lookup forwarding table to decide where to route out, and freebsd/linux use patricia trie algorithm which is quite inefficient... so it takes a HUGE toll on CPU.

    but you I'm sure you know all this. right?
    Tests have shown radix trie lookups don't take huge enough toll on CPU to kill a P3 processor on 100Mb circuit, especially on IPv4 32-bit lookups.

    -J
      0 Not allowed!

  25. #50
    Join Date
    May 2004
    Location
    Blue Springs, Missouri
    Posts
    366
    heh. another one of shawnm's friends?

    pretty sure all this has nothing to do with wholesaleinternet being down. tsk tsk. shame on all us!!

    but I'm done with this, turned into a flame war. when I could care less about you people. hope you all go far!!

    cheers!!
      0 Not allowed!

Page 2 of 3 FirstFirst 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •