Results 26 to 50 of 59
Thread: Wholesale Internet Down?
-
05-13-2005, 08:42 AM #26New Member
- Join Date
- Aug 2002
- Location
- Kansas City. MO
- Posts
- 1
Originally posted by WII-Aaron
... One of the contractors didn't plug a storm drain they had cut from the roof. It's been raining here for several days. The storm drain has been emptying into the basement and sub-basement.
At this point the sub basement in the building is completely under water, to the point where the basement above it has an inch of standing water on the floor. Power comes into the building in the sub basement. All the transformers and power distribution hardware is under water. Our generator is also in the sub basement along with several other providers.
...0
-
05-13-2005, 09:11 AM #27Junior Guru
- Join Date
- Mar 2004
- Location
- Bristol, UK
- Posts
- 232
My server has just come back online and looking at uptime it has been powered on for over 2 hours so it looks like the switches are being brought up after the servers are getting power.
Good effort to all involved at Wholesale Internet.0
-
05-13-2005, 12:22 PM #28WHT Addict
- Join Date
- Nov 2004
- Location
- Toronto, Canada
- Posts
- 164
My server at WSI says it been up 6 hours, but the site was still not working checked the dns report sure enough lame nameservers. Configured the DNS Administration again and "Bingo" she's lit up.
Good job Aaron and James.0
-
05-13-2005, 06:59 PM #29Web Hosting Guru
- Join Date
- Mar 2004
- Location
- Edmonton, Canada
- Posts
- 339
Thanks for the update Aaron, I for one as a Customer totally understand. You guys have been good to us, and this I can live with
0
-
05-13-2005, 08:22 PM #30Aspiring Evangelist
- Join Date
- May 2004
- Location
- Blue Springs, Missouri
- Posts
- 366
ha. ****** day to be in the bryant building ...
even bigger providers suffered outages due to their generators being under water from what I've been told.
somebody's going to be getting sued. heh0
-
05-13-2005, 11:59 PM #31Retired Moderator
- Join Date
- Aug 2003
- Location
- PA
- Posts
- 1,914
even bigger providers suffered outages due to their generators being under water from what I've been told.
somebody's going to be getting sued. heh
Good job WSI at restoring this mess back to working condition again.
-Justin0
-
05-15-2005, 03:32 AM #32Aspiring Evangelist
- Join Date
- May 2004
- Location
- Blue Springs, Missouri
- Posts
- 366
heh.
gotta love all the *'s in the last two posts.0
-
05-15-2005, 11:04 PM #33Newbie
- Join Date
- May 2005
- Location
- Kansas City
- Posts
- 9
Originally posted by WII-Aaron
Hello All,
Here is a brief synopsis of the issue so far.
The building we are in is undergoing some renovations. One of the contractors didn't plug a storm drain they had cut from the roof. It's been raining here for several days. The storm drain has been emptying into the basement and sub-basement.
At this point the sub basement in the building is completely under water, to the point where the basement above it has an inch of standing water on the floor. Power comes into the building in the sub basement. All the transformers and power distribution hardware is under water. Our generator is also in the sub basement along with several other providers.
We are waiting for some backup generators to arrive at 6am (central). This will restore limited service. The power systems in the building are trashed and may take days or weeks to fully restore.
We are somewhat fortunate in that our transformers are on the 9th floor so as soon as the fire department gets the basement pumped out it should just be a matter of KCP&L flipping our cutover switch and we'll be back up.
At this point I don't have an ETA. Williams, Cogent, Qwest, Expedius, Broadwing, MCI, SBC.... etc etc etc are all in the same boat and thier NOC techs are all on site as well.
I will update you again when I am able.
Aaron
Your 'generator' is in the sub-basement along with everyone elses? Excuse me, but NO ONE has a generator in the sub basement.
Your stepdown transformers are in your space, they step-down 13,200 volts to 110/208 volts. So you have a generator putting out 13,200 volts in the sub-basement?
I work for one of 5 companies that have perm generators installed. MCI, WilTel, Alltel, SBC, and KCNAP. (I'm with KCNAP)
MNA, Broadwing, Cogent, and several others didn't need to have generators on site, because they use DC plants powered by several battary packs. Cogent has a small pack, and they last 12 hours.
I personally helped K&W Underground hook up Cogent's generator, and talked with the site tech who lives out in Denver. I watched 2 other companies hook theirs in as well. The building dosn't have a generator, and won't have one for another 6~9 months.
What did I see from you... Standing around for 3~4 hours with a dumb look on your face.
You know what pisses me off the most (and the whole reason why I signed up here) is one simple fact.
You haven't publicly thanked MCI for letting you run extention cords down the stairway from the 9th floor, to the 7th floor to gain access to some power. Instead your saying that you got some generator of yours up and running for some of your customers. Probably some of your customers that were comming to me asking if they could move their equipment to me because I had power.
What I'm going to do tomorrow is print out this thread, and hand it to MCI so that they can see how thankful you were for them helping you at no cost.
KCNAP had a diaster plan, and when it came to needing it, it worked flawlessly. KCNAP's network didn't blink, with the exception of some customers in the building that didn't have anything more than UPS power.
The power went down in the building at 22:57 CDT, and returned at 05:05 CDT.
The fire department told me they wouldn't pump out the water, and you just happen to be standing over in the corner evesdropping. I'm the one that took them to the sub-basement in hopes they could be of some help. Due to the high voltage, saftey wouldn't let them.
One other thing for your customers, what's your current plan when KCPL turns off the old-substation in the sub-basement and switches to the new one they are currently building on the first floor? You know there will be a power outage of several hours. They are also only providing 480 volts up the building, and your transformers have a 13.2k volt input.
KCNAP's generators are powered by natrual gas, so we don't have to worry about fuel.
At least James had enough gumpsion to help out Duke, the building engineer, by carrying out buckets that water was falling into.0
-
05-15-2005, 11:34 PM #34Temporarily Suspended
- Join Date
- May 2005
- Posts
- 1
I was down at the Bryant Building when this happened with Shawn. I am collocated in the KCNAP and I can testify that nothing in the NAP blinked when the power went out. I have pictures to PROVE that Wholesale Internet DOES NOT have any generator in the sub-basement let alone ANY generator that they own.
I will post pictures of the SUB-Basement, and the generators that belong to WilTel, MCI, and Cogent.. not WII.
Putting the smack down on BS..0
-
05-15-2005, 11:55 PM #35Hosting Specialist
- Join Date
- Dec 2003
- Location
- Boston, MA
- Posts
- 608
People really need to have disaster plans and invest this money to make them work and not take for granted and lie when someone else assists you. If I had space in that building and MCI was helping me out I would sure post a thread “Thanks to MCI Major PROPS”.
Good Job KCNAP0
-
05-16-2005, 12:10 AM #36Temporarily Suspended
- Join Date
- May 2005
- Posts
- 1
Here's the pipe that was cut:
http://xero3.net/pics/IMG_1442.JPG
http://xero3.net/pics/IMG_1443.JPG
Hmm.. here's the sub basement where KCP&L comes into the building.. I see water but no generator.. I smell BS.
http://xero3.net/pics/IMG_1444.JPG
http://xero3.net/pics/IMG_1445.JPG
Here's MCI, WilTel, and Cogent Generators showing up:
http://xero3.net/pics/IMG_1449.JPG
http://xero3.net/pics/IMG_1452.JPG
http://xero3.net/pics/IMG_1453.JPG
Anyone else feel they deserve the truth?0
-
05-16-2005, 01:01 AM #37Newbie
- Join Date
- May 2005
- Location
- Kansas City
- Posts
- 9
btw...
edge-01.kc.kcnap.net# sh ip bgp sum
BGP router identifier 66.39.192.241, local AS number 31985
54878 BGP AS-PATH entries
7 BGP community entries
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
65.77.99.x 4 7911 3616525 126014 0 0 0 12w3d10h 158665
66.39.192.x 4 31985 203623 5294898 0 0 0 10:35:37 97
66.39.192.x 4 25994 124915 4337204 0 0 0 2d23h05m 18
206.165.102.x 4 31932 4836335 126001 0 0 0 12w3d10h 158725
This is from just one edge router...0
-
05-16-2005, 01:27 AM #38Junior Guru Wannabe
- Join Date
- Aug 2003
- Posts
- 32
Having an outage of anytime is not a fun thing to deal with. It can dumbfound you for a while. The one thing that we know is that this outage could have been prevented if the demolition contractor would have used his head.
Why did this outage take so long to resolve?
Kansas City Power and light had over 1200 customers out in Kansas City. The Bryant building called KCPL right after their service was interrupted. For some reason KCPL decided that their residential customers needed their power on first. The first tech that arrived from KCPL was not sure about correctly operating the 1929 power substation in the subbasement of the Bryant building. So he decided to wait until a senior tech could arrive. Once the senior tech arrived the Bryant Building was reconnected to the KCPL Network.
The Bryant building rides the power grid called the KCPL Network. This grid rarely fails as it houses Major Telecommunication’s companies and large banks. It also houses the main CO for SBC in Kansas City.
Why doesn’t my host have a generator on site?
Most of the companies in the Bryant building have offsite generators that are deployed within 2 to 6 hours. We store our generators in Harrisonville. Why because the Bryant Building is currently out of space for new generators. Also the NEW building owners have assured all the tenants that the building will have a new building wide generator by the year end. (I think they said that in 2004) We are still waiting for them to install the new generators and power substation.
As for having generators on site they can still fail. Some months ago Switch & Data across the street from the Bryant Building had a 6 + hour outage because their generator would not start after their battery backup system failed.
The one main thing that this outage did was bring this building together as a whole. Many large companies and small companies worked together to resolve power issues throughout the whole building.0
-
05-16-2005, 03:31 AM #39Aspiring Evangelist
- Join Date
- May 2004
- Location
- Blue Springs, Missouri
- Posts
- 366
wow. I'm almost at a loss of words ...
guess things don't change.
all I have to say to a few of you is GROW UP.
didn't your mom ever teach you that if you didn't have anything nice to say not to say anything at all?
it's almost sad that you feel the need to put other people/companies down to try and somehow benefit yourself. just a newsflash for you guys, it's not working. all I see is COMPLIMENTS from that entity that you're trying to put down or prove wrong's CUSTOMERS. that has to say something doesn't it?
I'm sure they could fire back with just as much dirt as you *think* you have. but they don't. why? probably because they have a maturity level greater than a 12 year old ...
just my two cents though.0
-
05-16-2005, 09:18 AM #40New Member
- Join Date
- May 2005
- Location
- Kansas City
- Posts
- 0
Originally posted by omaha.stylee
wow. I'm almost at a loss of words ...
guess things don't change.
all I have to say to a few of you is GROW UP.
didn't your mom ever teach you that if you didn't have anything nice to say not to say anything at all?
it's almost sad that you feel the need to put other people/companies down to try and somehow benefit yourself. just a newsflash for you guys, it's not working. all I see is COMPLIMENTS from that entity that you're trying to put down or prove wrong's CUSTOMERS. that has to say something doesn't it?
I'm sure they could fire back with just as much dirt as you *think* you have. but they don't. why? probably because they have a maturity level greater than a 12 year old ...
just my two cents though.Last edited by mnicks; 05-16-2005 at 09:22 AM.
0
-
05-16-2005, 11:59 AM #41Aspiring Evangelist
- Join Date
- Aug 2003
- Posts
- 380
Wow omaha.stylee, I was wondering last night if I was the only one thinking the guys from KCNAP needed to grow up. You can prove a company wrong, by not coming off like a completely arrogant person.
Are we supposed to be impressed the KCNAP space didn’t lose power?0
-
05-16-2005, 01:35 PM #42Aspiring Evangelist
- Join Date
- May 2004
- Location
- Blue Springs, Missouri
- Posts
- 366
*gasp* shawnm ... arrogant? ... NEVER!!
TRUST ME, there is nothing to be impressed by. AT ALL.
On another note, I can too attest to the fact that KCNAP has a backup plan.0
-
05-16-2005, 01:50 PM #43Newbie
- Join Date
- May 2005
- Location
- Kansas City
- Posts
- 9
I'm not trying to impress anyone, only bring attetion to the fact that wholesale internet dosn't have things that they claim to.
What happens if the room get's up to 90+ is easy. Environmental alarms go off well before then and notify 4 diffrent people. If the HVAC units are down, there's large 42" fan's that move 20,000 CFM that are pulled in from storage on the same floor.
Before you go knocking FreeBSD as a router, keep in mind Juniper networks uses FreeBSD on their routers.
KCNAP was one of several companies that didn't loose power, Wholesale internet was the one begging MCI for some power, then didn't thank them, but instead spew out more lies.0
-
05-16-2005, 01:56 PM #44Aspiring Evangelist
- Join Date
- May 2004
- Location
- Blue Springs, Missouri
- Posts
- 366
attetion? not too sure what that is.
juniper may use FreeBSD, but it's used on a TRUE router, not sure that you'd call p3 dell boxes TRUE routers.
maybe you don't get it, but people really don't care that kcnap didn't lose power.
and in all reality, who cares? because you all suck.
grow up shawnm.0
-
05-16-2005, 01:58 PM #45Junior Guru
- Join Date
- Dec 2002
- Location
- acton, MASS
- Posts
- 229
Originally posted by omaha.stylee
*gasp* shawnm ... arrogant? ... NEVER!!
TRUST ME, there is nothing to be impressed by. AT ALL.
what's their backup plan for their "router room" full of FreeBSD routers when it gets up to 90+ degrees?
A router is nothing more than a device that performs longest match lookup and forward packet to egress interface.
Before you bash on FreeBSD or Linux based routers, get a clue and realize that in this business, nothing matters but the Bottom Line. If you can maintain stability and uptime using FreeBSD routers and have sufficient clue to operate them (while more difficult to do so than say Cisco, et al), then you have a lower opex.
-J0
-
05-16-2005, 02:15 PM #46Aspiring Evangelist
- Join Date
- May 2004
- Location
- Blue Springs, Missouri
- Posts
- 366
but on a p3?
what's the bus speed on most p3's ...
I know for a fact it's not fast enough to do any SUBSTANTIAL traffic without saturating the pci bus.
I'm not going to sit here and tell you that I know everything that there is to know about this stuff. but I do have somewhat of a clue and lots of smart friends, and that seems to help my ignorance out a bit.0
-
05-16-2005, 02:19 PM #47Junior Guru
- Join Date
- Dec 2002
- Location
- acton, MASS
- Posts
- 229
Originally posted by omaha.stylee
but on a p3?
what's the bus speed on most p3's ...
I know for a fact it's not fast enough to do any SUBSTANTIAL traffic without saturating the pci bus.
I'm not going to sit here and tell you that I know everything that there is to know about this stuff. but I do have somewhat of a clue and lots of smart friends, and that seems to help my ignorance out a bit.
If my edge router with 3x100Mb interfaces can route them fine, why would I care when the device is routing the circuits I'm throwing at?
-J0
-
05-16-2005, 02:35 PM #48Aspiring Evangelist
- Join Date
- May 2004
- Location
- Blue Springs, Missouri
- Posts
- 366
sure.
but you need minimum of 2ghz to barely do things proper with pc hardware. and what i mean by 'proper', is doing gig-e fully saturated with small 64 byte packets at 1.4 million pps
for every packet, the router has to lookup forwarding table to decide where to route out, and freebsd/linux use patricia trie algorithm which is quite inefficient... so it takes a HUGE toll on CPU.
but you I'm sure you know all this. right?0
-
05-16-2005, 02:40 PM #49Junior Guru
- Join Date
- Dec 2002
- Location
- acton, MASS
- Posts
- 229
Originally posted by omaha.stylee
sure.
but you need minimum of 2ghz to barely do things proper with pc hardware. and what i mean by 'proper', is doing gig-e fully saturated with small 64 byte packets at 1.4 million pps
for every packet, the router has to lookup forwarding table to decide where to route out, and freebsd/linux use patricia trie algorithm which is quite inefficient... so it takes a HUGE toll on CPU.
but you I'm sure you know all this. right?
-J0
-
05-16-2005, 02:50 PM #50Aspiring Evangelist
- Join Date
- May 2004
- Location
- Blue Springs, Missouri
- Posts
- 366
heh. another one of shawnm's friends?
pretty sure all this has nothing to do with wholesaleinternet being down. tsk tsk. shame on all us!!
but I'm done with this, turned into a flame war. when I could care less about you people. hope you all go far!!
cheers!!0