We got hit with a 20 minute outage, confirmed from our Seattle / Phoenix / LA locations... started at 08:00:00pm PDT...

Indeed, I noticed it as well.

It was not an outage at the NOC. If your traffic passed over Level3, it was likely routed overseas to the UK once the traffic hit their node in NYC (at 209.247.9.89, so-2-0-0.mp2.NewYork1.Level3.net) and never came back from the UK. We witnessed the same thing, and it was only Level3 that was affected. Traces from Stanford University over above.net were unaffected, for instance, and traces that used UUnet were likewise unaffected. Our current traffic is still going over Level3, still being routed to the UK from their NYC mode, but coming back, at least.

I don't usually post here any more. But threads like this really, intensely bother me.

Two of nacs oc3 where cut out by the WTC disaster. Telehouse NYC on broad st about a block away, where there is no power and they arn't allowing fuel trucks, the generators ran out of fuel. Those two oc3s to NYC went down. Leaving the OC3 to MAE-East threw the OC3(soon to be oc12) to newark. The Level3 connection at Newark the primary provider there, was having issues.

That is as simple an explantion as I can give.

Annette: >> But threads like this really, intensely bother me. <<

Do you care to explain this comment? I started this thread, as I feel the purpose of this forum is to share actual customer experience. A 20 minute outage confirmed from 4 different paths; take it or leave it.

So what if it it was an upstream provider that caused it, it doesn't change the fact that the servers were unreachable.


Annette:
Are you suggesting that only a few routes were impacted?
Both of our active remote servers, esnipe02.esnipe.com (Phoenix) and esnipe05.esnipe.com (LAX) could not reach esnipe01.esnipe.com, plus I could not reach it from both Sprint (208.30.235.92) or Qwest (63.229.23.90) links in Seattle. Maybe you found some path in, but I sure didn't from anywhere up and down the west coast.

jayglate, when does the upgrade from oc3 happen? any set date?

Craig

jayglate - I'm sure you have had a busy week. Sorry to hear that the WTC mess is impacting you guys. Thanks for the details.

All of NAC's oc3 are being upgraded to OC12. In addition Pegasus is bringing its OWN OC3 Line from UUNet scheduled to be in within the next 15 to 30 days. Giving us additional redundancy beyond NAC network.

Stephen: What bothers me is that a 20 minute disruption in service is unremarkable, particularly when a visit to traceroute.org would have yielded routes that were working. The servers were unreachable for you. Ours servers were also unreachable - for us. The majority of our clients had no idea there was anything wrong until I posted an announcement in our forum, as their traffic was not impacted since it did not travel over Level3. This does not equate to an outage at Pegasus/NAC, which was the subject of your thread (and which I see the mods have changed to reflect further information). You found no paths in - but Exodus West, Stanford, YNN.com, and San Diego State U (several among the list that we checked from the west coast) were all happily finding their way to the NOC. Traces from Yahoo failed - because that traffic, like our own, uses Level3, and if you look at a current trace for them you'll see that bounce to the UK and then back again.

This is not a newsworthy event, in other words, and serves only to give the impression that there is somehow something wrong at the NOC because you, for 20 minutes, could not reach it. If everyone posted about every 10-20 minute hiccup they saw with zero other information and called it an 'outage at such-and-such NOC', this forum would be filled with nothing but that.

Just drop it.

My goodness, such venom. Did I say that you shouldn't tell people that you couldn't connect to your server? No, I didn't. Did I say that 20 minutes doesn't matter? No, I didn't. Did I say that threads like this bother me? Yes, I did.

Our servers - all of them - were indeed active throughout the entire period. Our clients - almost all of them - were indeed able to connect to their sites/their mail. Mail forwarding offnetwork was working just fine as well, unless the route hit that level3 node, at which time the mail remained in the spooler on the server from which it originated until that route was restored. So much for all outbound routes being broken, too.

If you 'personally did not know' about what was going on, then perhaps you could have contacted Jay and asked. It isn't as if he's unavailable, as I got him on ICQ about ten seconds after level3 fixed their routing issue, if only to let him know what we saw with level3 and hopefully warn him that he might get questions about unreachable sites/servers. Or, perhaps you could have visited NAC's network page and taken a look at the issues they were facing in NYC - not directly related to what we saw during the disruption, but helpful enough so that you would have known that they were in fact facing issues related to the WTC attack.

My contention is not with the fact that you had trouble connecting or that you wanted to shout it to the world, because that's just a fact of life at this forum. My contention is with the fact that you immediately, and with no supporting evidence whatsoever, posted a thread like this implying that there must have been some problem at the NOC. This is just as irresponsible as the (numerous) people who post '(Insert hostname) is down!' threads without checking anything or providing any information about what they see (traces/pings) only to have people chime in and tell them (insert hostname) works fine from their location. First, it isn't true, and it casts a pall over any host who maintains servers at that NOC, without reason. And second, even a little bit of troubleshooting - like those aforementioned traces or pings, done from various locations - would have shown where the problem really was.

You're upset about the disruption in your service, and that's understandable. That is no excuse, however, for being rude to me for voicing an opinion contrary to your own.

Just drop it.

Please indicate exactly where Jay indicated there was an outage of the NOC. If you reread his post, you can see that at no time did he indicate NAC was down. In fact, since I have already told you that all of our servers continued to receive traffic - even though we ourselves were unable to reach them - I'm amazed at how you think this would be possible if the NOC was down.

That 'raw information' you provide does nothing, period. Without any context, what exact relevance do you think it could possibly have? Your immediate conclusion was that the NOC was down, which was wrong. So exactly how is that helpful? I knew it wasn't the start of a major, system-wide outage. Know why? Because I took the time to actually do a little troubleshooting and research before running off to a forum well known for being like a pack of wolves, claiming that the entire NOC was down. Because of the steps I took, I was quite secure in the knowledge that a) not everyone would be affected, b) that it was not the NOC, and c) that since it was level3, they probably already knew about it and were working to resolve it. I'm sorry I don't buy into this whole sharing of raw knowledge idea of yours - which doesn't look to be too knowledgeable at all - and trying to make some company look bad over something that is completely beyond their control and completely incorrect - based on the original title of this thread, which you started. Interesting that you will accept the word of someone else who 'confirmed it' and yet don't accept my own reports, which indicate that there was never an outage at all while giving information about level3 and the problems they were having. How handy a double standard that is.

You asked why threads like this bother me, and I told you. If you take offense at answers to questions you yourself ask, then don't bother to ask them. You accuse me of making assumptions about you being upset or not (and if you weren't, why bother to post this thread in the first place?) and then proceed to do exactly what you have just complained about. Yet another handy double standard.

Oh - and thanks for continuing to insult me in the face of everything that shows you have zero interest in providing any sort of rational, reasonable, responsible reporting of issues whatsoever.

Everyone has downtime, theres not a single company out there that hasnt had downtime, it really doesnt do any good for someone to post everytime theres a small amount of downtime, if everyone did that, this forum would suck, all it would have is thread after thread of how someones server was inaccessible. The more appropriate way to notify someone of your expereince with Nac would be if someone started a thread asking for people opinions of NAC or their history of uptime/downtime, at that point it would be approp. to let them know that overall you have received good uptime except for a 20 minute outage a couple weeks ago.

Of course you have the right to post anything you want if its within the forums rules, but really should we post threads that we know will or could start a controversy? I myself am guilty of this too, perhaps we should all think before hitting the good old Submit button.

BTW, just so you can be assured that traffic was indeed making it to its destination, since you seem disinclined to take my word for it (too bad I didn't save any of those traces) and only interested in taking things personally instead of understanding why threads like this bother me after having asked that question yourself...

I posted my first notice on our forum at 2325 Eastern, and another at 2342 Eastern when level3 had repaired their routing. I selected two servers out of all the servers we have at NAC and pulled the FTP logs. From those, I snipped out a couple of random entries from that timeframe. Remote hostnames and usernames have been removed, for obvious reasons.

Sep 14 23:32:13 kronos proftpd[6585]: IP here (c1331724-a.schrls1.hostname.removed]) - USER username: Login successful.
Sep 14 23:29:07 demeter proftpd[28097]: IP here (modemcable.hostname.removed]) - USER username: Login successful.
Sep 14 23:29:19 demeter proftpd[28118]: IP here (dialup-hostname.removed.Detroit1.Level3.net[IP]) - USER username: Login successful.

I can almost guarantee you that if the disruption had actually been at the NOC and all traffic was affected, that there would be another pages-long post about it right here.

Now, you're more than welcome to call my posts 'stupid' if you like. I will leave to you the duty of explaining how traffic makes it to a NOC that by your account is down.

allright people. Calm down, or im gonna have to lock this thread..

Just drop it. Reporting a minor outage raises questions like "should we post threads that we know will or could start a controversy?"

Just to clarify things - I said that the outage was from 23:00:00.00 to 23:20:00.00 EST and I never even used the words "NOC is down". I reporting an outage measured by our own systems in LAX/Phoenix/Seattle.

Your thread title is "pegasus (pwebtech) 20 minute outage". Too bad you've edited all your messages so that people can't see what you really posted, which is in direct contrast to your last message since you were arguing that the NOC must have been down because you couldn't reach a server there, even though you had already been told reasons why you might not have been able to get there.

Of course this raises another issue now that you've edited your messages, because it now appears that you believe pwebtech was down for that time. I'm sure Jay will appreciate that, since it is fairly obvoius that they weren't and that nobody really was.

Some things in life are really simple, and this isn't really controversial, it's just idiotic. If it's so minor, in your words, why post it at all? What could your motivation possibly be if you didn't think it was serious? To make pwebtech or the NOC look bad? To make yourself feel important because you happened to notice a 20 minute period of time? Or to demonstrate to all of us just how smart you must be because you proudly diagnosed the issue for everyone else? A diagnosis, I might add, that was laughably wrong. This whole thread annoys me too, since I have a site at NAC via pwebtech, I could reach it just fine, and you seem intent on doing what 90% of the people that populate this forum always do: cut the legs out from under somebody else. Two people have pointed out that this forum would be filled with nothing but that initial post by you if everybody decided to run right over here and do exactly what you did without any other information at all. You started this thread with ambiguous information at best and erroneous information at worst. You chose the offensive route. You bear the consequences for it.

The mods ought to go ahead and lock this thread since it's also obvious that you're now working your way to covering your tracks for the wrong/bad information that you posted to make yourself look like less of a fool and ass. Won't work on me, I'll tell you that, because I saw your original posts. You should be ashamed of yourself. "Just drop it" indeed.

Peeps -- I am sorry. I drop this debate. I stand by my post that there was a 20 minute outage. The rest was just foolish debate over the term "outage". again, I"m sorry you guys are so upset over my little post.

Everybody needs to take a chill pill after the WHT. I've said I'm sorry, and just drop it.

Good Idea lets just all drop it. This week I will be setting up a password protected messageboard for pegasus clients.

I think we need to be fair to both sides. Both Peeps and Annette have made comments along the lines that "90% of the people who populate this forum" are just here to "cut the legs out" of vendors.

That is NOT what this thread was about, I never had that intention. I think there are different groups of peole who view this (WHT) forum in different ways.

Reporting an outage is not about making a host look bad. But I can see that perspective from some of the less operations-oriented people. The Co-Location/Dedicated forum is often heavily dominated by people ware are interested in making money ($), where another group (me) who are more interested in the technical talk and finding out what is going on with the technical front.

Sometimes these different perspectives and needs clash.

I represent a Pegasus customer, I am NOT out to "cut the legs out" of them. That is just foolish, I am not some idiot newbie.

It is obvious that some people feel this WHT forum isn't the place to discuss outages. Fine, I hear that feedback. But I still want a place to do that,...

In that sprit, I have posted in the WHT suggestions forum that a new forum topic be created so that the "sales related discussions" are split from the "outage reports".

http://www.webhostingtalk.com/showthread.php?postid=157481&t=5097#post157481

Please try to understand that there is a purpose in getting idependent information about outages. And the vendor itself is not always the most reliable source on the matter (am I the only one who has worked with ISP's who have tried to cover-up outages?!).

Thank you. And I am sincerely sorry that outage post caused so much grief. I've dropped the debate on the outage itself, but I do feel a need to try and explain why I started this thread -- it isn't to cause Pegasus harm, it was to share and gain informaton. Again, I'm sorry, I intend no harm - I view discussing bad things as a way of solving them.

No offense inteded Stephen, but, no matter how much we discusssed any issue regarding an outage or most outages there is absolutly nothing that anyone except the actual vendor can do is but talk about it.

Jay - I can understand why you say that - but from my persective, there is something I can do during an outage.

I manage a a web site that has three sites with three different ISP's (in three different cities) - and if your site goes down - I can shift my users (using DNS changes and other methods) off your site.

Even if your site is having less than 100% failure, say that 50% of the traffic is being dropped... I want to know that - so that I can do something about it (route my users to the other two sites).

Often I want an idea of what is going on, and discussing it with other customers helps get more detail and interpretation on the matter. Don't take it personal, often times ISP's don't fully disclose problems or are too optimistic about when they will be fixed... and in the worst case, deny them.

Do you see my perspective?

Originally posted by jayglate
This week I will be setting up a password protected messageboard for pegasus clients.

That's great!! Will you send a notice to all pegasus customers letting them know the URL etc?

I am not some idiot newbie.


HEY!!!! I RESEMBLE THAT REMARK!!!!! :D



..... Comic relief.. exit stage left....



There has been too much grief these past few weeks.. wanted to liven up this thread...

I'm done now... as you were...

;)