Well, welcome back everyone after that wonderful 9+ hour outage at VDI. Well, I have nothing to say except to post the letter I sent out to my clients:

--------------------------------
I would like to personally apologize for the outage that was experienced today. I can assure you this is not something that happens often and has taken us by complete surprise.

The reason this outage occurred was due to a massive router failure between the data center and MCI/WorldCom. Somehow the routing tables were completely deleted and all servers in the data center became inaccessible due to this.

I can assure you that I am in the process of taking steps to make sure that our clients are informed during an outage like this and regret that these steps were not already in place. In the coming weeks Webfors Technologies will be setting up a "Status" server that our clients can refer to during an outage. This server will be situated in a completely different data center therefore any network outages experienced at one data center will not affect the other. The website on this server will also contain alternate contact information (phone, email, etc...) in case you need to reach us during outages similar to today.

The internet is not perfect and many things may go wrong. Although we make every effort to prepare ourselves and our clients for problems that may arise, there will always be situations like today that are unavoidable.

Again, I apologize greatly for the inconvenience this may have caused you. I can assure you that we also feel this to be completely unacceptable for our clients. We have added a free month to your account to show our appreciation for your understanding in this matter.

If you have any questions please feel free to contact us.

Sincerely,

Eric Forsyth
Webfors Technologies~

-----------------------------------

Feel free to use it if your racking your brain to come with something.

Have a great weekend everyone :)

WHT is back up and my site is back up....all is well...Ahhhh!!

I just want to let you know that while everyone was in the middle of dealing with the VDI outage, Jay Silver of pwebtech.com saw it as an oportunity to spam sitepoint members (through the private messaging system on the sitepoint forums) to peddle his dedicated servers. <<ADMIN EDIT>>

http://www.sitepointforums.com/showthread.php?s=&threadid=18748&perpage=25&pagenumber=5

[Edited by Chicken on 03-16-2001 at 11:36 PM]

517-505-6191

Press Release

VDI experiences Multiple Network Outages


NEW YORK -- March 16, 2001-Virtual Development Incorporated (VDI), a leading
provider of Application Service Provider hosting (ASP), experienced multiple network outages.
This morning at 9:14am EST, Verizon had an OC-12 outage in the Clifton, New Jersey area. This OC-12 provided VDI with their connectivity to various providers such as Level3 and UUNET. Verizon was on location with in 2 hours
to correct the problem with the OC-12 multiplexor. Once the OC-12 was restored, connectivity to Level3 and UUNET would not come up properly.

The second problem was that our connectivity didn't establish full routes. During the times when we were down, we were told that at that exact time when VDI went down, other multiple providers went down. "The network outage we experienced today was unfortunate" said William Jensen,
president and co-founder of VDI. He continues, "We are taking all the precautions necessary to make sure an event like this never occurs again." VDI is currently working with Cisco systems, Inc., investigating what appears
to be a glitch in the software. By 6:05pm EST, VDI reported that all services had been restored. They are still
investigating the root cause of this issue.

We also would like to publicly thank http://www.nac.net for coming down to the VDI data center and helping VDI out.

[Edited by William on 03-19-2001 at 03:07 AM]

I have talked to Jay from Pegasus Web Hosting / http://www.pwebtech and he seems to really regret what he did (the whole spam thing and all). I am willing to bet he doesn't make that mistake again. I for one am willing to give someone a second chance who is nice enough to help out a fellow company get their network back up. I think calling someone who made a bad judgement call a scum bag is going to far. If it happens again then I am willing to stand behind that statement but until them I am willing to let it slide seeing as he was helping to get everyone's site back up.

Well I have gotten some more info that suprised me. If this info is correct then Pegasus spammed no one.

It really sounds like this has been blown WAY out of proportion.

From what I can tell. Pegasus ICQed 1 person, PMed 1 person and made only 3 sales calls. First off the sales calls are standard business practice for most all companies that do sales. Doing one PM and one ICQ to me does not qualify as spam. It is another type of targeted sales call to be used when a phone number is not availible. It's not like they sent the same message to 50 people. They send a message to someone who they wanted to earn their business. If a business is not allowed to do this then the whole US econemy is in trouble.

i dont get how he can help vdi out and yet spam other people.

Snicker! In Quebec (Canada) Tabernac is a major swear word.

Was that the intent?

I'm *sure* this was poor judgement, but I will not comment on the number of PM's or anything like that as it only concerns one person (Jay).

I will say however, that advertsing via PM or any other means, at Sitepointforums is strictly prohibited (in case anyone gets the bright idea), as it is here (though we don't have PM *yet*).

I'll also add that we prohibit name calling and thus the message has been edited. It still retains the idea, without the insult.

I apologize for violating Sitepoint rules, I was/am new to sitepoint with only I believe one or two posts. And was not aware of the rules at sitepoint. And in terms of nac/pegasus helping out VDI. Having peoples businesses down that long is just not right. That is how they feed there families and support themsevles, freind or foe, that is the facts. Peoples lives depend upon their business/website being up and running. But on the flip side, we live in a capitalist country, and soliciting business is not against the law or wrong.

so how did you guys help them out?

They needed a new cisco flash card with a clean IOS for their router.

Ya im from Ontario and man thats not appropriate for this Board!

What board are you talking about WHT?? I didn't do anything with this board, of the sort.

the word tabarnack not u!:-)

Originally posted by jayglate
They needed a new cisco flash card with a clean IOS for their router.

awesome, you had all those and just dropped by vdi and helped them? from the whois info on your domain, you're either in arizona or california. i didnt know you were local to vdi.

damn, took me forever to get that post in :)

about 35 minutes from them.

Hey, sorry for going overe top in my first post. I think the juxtoposition of the incident with the VDI outage got my blood pressure up. I feel for the smaller web hosts out there who have to put up with the ire of their customers because of someone elses problem (VDI. level3, whoever) and end up in a squeeze.

do vdi refund their clients?

That would be a question for bill....

there are a bunch of hosts who have multiple servers there and most hosts are offering credits or upgrades to their end users, taking a loss for now.

VDI will be giving out Coupons - With Credit - Clients have the OPTIONS of Using this anytime they want toward thier Bill @ VDI.

Now, that is good business :)

Very! Nice job... ;)

NOTE THE VISIT FOR THE FLASH CARD WAS NOT FREE :)

But it did help us out, cause no one in the area had a clue what was going on. if we listened to our Upstream instructions, we would not be here today to make chicken jokes.

I was not aware that Jay and NAC did call just about everyone of the clients since 10am, 1 hour after the outage.

But what can I say ? Business is Business and just lets hope clients make the right choice...

But NAC and JAY Did help us out when we did ask them; when we called.

Nice way of describing everything. Anyway, are those cards expensive? I take it they cost much more than any old pci card for my computer.

VERY expensive. Especially on such short notice. If you order them in advance, I'd imagine they are less expensive. If your entire network is offline and you need one NOW, I'm sure the price goes up quite a bit. ;)

Matt

I love VDI... if yoy really love someone, you have to appreciate its outage.

Yesterday a lot of customers called me about the VDI outage. I tried to learn them that they must love VDI, because SHE care about us. She brings us on the road of success... I love VDI...

I love VDI... if yoy really love someone, you have to appreciate its outage.

Yes. This is true.

I'm sure I speak for everyone when I say we appreciated the outage very very much. :agree: :agree: :agree:

*more outages, more outages!* :D

~Matt

[Edited by Site5-Matt on 03-17-2001 at 04:10 AM]

I am sure they could get a contract with cisco that could supply them with one in emergency cases like this time.

I am pretty certain Cisco's swap out service is 9 hours so doing what they did was probably faster.

I am sure someone will tell me I am wrong!

Thankfully we have servers in three locations so we were not completely wiped out, but it was not a nice experience....

I had an ad campaign running for domain registration.
Fortunately I managed to pull the ads before they went out as our dom reg server was knocked out by the outage.

On the bright side.... once the system came back on we made about $50 in domain sales without the advertising.

We live to fight another day.

Gordon

Originally posted by Site5-Matt
[quote]
I'm sure I speak for everyone when I say we appreciated the outage very very much. :agree: :agree: :agree:

*more outages, more outages!* :D

Damm, I am so pissed that is not even funny ;)

This should have never happened, when you have some of your biggest d-server customers calling you and telling you YOU suck, and they what their machines out of here .. you will know what I mean

This outdge cost me more money, that some people make in 3 or 4 month's work... we survive, but man, does it make me mad !!

I know how you feel Carlos.

By the way, the outage started just as I was trying to access your web site.......... sorry.........

Gordon

GordonH

No I am sorry ..

10 Hours .. the number is still ringing in my head .. it's been 12 hours since we came back, and I can't seem to easy my frustration.

I mean .. what do you do ?, how do you explain customers that all your redundancy promises where a flaw ?, hell it reminds of the backups in Alabanza (inexistent)

VDI is still number 1 in my chart, but this is a big wake up call, it's not good to have all your eggs in one nest, I am contacting other providers, not to Leave VDI (unless something like this happens again) .. but definitely starting a backup plan

And Gordon, you are welcome to contact me if you need anything

Let's hope VDI has taken the steps needed to prevent this from happening again. I'm sure they have. This is not something that they want to happen again, I'm sure ;)

Thumbs UP to NAC.net for assisting VDI during the outage.
I've used them as an ISP for about 5 years now. As far as I'm concerned, they're a great company.

As someone who has taken a cautious approach to starting
a reseller hosting business, the recent site outage
experience that many of us have had to react to, prompts
me to post my opinions here:

We pay for hosting from both Venturesonline and Burst for
2 different startup reseller domains. We received an
informative email explanation about yesterday's service
interruption from on of the 2 providers so far.

After we discovered that our sites were inaccessible
yesterday morning, we turned to a search at sitepoint.com
to get information about why our sites were down.

The major story here seems to be prohibitive pricing
policies and inadequate response times and tech support
from big companies like Cisco Systems and Verizon.
We believe that Verizon has deliberately impeded the
avalibility of DSL services to the masses, and has not done nearly enough to rapidly upgrade the infrastructure in
it's own central offices and network grid to enable and
support internet broadband reliability. AT&T/@Home, TimeWarner/Roadrunner, and other RBOC's (SBC, etc.) have
not done whatever it takes to provide reliable broadband
connectivity at the predicted pace of expansion that was
expected by so many investors in tech stocks.
This has resulted in 91 percent of U.S. internet users
having no option but 56K dialup. The result is not
incentive to upgrade PC's, since slow internet connections
are the bottleneck, not processor performance.
Verizon and other RBOC's were deliberately slow to cooperate
with DSL providers like Covad, Northpoint, and Rythm.

Cisco seems better at charging monopoly level prices, than
in providing highly reliable, state of the art routers.

In the middle of the foodchain, companies like VDI seem
caught in a vise of having to meet extremely high customer
expectations for performance and reliability, while dealing
with poor tech and network support from Verizon and Cisco
pricing and product performance.

Jay What's-his-Name, seemed to quickly react to VDI's
crisis by soliciting it's customer list and by publicizing
his offer of "help" to VDI by providing a replacement for
a failed Cisco router flash card. VDI seems to have put the
welfare of it's customers above it's own reputation by
revealing to Jay that it did not have it's own backup for
the troubled Cisco card, or the depth of tech talent to
diagnose the Cisco router fault sooner. VDI could have
waited for Cisco to solve the problem and extended the
downtime, rather than reveal it's failure to have a
backup router flash card on hand to competitors like Jay,
and to existing and potential customers.

Jay and his company helped VDI, but apparently gave nothing
without full financial reimburesment, and then defended it's
opportunistic and predatory marketing effort quickly
launched when VDI was down, by public
statements that it came to the rescue of VDI because Jay
had invested in an expensive backup Cisco router card and
VDI had not.

IMO, the competitive dynamic of companies involved in the
internet is hurting, growth, reliability, and service and
performance enhancements. Web Hosting customers demand and
get great hosting prices, most resellers and NOC operators
are under great pricing and performance pressures, Verizon
sleeps while most internet users are stuck with 56K dialup,
NOC operators are at the mercy of Verizon and other RBOC
indifference and incompetence, and Cisco router product
price gouging. PC sales suffer, Cisco stock price sinks
from $80 to $20 due to slowing investment in internet
build out, and the Nasdaq Stock market tanks.

Cisco must build cheaper and more reliable products,
Verizon and AT&T must quickly solve the broadband last
mile connectivity problem, and Microsoft, Intel, and other
software and semi-conductor companies must invest in
AT&T and Verizon efforts to reverse the broadband stagnation.

NOC operators like VDI must pay less for network components
and tech support, and charge more for their hosting
services. Resellers must pass these increases along to
their customers. All players must charge enough to sustain
themselves for all services, because loss leader initial
pricing with the intent of "brand" building, will only
insure that no one makes enough of a profit to provide
reliable and high speed service 6 months or a year from now.
Yesterday shows that cheap, high speed hosting service with
great reseller management software has less value when there
is uncertainty about reliability.

So far.....it seems like the answer for most of those who
consume web hosting services and post here, is to endlessly
switch providers in the quest of improved reliability,
pricing, and performance. This activity does not seem to
be successful for most. Cisco can no longer net 50 percent
on each router, Verizon can no longer have a 4 week backlog
to test a residential line for DSL service, and resellers
cannot offer $8.00 month web hosting rates. Everyone must
charge more rationally, perform more reliably, and exhibit
more loyalty and compete more fairly to reverse the
negative trend in growth.

Originally posted by yellowed
Jay What's-his-Name, seemed to quickly react to VDI's
crisis by soliciting it's customer list and by publicizing
his offer of "help" to VDI by providing a replacement for
a failed Cisco router flash card.

Jay and his company helped VDI, but apparently gave nothing
without full financial reimburesment, and then defended it's
opportunistic and predatory marketing effort quickly
launched when VDI was down, by public
statements that it came to the rescue of VDI because Jay
had invested in an expensive backup Cisco router card and
VDI had not.

IMO, the competitive dynamic of companies involved in the
internet is hurting, growth, reliability, and service and
performance enhancements.

Hmm, I don't think you understood who was doing and posting what. First off, pwebtech wasn't the one who posted that they had helped out VDI. William from VDI released a press release stating that this was done. That how I found out about it and I had talked to Jay personally and he didn't even mention that this had happened. If you say jay did make public statements that they had come to the rescue of VDI please provide a link because I have seen no such thing here or on sitepoint.

Also no one ever said the hardware given to VDI was free. As it has been stated the card was something that is very expensive. Did you really expect for pwebtech to charge for it? You are making them out to be A-Holes out for no one but themselves. The fact is that if they really wanted to hurt VDI they would have said no when a card was asked for, making the outage last longer until VDI could get cisco or someone to send them a card.

As far s being competative goes, I do not want to be hosted with a company that is not competative. That means they will go out of business. No if, ands, or buts.........
On the internet we all like the companies with the best product but the fact is, Sales is the name of the game, not the product. If you have both you are in great shape but without a good sales team then you will fail with the best product in the world.

Not to step on anyones shoes but it appears this was very much VDI's problem and they or others are trying to shift more of the blame :(

-BW

Hi yellowed,
Overlooking a point or two (in possible contention) I really enjoyed your post and feel it is the kind of thinking/leadership that we need from a hosting-industry, professional association.

And around here, such a long post with only a point or two in contention is rare.
Gary

One question I have for VDI is how come for starters they don't have 2 different data paths into the NOC?

Seems a bit trusting to hope that one company (ie Verizon) are going to have a stable system 24/7 without any problems. If VDI had a different provider backup to link them onto their other data pipes that they could switch onto if a problem like this arises perhaps a lot of the pain could have been adverted in the first place.

Even if it was only a T3 backup, who really cares....it's a lot easier to explain to clients that they're going to get reduced speed for 9 hours, then sorry guys your sites are going to be unavailable ;-)

I'm not that familar with VDI's network system, but from the reading I've done the verizon OC/3 links them to mci and level3. All eggs in one basket...perhaps.
The router failure is a different issue again, but I'm sure with some creative thinking the chances of failures like this can be drastically reduced without blowing a huge amount of money.

I'm not a network guru...more of a *nix admin/programmer so feel free to correct me if I'm wrong :-)

regards,

Simon

This whole VDI outage is just ugly.
I would like to see how they recover from this.
It could go either way for them now.
They are at a turning point.
I hope they do not loose the ground they gained from Alabanza's recent problems/issues.

hostpolicia

I dunno...I find this real interesting. I mean it -seems- (I may be wrong) that every VDI based host, be it reseller, co-lo, dedicated, or whatever goes on and on about the security of the VDI NOC, how theres armed guards, and those wonderful multiple redundant connections. Of course, I couldnt find any network info on VDI's page, but I honestly didn't really didn't look for it too hard either.

comments from several hosts "Network Info" pages:

"The NOC is connected via redundant fiber connections to our backbone providers" (From UltraspeedUSA)

"Having direct connection to the backbones" (From both UltraspeedUSA and Wizardshosting)

"Lightning Fast Direct-On-Backbone Fiber Optic Connections"
and "Multiple Redundant Network" (from Wizards)

My First question is, based on the info I've seen here, these were multi-redundant through a -Single- Verizion line?

Of the 3 hosts I checked, Wizards was the only one to mention the BGP tables which seem to have caused the majority of the headache.

My Second question is that for a company (VDI) that has so many clients promoting based on their "Reliability" and "Redundantcy", why was a backup allowed to go "stale" as it's been put? For a mission critical piece of equipment, my opinion is that based on the size of their facility, and the volume of trafic they deal with, Hourly backups seem to be more in order.

Please note, I am in no way knocking the companies I quoted above. I'm just using their verbage as an example, tis all. I only wish to seek out the truth here, since obviously having that tank platoon in the basement at the NOC didn't help. :) Plus I'm a bit annoyed that I'm going to have to eat some charges to make amends to my clients for a ****-up that on paper, shouldn't have happened.

http://www.vdi.net/network

The Fiber lines have to leave the building one way or another and then connect to another building. The point to point connection is called the local loop, this portion of the service is controlled by the phone company, hence then have to run the lines via poles or underground. These lines have to process through a POP or a pass through location; this is where Telco;s get monitored as well.

VDI issue was combination of various things.

A: The routing table was wiped "no routing" so nothing would have passed through the border routers

B: Verizon went down a few minutes later "according to the time frame they gave us at the end. Local loops were out, "which was supposed to be a slim to non issue. We learned about this outage at 4 pm even though another section of Verizon told us the lines were good.

We still do not know if both are related in someway way shape or form. Once we were down, we were told alabanza went down the same time we did, as well as another backbone in Europe. We are still investigating whether this was a malicious act.

Now, no routing table, no local loop and no upstream engineers with a clue what was going on.

Between the 3,908,985,001 phone calls "We still managed to pass all calls through" and drank 2 cases of amodium AD while we were in communication with upstream.

During the 9 - 2 we did not know anything about Verizon outage, just the engineers going through every line in the router to see what could have happened.

Then we decided that we could not get the routing back, we decided to just Swipe n wipe the router and then restore the IOS. The new routers have a pmcia flash card with everything on it. Whelp the First card did not even read, no big deal, so we opened up the emergency back up card................... and then it did not work.

3 pm we called Cisco smartnet program and they said they would have a card delivered by 7 PM, well we didn’t want to wait. We called up a local large access provider NAC and ask if they had a Flash card,

At that point we didn't know NAC and Jay silver was calling every client VDI had and solicited them to move to their data center. Many clients did inform us later that day about all the phone calls about Jay Silver and Alex owner of NAC.

4 oclock Verizon restored the Local loops, "Level 3 called to inform us that the local loops were fixed.

5 oclock Nac showed up, we got the ios back in ...........
And restored the data, and then 5:30 on everything was back to normal.

We now keep 3 ios Flash card in stock :)

Now, just for the record.. We have a new Cisco catalyst 6500 series Switch with layer 3 coming in on Wednesday of next week. We will inform when this will be getting installed. there will be minimum outages while we just switch the many many cables.

our next phase is to get a 3rd local loop from a carrier that is not Verizon Dependent.

Thats the whole story and nightmare in 1 sitting :)

VDI is always looking to improve business and will never repeat the same issue more then once.

I do thank you for your understanding in a tough situation, I was very impressed to see that there were many clients who hung out in there. I was even told there were people calling to order accounts while we were down. "That tells you that you have a good thing going.

If there were outages during the same time as VDI, I’m still interested to see if this was all related in 1 way or another.

3 clients bill, only 3, NOT EVERY CLIENT.

Sorry :) a few Clients.

Originally posted by jayglate
3 clients bill, only 3, NOT EVERY CLIENT.


it was yet a cheap shot!

[Edited by kunal on 03-19-2001 at 03:25 PM]

So technically, if Verizon goes down, VDI is down?

Keep us posted on the 3rd local loop :)

As an aside, I mentioned to one of my clients the Verizion info. He just nodded his head and said "ahh...its the phone companies fault". Seems he's had some "DSL" issues with them so got very understanding at that point.

Sometimes its easier to blame a known evil rather than explain the details. :)

gay

Well outages do happen....

Originally posted by William
...
Between the 3,908,985,001 phone calls "We still managed to pass all calls through" and drank 2 cases of amodium AD while we were in communication with upstream.
The network outage gave you diarrhea?

That's understandable, I probably would have soiled myself too had I been in your position. :)

I think Bill was taking the Imodium to deal with the amount of $h!t he was being fed from his upstream providers.

We have our servers at VDI and we also did have the problems to explain our customers why the servers are down.

If I read the posts and see that a lot people blame VDI for this problem I am just not sure if these people understand what the internet is....

Of course such a downtime is always not very nice but, hey, the internet depends on a lot of various individuals, companys and people. If now some company has a problem then it is somehow consequential (at least for me) that the people which depends on this company also have a problem.

Only to blame VDI is just not fair and, in my opinion, all people which blame VDI are just jealous of them because they are successful.

Sandy

P.S. For these individuals which now wanna post "You are a VDI believer" - we do also have serves at Alabanza. I just look objective to the internet problems and not emotional.

coool off sandy... :) things have been cleared now...

Originally posted by Sandy

Only to blame VDI is just not fair and, in my opinion, all people which blame VDI are just jealous of them because they are successful.


Hmmm, are you saying that if you blame VDI you are not successful? I beg to differ. Although I do not blame VDI for the outage, it is clear that a backup plan was not in place. Who do you blame for this?

That does not mean that I am jealous.

William,

In another post, you mentioned VDI will be getting some new bandwidth. How has that gone? What do you guys currently have?

Hmmm, are you saying that if you blame VDI you are not successful?


Micheal, I didn't say that if you blame somebody you not successful. Sorry but I didn't say it clearly:

Fact is, that it starts with Alabanza. Everybody was saying how cool Alabanza is. After a while some people realized how successful Alabanza is and then they said Alabanza is bad and they have only problems.

A lot of people went to the "small" company called VDI and the game started again.

What I wanna say with "the blame thing" is that for sure some people will leave VDI and go to the company XY and say company XY is great and a view month later they will say the company XY is bad and they have only problems.

See, thats the point on my statement. You can't blame companys if they get successful and this is happen here.

Sandy

anyone else notice that 3 min outage on vdi?

Nope.

nope