Press Release from Virtual Development, Inc
Re: Level 3 down time


During the last two months, we have been preparing for the installation of an OC12 circuit. This circuit will be in a sonet ring, which provides substantially higher uptime than non sonet ring based circuits. In addition our new OC12 will give us access to several different bandwidth providers allowing for bandwidth provider diversification.

About a month ago Globix filed for bankruptcy so we decided to disconnect the circuit since Level3 seemed solid enough to hold the line for about 45 days. We incorrectly assumed there would not be any issues.

In October of 2001, we recieved an incorrect bill from Level 3. That incorrect invoice amount was put into dispute. We had been working with Level 3 to have the invoice corrected. During this time, several Level3 employees (who were working on our account) were laid off, including some management. The dispute then went unanswered since October.

Today, someone from the Billing department looked at the bill, saw the portion of the bill that was in dispute and made a mistake of assuming it was in collections. The person then had the line disconnected and that person went home.

It took about 2 hours or so to get to the Level3 night shift manager and have them look at the account and see that the circuit was disconnected in error. Their response was we (Level 3) are sorry, and will turn the circuit back online.

Once again, we deeply apologize for any inconvenience and will deal very strongly about the issue. As stated above, our previous preparation of the sonet-ring based OC12 circuit will allow us to connect to several different providers and avoid a similar issue in the future. The ETA for the Install is the beginning of March.

I know it takes time for these things, but can you not get a temporary backup asap? I understand these things happen but you should have had a backup ready for when you took down Globix.

Thank you very much for the press release, Bill ... All of us are looking forward to the OC12 installation. In the mean time, I do surely hope there would be no more outages ;)

cheers,
:beer:

Originally posted by Haze
I know it takes time for these things, but can you not get a temporary backup asap? I understand these things happen but you should have had a backup ready for when you took down Globix.

I do agree, but then how fast can somebody install a DS3 or OC3 or whatever? I think they oversighted the backup lines installation thinking that the OC12 would originally come in January anyway and it would take the same time for them to install a backup line? I know that VDI is ordering cogent that will kick in on March, so by March hopefully we all will get much more redundant connectivities and more peerings....

cheers,
:beer:

Yep, that why I said: " I Know it takes time for these things". Heck, It was worth a shot!?

No one can get a "provider in less then 60 days"

What blows my mind is the concept of a person who does billing in level3 to post a shutdown, and not "legal form of notice"

If they called and said "We don`t care and want the disputed portion paid in 30 days, then we would have no choice to pay that portion"

Here is some interesting reading..

http://www.carrierhotels.com/news/January2002/level30129.shtml

"look at this part"

"If the current rate of sales, cancellations, and disconnects were to continue, the company may violate a revenue-based financial covenant as early as the end of the second quarter,"


I really wonder at this point ....

William,
Thanks for the infomation. Is the current outage the same problem (the person to shutdown the acount came back in and did it again)? Hopefully, this outage will be shorter than last nights.....

VDI seems to be down again...

It is down again.

Ah..gosh...I was happy last night and this is only morning..

:angry: :rolleyes:

is this just "one of those things" that webmasters are supposed to deal with? there's no chance for recoup here?

just last night during the window when VDI went down and then was back up and this morning, I sent out advertising requests/information to potential advertisers. and then this morning, bam. obviously, my sites stability and reputation is diminished because of the downtime as the advertisers won't even be able to see the site. thousands of dollars lost.

there is nothing i can do, i imagine?

Three days ago I received 3 new clients for webhosting.

This morning, all of them called me and cancelled their accounts with me.

One of the saddest day in my life. :(

Level3 cut the connection again through human error?

Originally posted by voxtreme-matt
Level3 cut the connection again through human error?

All I know is that it was a Level3 problem..

The morning shift started, and pulled the line and then we called in, they put it back up.

Originally posted by William
The morning shift started, and pulled the line and then we called in, they put it back up.

Level3 won't be on my Christmas Card list... :rolleyes:

Level 3 is not as bad as they are made out to be here. Sorry. We have used them for 3 years at cari.net. Many can back me on this, their uptime is outstanding!

Also, no offense to VDI, but I have heard this all before from them. Does anyone rememeber the Qwest issue a while back??? It seems it is never the colo's fault, but rather the providers.

Theres is nothing on our end, you could even do some research about thier billing problems.

also there are public reports of this situation.

yes, level3 "data server is outstanding, but a company that is laying off 1/ 2 the force and about to file chapter 11, I think they are not in a position to give credits or deal with billing errors.

yes, level3 "data server is outstanding, but a company that is laying off 1/ 2 the force and about to file chapter 11, I think they are not in a position to give credits or deal with billing errors.

Debatable. However, if I had a single point of failure, I sure wouldn't dispute the bill until after the smoke clears. :)

We sent the dispute in, even though it was I assume blackmail..

yes, level3 "data server is outstanding, but a company that is laying off 1/ 2 the force and about to file chapter 11, I think they are not in a position to give credits or deal with billing errors.

I hope this doesn't mean you're going to drop your connection with L3 before you get this OC12 up! :confused:

Now why would they go and do that?! They have a business to run... having days of downtime is not exactly.. good for business :rolleyes:

Well they dropped Globix because they filed.. This is the second chance I have given VDI, and Im starting to loose faith again. Plus, we were lied to about multiple redundancy, which does not sit right for us or our customers at all. Not only by VDI, but by our provider that sells through VDI. The worst part about it, is I don't even think my provider new what was going down.

P.S. Sorry if I come off a bit wierd today ( And I appologize for any other day as well, while im at it ), I couldn't sleep a wink last night.

Originally posted by Haze
Well they dropped Globix because they filed.. This is the second chance I have given VDI, and Im starting to loose faith again. Plus, we were lied to about multiple redundancy, which does not sit right for us or our customers at all. Not only by VDI, but by our provider that sells through VDI. The worst part about it, is I don't even think my provider new what was going down.



Normally I would argue the point of this issue, but the oc12 is the same as 4 oc3 circuits being run to a location, the loop is protected in 4 pops, we sat down with verizion and ask about the event of failure, Oc12 and up are very important to the verizion corporate account, so they actually provide an SLA for response and 100% up time. So dropping wasted overhead in exchange for better equipment was a choice that needed to be made.

Globix situation was more than just "bankrupt" they lied about several items, and there was more to the story.

http://www.whafh.com./cases/globix.htm

I did not want any surprises and have an unexpected down fall and have to wait 4 months for another line.

However with the new network we can actually offer SLA`s and contract for clients and not worry. The corporate accounts would like to see more of that, as well as the freedom to resell 50 mbps or more.

The Level3 situation was a blind kick that no one saw coming, as well as them pulling the line on purpose was an insult. I wonder if they are looking for short-term immediate money or looking for a long-term relationship.

Thanks for responding Bill ( can I call you Bill, Bill? ) You put my mind at ease.

As usual Bill takes care of the situation in a professional manner. Don't worry guys you are in very good hands with VDI. Bill is definitely someone you can put your trust in. Best of luck on your connectivity issues.

Yeah.. good luck with the connection issues, VDI... That issue with Level3 sounded very -not- fun. Especially when I have a site hosted on the HR servers still at the VDI NOC.. and I try to update, and I'm blocked.. But at least It was not HR's fault.. this time