We currently have over 30 dual xeon "Total Control" servers at the planet. It currently appears that they are having significant packet loss at their routing borders.

A review of various monitors and traceroutes shows that their Global Crossing and AT&T circuits are experiencing major problems. Possible fiber carrier issues?

I have always received excellent service from the planet, but guys.... it would be nice for your night/weekend staff to be able to adjust BGP weights when issues like this happen until circuit analysis can be completed.

For reference, this is a datacenter wide issue effecting both SM & TP customers, also effecting LT and TS customers in the dallas datacenters.

Any news is welcome!

David

I am seeing the same thing with my servers as well.

See http://www.webhostingtalk.com/showthread.php?s=&threadid=335731&perpage=15&pagenumber=1

Horrible and severe amount of customer loss in last 2 weeks

same here, can't even get to their forums or orbit , network has been unreliable for the last month or so.

Everything about them is slow at the moment, network, support, sales.
I ordered additional ip's yesterday and still haven't got them.

We've been seeing the same thing. Personally, I know they are getting more lines and all but the tech support responses to the issue are what really bother me the most.

I call and get "The issue is with NLayer call them. You can get their phone number via a whois. We can't help you with this issue of latency and packetloss." What?!?! This your link, you could atleast admit to the issue. I dont have any pull with NLayer. You do. You purchase the line from them.

Suppor tickets get "We see the 50ms increase in ping times leaving ThePlanet's network, however this is normal and is not an issue. Its probably just an issue with your ISP as it has alot of hops." Since when did hops signify a huge latency issue???

Anyways, this isn't an issue I dont think wont be resolved, but I do think it could be handled a bit better. Maybe inform customers that yes, we are taxed right now on bandwidth. Yes we are working on it and blah blah blah is being done to fix this issue.

Communication is the key..........

They did inform customers, in their forums. The VP of networking (William Charnock) posted:

"We've notified nLayer about the packet loss on the link and are waiting for them to update us."

EDIT: That was 5 days ago BTW.

From what I can tell it seems to have improved. Can anyone else confirm this?

Looking at the nlayer graphs, it appears as though the daily peak period has passed and the nLayer link now has some free bandwidth. That could be a contributing factor.

My brief testing shows no packetloss, but I haven't noticed any throughout this whole thing... I'm going out via Above.NET anyhow.

75% of their lines reach their capacity at peek hours, it is bound to have packet loss or slow speeds... =(

Just take a look at the network graphs on orbit.

They are definitely showing irresponsible reactions to growth, they should have slowed sales or stopped them completely before things got to such a point.

Can anyone imagine how bad things would get if they lost even a single one of their 12 gig links? Fiber is not something that can be provisioned quickly, so how far along their build outs are would be very interesting to know. If they have truly been caught off guard it could be many weeks before a resolve is reached.

TP has done reasonably well with their growth, although their main selling point now is their network, and that is going down the drain, so there is not much reason to go with them anymore. Glad we started provisioning all our new servers with another provider months ago. :)

Originally posted by BitError
They are definitely showing irresponsible reactions to growth, they should have slowed sales or stopped them completely before things got to such a point.

Can anyone imagine how bad things would get if they lost even a single one of their 12 gig links? Fiber is not something that can be provisioned quickly, so how far along their build outs are would be very interesting to know. If they have truly been caught off guard it could be many weeks before a resolve is reached.

From what I've gathered from official announcements by William Charnock in the SM forums, they have 7 new GigE lines already installed and ready, they're just waiting for the DWDM (Is that right? Whatever, you get the idea) equipment to arrive. Looks to me like they only half got cought off-guard; they saw it coming, and tried to order more GigE lines, but they didn't see it coming far off enough.

They apparently don't want to make the same mistake again though, in 2005 I think they have two 10gbit lines in the works. They haven't mentioned what other GigE lines they plan to install in the longterm.

DWDM gear? Interesting. That makes me wonder if they are abandoning their OC48 Sonet strategy for GIG hauls for a less fault tolerant straight dense wavelength division multiplexing strategy with raw ethernet gig links over single fiber pairs.

DWDM gear basically spreads multiple wavelengths of light over the same strands of fiber.

If they get enough links in it shouldn't be a problem I suppose, I'd be interested to know the specifics. DWDM gear is pretty easy to come by, if you don't care about fault tolerance (especially to the level of frame level CRC's like within direct sonet) you can fit a full 20 gigabits over a single set of fiber strands. The only scary part being that you really cannot afford a fiber consistency interuption when you have that much data flowing over a single strand set.

Interesting in any case.

They actually have two 10gbit lines planned for next year. I guess buying 1gbit lines becomes a waste when you have 40+gbit (which is what they plan to have in 2005)

Yeah if they are waiting on 7 gigabit lines for a single piece of gear then to me it would sound like they are lighting all of them over a single fiber strand which is sort of what I had gathered from the time table set.

Perhaps it's merely a momentary measure to cover the immediate bandwidth crisis until they can get something else installed or perhaps my impression is entirely false. :D

The 10gbit lines are probably not through single providers, 10gbit is OC192 equivalent which is the largest backbone link that any Tier1 carrier runs right now. Cisco's next generation gear (CRS) is capable of doing 40gbit (OC768) but I am not aware of any providers yet running lines of this capacity.

Seems to be through single providers. From the horses mouth:

"we are in negotiations with 2 of our providers to have 10GbE ports turned up"

I read that as two 10GigE ports, through 2 providers.

That is indeed very interesting, if they do in fact complete the deals I'll be interested to see what providers are able to deliver on that type of capacity. Definitely not a normal circuit install to say the least.

I'd be more interested to know how they plan to do failover. Only way I can slice it is they're getting the two ports so that they can keep each at about 5gbit in case they need to take it offline.

Though, who knows, maybe the rest of their network will be able to absorb a 10GigE worth of traffic by then.

They also said they're getting more cogent lines up; the 2 that they have now are hitting 800mbit plus each. Which is funny because it was what, a month ago, that they were at 600mbit, maybe two months before they were at 400mbit. Must be selling like hotcakes.

If Cogent provides a discount down to 8$ per megabit on 2gbit commits, I wonder what they would charge on three or four gigabit commits.