WHOLE OF DV2 DOWN!!!

Geeeeze none of our servers, our friends servers or www.dv2.net replying to pings not just on my computer, or my colleagues but I tried around the world. 100% Network UpTime - HMMMMMMMM - just get it sorted guys :)

/me thinks maybe something to do with the email I receieved regarding sorting a slow link or something - but still.....

Anyone else in DV2 thats also down?

Same here, server down, using Transxactglobal

Well we did receive:

----------------------------------------------------------------------------------
Customers,
DV2 will be testing the redundancy of their bandwidth carriers, this is a standard test we perform every couple of months. You should not see any outages, however you might see what looks like congestion when the BGP tables update. This testing will begin at 0500 eastern and will be completed no later than 0600 eastern time zone on 5/9/2002. If you are seeing issues after 0600 please open a trouble ticket on the Tranxact Global site (www.tranxactglobal.com) or call our support line at 866-344-4382. Thanks

Scott Spiegel
VP of Engineering
-------------------------------------------------------------------------------

Open a trouble ticket on www.tranxactglobal.com - down as well :rolleyes:

Anywayz hopefully it will come back up at 0600 Eastern - part from this one incident though they have been a truley excellent facility - we're happy :D

The whole 209.51. seem to have routing problems. I have a server at ventures online and it's not reachable. DV2 down as well.

wish I'd had an email about it, how strange

Originally posted by hardweb
The whole 209.51. seem to have routing problems. I have a server at ventures online and it's not reachable. DV2 down as well.

Hmmmmmmmmm is ventures online marienta building as well (56?)

Hi ya

I just contacted Imran (Cabalstudios/eservers.biz) and he is going to look into it (I also got the above email).

IMHO I think dv2 started slightly early on the BGP tests and BGP wasn't working properly so they are probably fixing it.

If I hear any news I will post here.

James

I was informed yesterday along with all tranxact and eservers customers from Scott Speigel at DV2:

Customers,
DV2 will be testing the redundancy of their bandwidth carriers, this is a standard test we perform every couple of months. You should not see any outages, however you might see what looks like congestion when the BGP tables update. This testing will begin at 0500 eastern and will be completed no later than 0600 eastern time zone on 5/9/2002. If you are seeing issues after 0600 please open a trouble ticket on the Tranxact Global site (www.tranxactglobal.com) or call our support line at 866-344-4382. Thanks

Scott Spiegel
VP of Engineering

Down for me to... How did you contact Imran?

Originally posted by BurtonHost
I was informed yesterday along with all tranxact and eservers customers from Scott Speigel at DV2:

Customers,
DV2 will be testing the redundancy of their bandwidth carriers, this is a standard test we perform every couple of months. You should not see any outages, however you might see what looks like congestion when the BGP tables update. This testing will begin at 0500 eastern and will be completed no later than 0600 eastern time zone on 5/9/2002. If you are seeing issues after 0600 please open a trouble ticket on the Tranxact Global site (www.tranxactglobal.com) or call our support line at 866-344-4382. Thanks

Scott Spiegel
VP of Engineering

Same email as the one I posted above oh well waste WHT bandwidth why dont you :stickout

Originally posted by Myacen
Down for me to... How did you contact Imran?

www.cabalstudios.com has a contact number - it's his UK mobile number so overseas people might find it expensive - BUT you can get hold of him.

He's also on ICQ but not at his PC at this moment.

James

I'm half asleep.. didn't even see that lol..

Well.. at least we know why and it's a PLANNED outage.

Matthew

Thanks still waiting for our servers :P

Originally posted by BurtonHost
I'm half asleep.. didn't even see that lol..

Well.. at least we know why and it's a PLANNED outage.

Matthew

It was planned maintenance - not outage ;)
BGP should have kicked in and re-routed traffic - but it hasn't.
It's kinda good in a way as they can fix the fault so it won't happen if a backbone DOES go down.

The fact it started about 30mins too soon is another thing :)

James

Originally posted by BurtonHost
I'm half asleep.. didn't even see that lol..

Well.. at least we know why and it's a PLANNED outage.

Matthew

ROFL :P

While we are at it has anyone else receieved complaints that visitors sometimes find servers @ DV2 slow? It seems to be netsaint (i think thats what they are called anyway ;) ) hop slowing it right down - anyone else seen this? Happens on all our servers...

It's just come back up

James

Ok its back up as well :D (Sorry for the triple post ;) )

They probably just had to re-build the BGP tables...

Originally posted by terrastudios
While we are at it has anyone else receieved complaints that visitors sometimes find servers @ DV2 slow? It seems to be netsaint (i think thats what they are called anyway ;) ) hop slowing it right down - anyone else seen this? Happens on all our servers...

Yes - been having reports too - it's fast for me, fast for most of the UK and US, but France, Italy and Spain are reporting taking a long time to load pages.

I asked Imran about it earlier this week and he said something about Alter.net had massive problems - even Demon Internet were saying that.
But from the tracerts I looked at, it is NetRail that is slowing things down. Just before it hits the NetRail hop, the reply time is about 80ms - as soon as it hits NetRail it hits 500+ms.

James

She's back

Not NetSaint NetRail <<< DUGHH!!!!

Looks like we are back down again :(

Like myself after some Viagra - it is still up for me :)

Also like Viagra it wears off too soon, and it is down again :D

James

Down for me again :)

yes, it's up and down :)

down again here too.
I'm still wound up that I didn't get any advance warning that there was a planned outage, or I coulda at least put up some news for site visitors in advance.
Ah well, the service has been awesome since day one for me, and as far as speed goes, I've had nothing but good reports from visitors/users.

Originally posted by terrastudios
While we are at it has anyone else receieved complaints that visitors sometimes find servers @ DV2 slow? It seems to be netsaint (i think thats what they are called anyway ;) ) hop slowing it right down - anyone else seen this? Happens on all our servers...

The connection gets really slow at times to Sweden. There were times that we had pings over 700ms and about 60-70% ping timeouts!.. Most of the time, it only lasted for about 30-60 minutes per day, but a couple of times it was slow like that for over 24hs. This slowness prevented us from doing business locally :bawling:

We are now with another provider.

does SLA apply anymore?

yeh but i doubt the SLA 100% will apply to scheduled maintence :)

Hmmmm
tbh scheduled network maintenance is good. What we have learnt from this is that if there was an actual problem, then BGP WILL kick in and take over.
I think before this maintenance test BGP wasn't working properly so this has confirmed and fixed it.

James

Everything,

Is up and running sweetly now, this was a Planned outage, for the clients that didnt get an email, apologises on our behalf.


Kind Regards
Imran

Guess the email didn't reach everyone. Just for the record, per server monitor records, total downtime was two brief time periods for less than 45 minutes total. Let's see....three months, no unscheduled downtime, scheduled maintenance under 45 minutes...I'll take that.

BGP was fine before - we added a provider late last week and wanted to make sure they were handling and advertising us correctly. Well - they were not - we isolated their problem and they are fixing - actually for you techies - their IBGP tables were not propagating and exchanging routes internally - so we were not able to reach their gateways. So - hence - splat. The reason for splat is that we simulate catastrophic failure by shutting each connection in sequence one by one and see how the traffic flows. they were the last one (newest always goes last) - and they did not handle it.

We will run another test when we think they are all done - thanks.

Thanks Sailor - btw please change your nick... it is seriously giving me an image of you dancing on a deck with a mop and bucket singing camp sailor songs ;)

--James

ROFLMFAO @ wildwayz

OUCH - maybe I should escape and go up to the boat today - checking online today is brutal......

Originally posted by sailor
OUCH - maybe I should escape and go up to the boat today - checking online today is brutal......

LOL you know I am only joking, right? :)

James

Can I ask why I didn't get an e-mail concerning this issue? My site is tgnetwk.com and primary e-mail address is furton@tgnetwk.com

Hmmm not sure
who is it u have your server with? DV2 or eServers? Mine was eServers.

James

Originally posted by cabalstudios
Everything,

Is up and running sweetly now, this was a Planned outage, for the clients that didnt get an email, apologises on our behalf.


Kind Regards
Imran

That's great why don't you answer emails? Or ICQ? When will my servers arrive? How come you post here and delay server deliveries?. 5 Times now, and you won't get back to me.

You ARE costing us money. Can you please hurry up with our servers. Sorry it has come to this point Imran & Shazad. The way you have handled our account is poor. You are not responding to my requests and now I will need to contact you through here to get some action.

dv2.com

Myacen .. Same thing happening to us too.. :(
eServers was great before, what's starting to happen to them??? :(

They got popular - their last special offer brought loads of new clients :/

James

Originally posted by Myacen


That's great why don't you answer emails? Or ICQ? When will my servers arrive? How come you post here and delay server deliveries?. 5 Times now, and you won't get back to me.

You ARE costing us money. Can you please hurry up with our servers. Sorry it has come to this point Imran & Shazad. The way you have handled our account is poor. You are not responding to my requests and now I will need to contact you through here to get some action.



I've been waiting 12 days for a server from tranxactglobal....

Originally posted by sailor
BGP was fine before - we added a provider late last week and wanted to make sure they were handling and advertising us correctly. Well - they were not - we isolated their problem and they are fixing - actually for you techies - their IBGP tables were not propagating and exchanging routes internally - so we were not able to reach their gateways. So - hence - splat. The reason for splat is that we simulate catastrophic failure by shutting each connection in sequence one by one and see how the traffic flows. they were the last one (newest always goes last) - and they did not handle it.

We will run another test when we think they are all done - thanks.

I can honestly say that in thirteen years, I've never seen anyone test BGP failover on a live network. I'm not making a judgment call, but since BGP failover is a passive event, not an active one, you can verify that your routes are being propagated simply by going to a few routeview servers and looking them up. If the routes are there from each provider you use, then failover will work. No "live" testing required.

I'm not criticizing, mind you.

Kevin

I talked to jeff/imran.. its ok now.. server should be up soon...
:)

Originally posted by sigma


I can honestly say that in thirteen years, I've never seen anyone test BGP failover on a live network. I'm not making a judgment call, but since BGP failover is a passive event, not an active one, you can verify that your routes are being propagated simply by going to a few routeview servers and looking them up. If the routes are there from each provider you use, then failover will work. No "live" testing required.

I'm not criticizing, mind you.

Kevin

What can I say - I am a skeptic. - I dont believe anything that is "supposed to be that way" - I like to flip switches every now and then and actually see for myself. this was a problem that our routes were being advertised by an upstream border router - but they had problems with IBGP between local routers - therefore - we saw the routes - the world saw the routes - however - their routers internally would not pass us from one to the other - wierd little problem that does not come up often - only way to find it is to physically test it. I do this as a matter of practice. I am also one of those guys that if somone tells me that a device is fully redundant and a card can go away and no one will notice - watch out - I am going to physically test it. If I see people get a horrified look on their face - I give them another chance to change their position - or take responsibility if it goes down. they better be right. appearances can be deciving. :)

Originally posted by sailor

What can I say - I am a skeptic. - I dont believe anything that is "supposed to be that way" - I like to flip switches every now and then and actually see for myself. this was a problem that our routes were being advertised by an upstream border router - but they had problems with IBGP between local routers - therefore - we saw the routes - the world saw the routes - however - their routers internally would not pass us from one to the other - wierd little problem that does not come up often - only way to find it is to physically test it. I do this as a matter of practice. I am also one of those guys that if somone tells me that a device is fully redundant and a card can go away and no one will notice - watch out - I am going to physically test it. If I see people get a horrified look on their face - I give them another chance to change their position - or take responsibility if it goes down. they better be right. appearances can be deciving. :)

You don't drive into telephone poles to test your car's airbags, do you? ;)

All kidding aside, you're free to try to inflict any failure mode you care to, on your own network. Have fun with it.

Kevin

no definately not - but then again - if I did - they might not work - if they had a button I could press to test them and then have them retract - you bet I would. I do test my antilocks though - which is I think a better comparison. Any way - I know it is not for everybody - but we feel like it is an extra step to ensure uptime - and really make sure things do what we claim.

I'm sure you don't do too many of these tests, but just remember, even if it is just flicking a switch for you, for the people with servers at your facility it might be a very big deal. Especially if they don't get proper warning well in advance, a "test" which takes your server offline (seems) just as bad as a real network failure - customers calling (and you don't have the information to tell them), etc. etc. I can see how it would be pretty frustrating.

it is better for us to find out in a controlled situation that might last for a short time - than to find out in an emergency that could take longer.

I agree with Sailor on this. I also agree that more warning should have been given out - not sure why some got the email and not others - I got it about 8 hours before it was carried out.

James

it is better for us to find out in a controlled situation that might last for a short time - than to find out in an emergency that could take longer.
It's a controlled situation for you because you know what's going on. If you don't make 100% sure your customers know there is a planned outage well in advance, how long it will last, etc. it is *not* a controlled situation as far as they're concerned - it's an emergency outage.

(I understand what you're saying about testing - just sounded like you didn't take the relationship with your customers seriously enough - as a customer elsewhere I will give a company a lot of leaway if I know what's going on, but nothing frustrates me more than someone flicking a switch without notification which leaves me scrambling not knowing what failed where, etc... nothing worse than finding your site/server down unexpectedly)