View Full Version : BurstNET down ?!?
Pavlov 12-20-2001, 01:41 PM Can u guys please try going to http://www.burst.net
Tracert is not going through.
It was down (along with our servers) from ~ 17:15 GMT
Seems like everything at their data-center is down.
Am I right?
michaeln 12-20-2001, 01:46 PM Yes,
They said it would be back up in a few minutes.
Time will tell...
Pavlov 12-20-2001, 01:50 PM U r lucky to get through to them - I am still waiting for an answer on the phone.
It's been down for 28 minutes so far.
This totally not acceptable!
-no AOL IM
-no phone (ext: 7, 8, or 11)
-no status anywhere....
NOTE: though i did hear from a source that it is a Sprint issue... of course that is hear say so not sure....
Pavlov 12-20-2001, 01:54 PM Just how long can a few minutes last?
michaeln 12-20-2001, 01:55 PM That is what I felt.
I called and when the guy answered I asked, "so how is your day going?"
he says, "Not good"...
LOL
I can see why, neither is mine...
Anyway he told me someone flooded the system. I don't understand how that could have taken down a WHOLE datacenter but whatever.
He did assure me it would be back up in a few minutes.
I just checked and still can't get to my site....
Pavlov 12-20-2001, 02:17 PM It's back up!
After all, it wasn't all that bad.
If someone from BurstNET reads this. We forgive u guys ))) But make sure this doesn't happen again.
I have to admit, that this is quite unusual for them.
I'd love to find out what happened there. Anybody knows?
mdrussell 12-20-2001, 02:28 PM BurstNET down again? I can't reach them from here.
Pavlov 12-20-2001, 02:31 PM No, it's definitelly up now.
michaeln 12-20-2001, 02:38 PM They are down for me again....
MilkMan 12-20-2001, 02:44 PM Down here too
Pavlov 12-20-2001, 02:45 PM Actually u r right, it went up for a while and now it's down again.
Oh well...
mahinder 12-20-2001, 03:28 PM it down because my pager is ringing every 5 minutes. ;)
anyway i am sure they must be working hard to fix it. I feel cool because its not my server which is down it noc problem so it will be automatically solved when they are up.
downtime is truth of life lets face it. !!
Arsalan 12-20-2001, 03:43 PM any update on their status ?
jeremiah23 12-20-2001, 03:49 PM Still Down and ETA of over 2 hours is expected, I have customers that are threatning to kill me if there site doesn't come up. I am mad as all get out. They won't give a straight answer.
mahinder 12-20-2001, 03:53 PM Originally posted by jeremiah23
Still Down and ETA of over 2 hours is expected, I have customers that are threatning to kill me if there site doesn't come up. I am mad as all get out. They won't give a straight answer.
take a recorder and record there calls!!.
what the heck they are hosting with you!! is it so important for them :(
DigitalXWeb 12-20-2001, 04:02 PM You mean they only have one pipe coming into their NOC?? (If it is a sprint issue as suggested above). I'm sure they will explain after the issue is resolved. Good luck to all those affected.
Brian
bteeter 12-20-2001, 04:05 PM They were up the first time I read this thread. Now they are down again - and so is everything hosted there.
Sounds like we will all be getting some service credits this month...
Take care,
Brian
Arsalan 12-20-2001, 04:06 PM Any reason give ?
alchiba 12-20-2001, 04:24 PM I spoke to them about 90 minutes ago and was told that a server there fell victim to a DoS attack. They took the server offline but the attack moved to yet another machine. They were re-loading the BGP tables at the time of my call. I don't know how long that takes.
Yes, I too, was led to believe they have muliple feeds. Guess not.
Ahh, there's my pager now. . . must be back up again.
TotalHst 12-20-2001, 04:25 PM Running a trace to their domain doesn't go anywhere so this tells me their connection(s) is down or their main routing device is down. I do not know how their network is setup, but if they truely do have 2 pipes when one goes down the other should take over.. It maybe that they are using a one single cisco router using Cisco BGP technology which basicly switches between connections and takes the fastest route out, if one goes down everything is pushed out the other pipe. So if their network is setup this way all it takes is for their main router to go down and everything is down. Anyone have an ip I can traceroute to rather then trying trace to the domain?
bteeter 12-20-2001, 04:37 PM Originally posted by TotalHst
Running a trace to their domain doesn't go anywhere so this tells me their connection(s) is down or their main routing device is down. I do not know how their network is setup, but if they truely do have 2 pipes when one goes down the other should take over.. It maybe that they are using a one single cisco router using Cisco BGP technology which basicly switches between connections and takes the fastest route out, if one goes down everything is pushed out the other pipe. So if their network is setup this way all it takes is for their main router to go down and everything is down. Anyone have an ip I can traceroute to rather then trying trace to the domain?
web1.assortedinternet.com
66.96.208.10
TotalHst 12-20-2001, 04:39 PM They are now back up :)
| 13 | 70 | 144.232.191.26 | sl-burst-2-0.sprintlink.net | - | | 104 | -x--- | Sprint/United Information Service |
70 % packet loss on sprint, looks like they aren't redundent
RackMy.com 12-20-2001, 04:59 PM This is what they say
"DS3 Connectivity thru Sprint™ & AT&T™"
allan 12-20-2001, 06:35 PM A traceroute to the IP posted earlier does go through AT&T:
traceroute to 66.96.208.10 (66.96.208.10), 30 hops max, 38 byte packets
1 f4-1-rt2.minerva.net (206.131.224.1) 1.284 ms 2.165 ms 0.828 ms
2 f0-0-rt1.minerva.net (206.131.224.193) 0.852 ms 0.865 ms 0.639 ms
3 500.Serial1-11.GW4.TCO1.ALTER.NET (65.207.88.5) 3.066 ms 2.590 ms 2.563 ms
4 118.at-3-0-0.XR1.TCO1.ALTER.NET (152.63.39.14) 2.845 ms 2.984 ms 2.873 ms
5 293.at-7-0-0.XL1.DCA8.ALTER.NET (146.188.162.237) 3.756 ms 3.578 ms 3.712 m
s
6 POS6-0.BR1.DCA8.ALTER.NET (146.188.162.209) 3.683 ms 3.599 ms 3.716 ms
7 uu-gw.wswdc.ip.att.net (192.205.32.133) 3.838 ms 4.465 ms 3.709 ms
8 gbr3-p50.wswdc.ip.att.net (12.123.9.50) 4.133 ms 4.042 ms 3.894 ms
9 gbr3-p20.n54ny.ip.att.net (12.122.3.53) 9.903 ms 9.537 ms 9.492 ms
10 gbr5-p60.n54ny.ip.att.net (12.122.5.105) 10.521 ms 10.613 ms 10.347 ms
11 ar2-p310.cdknj.ip.att.net (12.123.192.9) 11.449 ms 10.648 ms 10.490 ms
12 12.119.155.82 (12.119.155.82) 22.434 ms 28.953 ms 22.762 ms
13 66.96.208.10 (66.96.208.10) 29.221 ms 23.551 ms 23.539 ms
TotalHst 12-20-2001, 06:41 PM Then they do have BGP, because it is taking the fastest route from you to them.. I just still don't see why sprint being down completely brought their whole network down..
allan 12-20-2001, 06:48 PM Originally posted by TotalHst
Then they do have BGP, because it is taking the fastest route from you to them.. I just still don't see why sprint being down completely brought their whole network down..
I could be any number of reasons (all these are speculative, I have no information about their outage):
1. They have both routes, running into the same router, which died.
2. They have two routers, running into a single switch, that died.
3. They may have configured their BGP community string or weighting improperly and the updates that the router was unavailable were not being properly propagated.
4. They may not have had their BGP filters configured properly.
Same here with AT&T:
4 dal-edge-07.inet.qwest.net (65.112.176.17) 2.730 ms 2.127 ms 2.059 ms
5 dal-core-02.inet.qwest.net (205.171.25.57) 1.744 ms 2.368 ms 3.094 ms
6 dal-core-03.inet.qwest.net (205.171.25.138) 3.042 ms 2.352 ms 3.102 ms
7 iah-core-03.inet.qwest.net (205.171.5.169) 6.804 ms 7.388 ms 6.368 ms
8 atl-core-01.inet.qwest.net (205.171.8.146) 25.849 ms 26.549 ms 27.215 ms
9 atl-core-02.inet.qwest.net (205.171.21.150) 28.900 ms 26.183 ms 26.729 ms
10 atl-brdr-03.inet.qwest.net (205.171.21.102) 26.958 ms 25.682 ms 26.400 ms
11 205.171.1.30 (205.171.1.30) 26.131 ms 27.199 ms 27.368 ms
12 gbr4-p50.attga.ip.att.net (12.123.20.254) 27.436 ms 25.948 ms 26.373 ms
13 gbr4-p30.wswdc.ip.att.net (12.122.2.226) 37.422 ms 36.983 ms 37.000 ms
14 gbr3-p60.wswdc.ip.att.net (12.122.1.129) 37.737 ms 37.227 ms 36.846 ms
15 gbr3-p20.n54ny.ip.att.net (12.122.3.53) 44.980 ms 45.712 ms 45.178 ms
16 gbr6-p60.n54ny.ip.att.net (12.122.5.113) 45.412 ms 44.791 ms 44.903 ms
17 ar2-p3110.cdknj.ip.att.net (12.123.192.17) 46.807 ms 45.495 ms 46.006 ms
18 12.119.155.82 (12.119.155.82) 61.547 ms 68.262 ms 60.594 ms
19 66.96.208.10 (66.96.208.10) 59.198 ms 60.581 ms 59.458 ms
Pavlov 12-20-2001, 07:43 PM I am glad everything is back to normal. Lets not crtiticise BurstNET. It seems to me that it was something out of their control.
They do provide good value for money. That is a fact.
OK. We will loose a few clients after this incident. We'll manage.
Also, they are constantly improving. The reason they are discussed so much in forums like this one is because they r popular. Everytime smtng goes wrong it gets here.
addaction 12-20-2001, 08:33 PM Their network is down again, if anybody asks..... :(
method5 12-20-2001, 08:44 PM Down again..
One Web 12-20-2001, 08:49 PM OMG this is bad. From noon (EST) to 1pm and then from 1:30pm to 3:20pm and now again from 7:30pm to God knows when.
bteeter 12-20-2001, 08:52 PM Originally posted by antireality
Down again..
Yup. It'd be nice if they could send a note to their clients explaining what is going on...
Brian
addaction 12-20-2001, 08:58 PM I agree with that Brian.... Burst do send an explanation to your customers, so we can tell our customers what is happening...
Kylecool 12-20-2001, 09:01 PM Man that totally sucks. I'm sure once it's done with, Sean or someone will post here about it... Seems like many people are perturbed....
-Kyle
BurstNET 12-20-2001, 09:17 PM router hardware problems, we are working on it.
Sean R.
BurstNET
bteeter 12-20-2001, 10:26 PM Any ETA for getting back online - at least in someway?
Thanks,
Brian
BurstNET 12-20-2001, 10:45 PM We have swapped out some of the hardware with our spare replacement parts we keep in stock, but they are not supporting the load fully. We have it holding now we believe, but may require a few router reloads throughout the night.
We will be back to full stability by 11am est, when some replacements parts arrive from cisco in ups guaranteed early am delivery.
Sean R.
BurstNET
Kylecool 12-20-2001, 11:13 PM 11AM? That's not for a while, lol. Downtime in the hosting industry can loose BIG BUCKS. :(
Let's just hope it holds..... Nothing goes down anymore.. :)
-Kyle
bteeter 12-20-2001, 11:49 PM Originally posted by BurstNET
We have swapped out some of the hardware with our spare replacement parts we keep in stock, but they are not supporting the load fully. We have it holding now we believe, but may require a few router reloads throughout the night.
We will be back to full stability by 11am est, when some replacements parts arrive from cisco in ups guaranteed early am delivery.
Sean R.
BurstNET
Sean,
Thanks for the update. I would definitely agree that it is not holding up to the load. This is what I've been getting for the past several hours:
C:\Documents and Settings\Brian Teeter>ping -n 100 66.96.208.10
Pinging 66.96.208.10 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 66.96.208.10:
Packets: Sent = 45, Received = 0, Lost = 45 (100% loss),
Control-C
^C
Please keep us in the loop.
Thanks - Brian
Daniel 12-21-2001, 12:38 AM I hate to post anything in a thread like this but Sean PLEASE let us know what is going on. We have been up 4 of the last 12 hours. And are still down now (11:30 EST). This is our livly hood we are talking about here. Throw us a bone once in a wile PLEASE.
BurstNET 12-21-2001, 12:52 AM router has been stable for the last several hours.
we have to reload it every 60 minutes or so, which causes about 30 seconds downtime though...
if your server is still down, contact support, cause it may not be a network issue, as the network has been up for the most part for the past several hours. Hardware shipment first thing in morning will fix it for good, and we'll be back to normal.
Sean R.
BurstNET
Daniel 12-21-2001, 12:57 AM I have 4 servers with you they all came backup about 10 min after I made the post. Thanks Sean!!
allan 12-21-2001, 01:00 AM Originally posted by BurstNET
We have swapped out some of the hardware with our spare replacement parts we keep in stock, but they are not supporting the load fully. We have it holding now we believe, but may require a few router reloads throughout the night.
We will be back to full stability by 11am est, when some replacements parts arrive from cisco in ups guaranteed early am delivery.
Sean -- this is not a dig, but are you saying that you have both connections running into the same router? Doesn't that defeat the purpose of redundant connections?
Arsalan 12-21-2001, 01:18 AM this is going to be a long night/day
TotalHst 12-21-2001, 01:48 AM So our assumptions were correct it was the main BGP router... As far as how BGP works, one router handles both connections and takes the fastest route out. So when running BGP it is imperative that you have a complete back up router.
Kylecool 12-21-2001, 01:51 AM Originally posted by TotalHst
So our assumptions were correct it was the main BGP router... As far as how BGP works, one router handles both connections and takes the fastest route out. So when running BGP it is imperative that you have a complete back up router.
Yep, I guess so!
-Kyle :cool:
allan 12-21-2001, 02:14 AM Originally posted by TotalHst
So our assumptions were correct it was the main BGP router... As far as how BGP works, one router handles both connections and takes the fastest route out. So when running BGP it is imperative that you have a complete back up router.
That's not necessarily true. You can, and should, have two routers each with a diverse connection to the Internet:
http://www.mentortech.com/learn/welcher/papers/bgp1.htm
Behind the routers, you can put two multilayer switches. Each switch would have a connection to both routers. They can run OSPF (or some other intranetwork protocol) internally and the switches will direct traffic to the appropriate router based on the number of hops, which it learns from the routers.
This leaves you pretty well fully meshed and redundant, at least at the network level.
Tetraboy 12-21-2001, 02:20 AM From burstnet's site..
The facility is centered on a Cisco™ 7206VXR Router,
with the new
I guess they don't.
Daniel 12-21-2001, 02:39 AM I just talked to burst the sprint line is working but the att line is not. I can ping cpanel.net fine over the sprint line. But as luck would have It I connect to my servers over the att line. They are going to see if they can change everything to the sprint line. That will get it "up" for a wile I HOPE.
Tetraboy 12-21-2001, 02:46 AM The att line appears to be working fine for me. And from serverhosts traceroute so does the sprint.
BurstNET 12-21-2001, 02:57 AM AT&T is having problems upstream somewhere....it's not us. I checked from about 20 locales across the world, and most can get thru on AT&T to us....has to be a failure point on their network somewhere though...
When it rain it pours....bad day all around :-(
Sean R.
BurstNET
Tetraboy 12-21-2001, 03:01 AM Dunno if it helps but I can reach it via ATT in Atlanta.
Samuel Mann 12-21-2001, 04:49 AM There is considerable amount of conversation about the statement of multiple back bone connections.
Burst since you have taken the time to post into this thread, could you please clarify the statement of multiple connections?
There is a lot of doubt I believe forming and some openess about the actual specifics of the topology of the network might be in order.
Again thank you for your reply.
TotalHst 12-21-2001, 04:54 AM We already know that they have 2 connections. One from Sprint, and one from ATT. Because some people on this board are going through the ATT route while others going throught the Sprint route. They are using BGP technology, and the downtime was cause by their main cisco switch failing which nocks both their connections out.
Samuel Mann 12-21-2001, 05:06 AM Originally posted by TotalHst
We already know that they have 2 connections. One from Sprint, and one from ATT. Because some people on this board are going through the ATT route while others going throught the Sprint route. They are using BGP technology, and the downtime was cause by their main cisco switch failing which nocks both their connections out.
Ahh so the switch failed. One point of failure though imo.
So the multiple connection statement has to do with one connection for these customers, and one connection for those?
Or both can use one connection if the other fails. A backup switch comprable at that point of failure was not existing on site?
TotalHst 12-21-2001, 05:48 AM The way BGP technology works is, it takes the fastest route from the server to the person trying to intercept the data. So people from different locations will be taking the fastest route from them to Burst. If one connection goes down obviously the faster route is the connection that is live. The connections were not the issue today. The main BGP router had major hardware failure, and the parts they had on spair were having issues handling the load, so they are being forced to do heavy monitoring on the router until new parts come in tommorow. This is what I took as the situation that has developed. I have not spoken to Burst about this but from the bits and pieces i have gathered from Burst customers and from my research this is what I can conclude.
Samuel Mann 12-21-2001, 06:04 AM Originally posted by TotalHst
The way BGP technology works is, it takes the fastest route from the server to the person trying to intercept the data. So people from different locations will be taking the fastest route from them to Burst. If one connection goes down obviously the faster route is the connection that is live. The connections were not the issue today. The main BGP router had major hardware failure, and the parts they had on spair were having issues handling the load, so they are being forced to do heavy monitoring on the router until new parts come in tommorow. This is what I took as the situation that has developed. I have not spoken to Burst about this but from the bits and pieces i have gathered from Burst customers and from my research this is what I can conclude.
Thank you for the synopsis!
As it is signifigant backup hardware is the problem here. This being the case I can understand to some degree. It is impossible to foresee every possible event a complicated machine such as a datacenter could be under.
Good luck Burst, a lot of people are interested in seeing you smooth this out.
TotalHst 12-21-2001, 06:05 AM Just talked to BurstNet tech support 2:00 AM PST. DoS attack just fried ram chip in main BGP router... The router was already having issues keeping up with the load this does not help you guys make it through till 11:00 am.. Most sites inaccessible, this router was already having issues keeping up with the load now that ram chip fried it will have even a worse time. From what the tech told me they are having him clear out BGP tables so that the router does not crash all together. Just thought I'd update you directly from BurstNET. Its to bad some hacker saw this as a great time to launch a full on DoS attack..
bteeter 12-21-2001, 09:39 AM Their network is obviously being saturated - but they are back online. I've been able to get to my server. But, I have to be REAL patient.
Is it 11am yet? I wonder how long it will take to get that new router hooked up and in the mix once they get it...
Take care,
Brian
Daniel 12-21-2001, 10:02 AM I talked to them a few times this morning. They are just wating on ram from cisco. The don't have enough memory to load all the BGP tables so they have to reboot each time before they all load or it will go down. That's what they said anyway. I have not been able to get to any of my servers longer than 5 or 10 min at a time in the last 14 hours.:bawling:
Arsalan 12-21-2001, 11:01 AM Any updates ? Any one ? this is taking too long to resolve!
Mike Shanks 12-21-2001, 11:05 AM It appears to be up for us now, its nearly 10am, do you reckon they could have got the delviery early?
Mike Shanks 12-21-2001, 11:09 AM woops spoke to soon,
The good thing is, I have been able to get ahold of burst everytime I have called them so far, I'm sure nick and sean will get it fixed as soon as physically possible for them.
Mike
Arsalan 12-21-2001, 11:46 AM Any time estimate given till their network is backup?
michaeln 12-21-2001, 12:10 PM I just got off of the phone with BurstNet....
FedEx hasn't arrived with the parts yet....
Still waiting.
I suppose that means either FedEx is running late or it will not make it until the 2:00 PM delivery this afternoon... :(
Michael
Arsalan 12-21-2001, 12:18 PM ummmmmm... thanks for the update... but shouldent they have bought some directly from a retailer on a urgent basis. instead of waiting for cisco? I dont know how it would cost more to buy the ram, then to give credit to every clinet they have....
michaeln 12-21-2001, 12:24 PM I certainly don't understand the logic. All I know is out of the last 24 hours my site has had approx 4 to 6 hours of uptime altogether...
It certainly isn't working at the moment and I have clients screaming at me... I can say this for BurstNet. Up until now I haven't had any realy problems, and throughout all of this every time I call someone does answer the phone and give me an update.
I just wish that this never happened, but then again don't we all... ;)
Michael
Mbarb 12-21-2001, 12:40 PM I may be wrong but I don't think you find this kind of stuff on the shelf at compUSA...
BTW.. my sites are down as well
Originally posted by Arsalan
ummmmmm... thanks for the update... but shouldent they have bought some directly from a retailer on a urgent basis. instead of waiting for cisco? I dont know how it would cost more to buy the ram, then to give credit to every clinet they have....
Samuel Mann 12-21-2001, 01:17 PM "The Network Operations Center utilizes ONLY top name brand hardware. The facility is centered on a Cisco™ 7206VXR Router"
several more paragraphs of blah blah blah...then we see...
"An additional Cisco™ 7200 series router is currently planned for install 4th quarter 2001, as a fail-safe for the primary router"
they were cutting it a little close weren't they?
Arsalan 12-21-2001, 01:20 PM they cut it too close... i guess they knew something is going to happen this week :)
Regarding the buy it from the retailer statement, a 500 mile road trip would cost much less then 24 hours of downtime with as many clients as burst.net has...
Arsalan 12-21-2001, 01:21 PM btw, can any one call them up and ask em whats up ?
Daniel 12-21-2001, 01:37 PM They still haven't received the part. They said they would have it back up this evening. I asked what plan B is but I don't think they have one. I mean what if FedEx does not come what if they drive the truck off the bridge or something!! If it were me I would have had someone sitting at a Cisco dealer when they opened this morning. But that's just me I would have a redundant plan.
Samuel Mann 12-21-2001, 01:49 PM As I was entering the limousine before flying back home from Chicago I was asked by one of the investors "So what is making it tough on everyone there?" I said slowly, "Access to the money we need to be a web hosting company"
8 months ago 2 weeks before the doors closed.
Hope it is not like that their Burst....I would feel your pain.
mahinder 12-21-2001, 02:30 PM just refunded 100's of dollars, some of my clients are claming they are not able to reach there web sites at all.
also i get 1 server down alert every 1 hours. Now i am going MAD, should burst have keept extra router,
how much this damn router cost any ideas ?
:(
bteeter 12-21-2001, 02:37 PM Originally posted by mahinder
just refunded 100's of dollars, some of my clients are claming they are not able to reach there web sites at all.
also i get 1 server down alert every 1 hours. Now i am going MAD, should burst have keept extra router,
how much this damn router cost any ideas ?
:(
ALOT. A basic Cisco 1700 series costs roughly $2000 new. I cannot imagine that a 7000 series that they have would be less than $10000 new...
Take care,
Brian
TotalHst 12-21-2001, 02:39 PM When running BGP on a cisco router, it is a heavy load. This is why it is immparitive that you have all the parts you need when somthing fails. I would suggest having a clone router ready to go, they were obviously not prepared for the main router frying hardware. Somthing that would have costs them a few 1,000 is costing them much, much, more in client loss.
Arsalan 12-21-2001, 02:45 PM gona go ta sleep and let some one else listen to the phone calls, perhaps in 8-9 hours there will be something better to listen on the phone... perhaps some friendly girly sounds ..... :pimp:
traceroute to 66.96.218.2 (66.96.218.2), 30 hops max, 40 byte packets
1 Cisco-M-Fe0-0.Space.Net (195.30.0.126) 0.478 ms 0.329 ms 0.530 ms
2 Cisco-M-XII-Fe0-0.Space.Net (195.30.3.210) 0.445 ms 0.482 ms 0.306 ms
3 Cisco-F-V-POS6-1-0.Space.Net (195.30.3.34) 6.697 ms 6.402 ms 6.118 ms
4 frankfurt41.de.eqip.net (195.206.66.61) 7.477 ms 7.026 ms 6.891 ms
5 frankfurt5.de.eqip.net (195.206.64.89) 7.295 ms 7.648 ms 7.130 ms
6 frankfurt50.de.eqip.net (195.206.65.149) 7.610 ms 9.331 ms 8.935 ms
7 newyork51.us.eqip.net (195.90.64.190) 89.315 ms 89.090 ms 89.907 ms
8 500.POS2-0.GW3.NYC4.ALTER.NET (157.130.22.149) 91.172 ms 89.986 ms 89.578 ms
9 149.ATM4-0.XR2.NYC4.ALTER.NET (152.63.24.138) 89.640 ms 89.723 ms 89.428 ms
10 188.at-1-1-0.XR2.NYC9.ALTER.NET (152.63.18.122) 90.251 ms 89.625 ms 89.634 ms
11 0.so-4-0-0.XL2.NYC9.ALTER.NET (152.63.9.90) 91.672 ms 89.782 ms 89.595 ms
12 0.so-2-0-0.TL2.NYC9.ALTER.NET (152.63.23.133) 89.561 ms 89.830 ms 89.646 ms
13 * * *
14 * * frankfurt41.de.eqip.net (195.206.66.61) 7.120 ms !H
15 * * frankfurt41.de.eqip.net (195.206.66.61) 7.240 ms !H
16 * frankfurt41.de.eqip.net (195.206.66.61) 7.593 ms !H *
17 neo.hostingmadeeasy.com (66.96.218.2) 115.405 ms 115.160 ms 115.015 ms
--- look maaa my connections bouncing off frankfurt! ;)
mahinder 12-21-2001, 02:46 PM few 1000's of dollars, hmm, i guess they should keep one clone ready if this is the case, and i guess it may not. anyway, i don't know anything about NOC business so i don't have right to comment. :(
hey i didn't emailed them because i know they must be already tens and must be working hard to fix it.
:bawling:
cbaker17 12-21-2001, 02:51 PM It looks like its still down how long has this been for!!
michaeln 12-21-2001, 02:53 PM Somewhere around 25 hours
mahinder 12-21-2001, 02:57 PM since last 40 minutes my pages is continusely ringing. :(
i guess they are replacing the router :rolleyes:
Daniel 12-21-2001, 03:06 PM I just talked to them they are still waiting on the memory module they need. They don't need a router they just need memory for one!!!! If the part did not ship we are SOL. I have been up 4 hours of the last 26 hours!!! :angry:
mahinder 12-21-2001, 03:11 PM what do you think we should do in this situation, i am thinking of propagating my main web site to my other servers. :(
if this is gone take long time i should better start changing name server of my main domain.
:o
michaeln 12-21-2001, 03:30 PM My site seems to be up again.
Maybe they got their RAM in.....
Fingers crossed.....
Michael
TotalHst 12-21-2001, 03:32 PM They are up! Most likely they got ram in.
mahinder 12-21-2001, 03:47 PM please please confirm it with them . . . somebody call them please. . .
hey sean through some words over here when you guys are fixed. :)
Samuel Mann 12-21-2001, 03:50 PM Yea, and also throw some words in when you might be getting that back up router.
Daniel 12-21-2001, 03:50 PM IT'S UP!!!!!!!
But they still don't have the memory. They said it's holding for now. please please please hold!!!:)
mahinder 12-21-2001, 04:03 PM hey Daniel, ther is nothing to spen around, if they haven't fixed the problem they will be down again, they are down every 1 hours since 18-19 hours.
:(
Daniel 12-21-2001, 04:15 PM True is may not last but this it the first time I have been up over 10 min since 7:00pm last night. It gives me hope.
alchiba 12-21-2001, 04:20 PM Originally posted by mahinder
how much this damn router cost any ideas ?
$9175, give or take. Not something you pick up at the local computer shop on the way in to the office. I'm sure the RAM is proprietary too. ;)
http://www.recurrent.com/specials/home.php3?prod_name=cisco
Kylecool 12-21-2001, 04:32 PM EXPENSIVE. However, the clients they might loose as well as complaints, etc. They should've just had the spare parts on hand. Not smart when you are such a large company....
As far as the second backup one, the end of the year is approaching. Let's just see if this actually comes through, or it'll be put off to the last min. like Cpanel.
-Kyle
mahinder 12-21-2001, 04:44 PM Originally posted by Kylecool
EXPENSIVE. However, the clients they might loose as well as complaints, etc. They should've just had the spare parts on hand. Not smart when you are such a large company....
As far as the second backup one, the end of the year is approaching. Let's just see if this actually comes through, or it'll be put off to the last min. like Cpanel.
-Kyle
not pretty expensive considering the no of server they claim they are hosting, sean once said they have more then 600 servers, and that is pretty large number, yeah, there noc is new so they may not be having proper planning, if you consider the cost of router and divid it with no of server then it will be very low. ;)
just to tell you there support is very good, the company have great future and this kind of problems should not come in there way.
anyway, its there biz. they know more about it. just my throught.
Kylecool 12-21-2001, 05:11 PM I meant expensive like for me, because I'm only 15. If you have 600+ Servers, they should be able to cover it no problem....
-Kyle
cbaker17 12-21-2001, 06:13 PM i wonder if this downtime was the longest of wht history due to bad hardware??? anyone know?
Arsalan 12-21-2001, 06:19 PM any one know if its back for good or they are just teasing us ? ;)
TotalHst 12-21-2001, 08:25 PM Router is holding, the shipment did not come in today, but they are expecting it tommorow. They put some ram from another cisco router they had into the router and it seems to be working with it. I just want to know why they didn't put this spare ram from another router in there in the first place..
mahinder 12-22-2001, 01:09 AM they may not be knowing this before, nobody wants trouble?.
yesterday i spoke to them and they said the FedEx didn't deliever the ram chip. too bad!! they must have sent somebody over the car or something to get the ram personally and put it in the router, yes i guess this is worst downtime i have ever faced in my life. :(
anyway now they seems to up because since many hours my pager is quit. :)
alchiba 12-22-2001, 01:33 AM Originally posted by cbaker17
i wonder if this downtime was the longest of wht history due to bad hardware??? anyone know?
How about last March? Another provider had quite a disastrous day or two then. Quite a character-building experience. Or, more precisely, it revealed character. ;)
mahinder 12-22-2001, 01:48 AM Originally posted by alchiba
How about last March? Another provider had quite a disastrous day or two then. Quite a character-building experience. Or, more precisely, it revealed character. ;)
can you name the provider, who was it and what was the story ?. :(
AlaskanWolf 12-22-2001, 02:56 AM 3 letters...starts with a V :)
Kylecool 12-22-2001, 03:19 AM VDI. I remember that.... I was with HostRocket at the time.. They were thinking about moving to Dialoneinternet, but stayed with VDI...
-Kyle
Arsalan 12-22-2001, 04:29 AM their network is responding well for the past 11 hours... i hope it holds.
Samuel Mann 12-22-2001, 04:40 AM Originally posted by Kylecool
VDI. I remember that.... I was with HostRocket at the time.. They were thinking about moving to Dialoneinternet, but stayed with VDI...
-Kyle
ditto
I remember your posts Kyle
padrian 12-22-2001, 02:21 PM Router is holding, the shipment did not come in today, but they are expecting it tommorow. They put some ram from another cisco router they had into the router and it seems to be working with it. I just want to know why they didn't put this spare ram from another router in there in the first place..
If this was the case and the had the other ram the whole time thats a bummer. I'm glad everything I working again now.
VDI. I remember that.... I was with HostRocket at the time.. They were thinking about moving to Dialoneinternet, but stayed with VDI...
Hi Kyle
my 2 cents , patrick
Shiva 12-23-2001, 09:23 AM BurstNet is up, but my sites (hosted by BurstNet for HOSTurbo) are down again...
I wrote them but got no answer till now.
Domenico 12-23-2001, 09:55 AM pom pom pom
Domenico 12-23-2001, 09:56 AM We are connected to them from Europe through AT&T ITS. We were on and of for two days but everything works fine again.
I hope Burst learned from this and makes sure the same thing will and can't happen again.
Too bad we can't discuss things like this on a forum at burst.net
Hey BURST! Buy vBulletin and mail ALL you customers the forum URL. I mean where can we discuss things like this or the following ???
All servers have remote cold (re)boot equipment installed for your password protected usage.
I'm still waiting :rolleyes:
oh, I'm still a very satisfied customer :D
Domenico
Daniel 12-23-2001, 11:37 AM I've been happy with the network for the last 8 months so as long as this does not happen again I'm happy. The problem did not bother me the lack of a solution did. But you live and you learn. We all make mistakes just don't let the same mistake burn you twice.:D
And Domenico they have a forum. I don't think anyone knows about it but they do have one.:)
http://burst.net/cgi-bin/UltraBoard/UltraBoard.cgi
Shiva 12-23-2001, 01:34 PM My domains are still not working. And nobody answers the phone. Is everybody lucky without me? ;)
Daniel 12-23-2001, 01:51 PM Target Name: ez-web-hosting.com
IP: 66.197.128.4
Date/Time: 12/23/2001 12:35:11 PM
1 27 ms 27 ms 13 ms [64.53.78.65]
2 28 ms 28 ms 28 ms [204.116.177.13]
3 27 ms 27 ms 27 ms atm1-0-9050103.r01.ncchrl.infoave.net [204.116.177.2]
4 27 ms 14 ms 28 ms atm8-0-7.gwy.ncchrl.infoave.net [165.166.126.148]
5 28 ms 41 ms 27 ms [12.124.58.61]
6 41 ms 41 ms 41 ms gbr6-p80.attga.ip.att.net [12.123.21.78]
7 28 ms 41 ms 27 ms gbr4-p80.attga.ip.att.net [12.122.5.50]
8 55 ms 41 ms 41 ms gbr4-p30.wswdc.ip.att.net [12.122.2.226]
9 41 ms 41 ms 41 ms gbr3-p60.wswdc.ip.att.net [12.122.1.129]
10 55 ms 55 ms 55 ms gbr3-p20.n54ny.ip.att.net [12.122.3.53]
11 55 ms 55 ms 55 ms gbr6-p60.n54ny.ip.att.net [12.122.5.113]
12 55 ms 55 ms 55 ms ar2-p3110.cdknj.ip.att.net [12.123.192.17]
13 68 ms 68 ms 96 ms [12.119.155.82]
14 69 ms 69 ms 55 ms 661971284.hostnoc.net [66.197.128.4]
We have been up 100% since around 2:00 Friday.:D
What is your domain name or IP?
Shiva 12-23-2001, 02:52 PM I can ping 66.96.210.235, but the DNS entry doesn't work anymore.
My websites are
www.tantralinks.com
www.swisslinks.net
www.clickwork.net
For example you can reach HOSTurbo at 66.96.210.235, but not at www.hosturbo.com.
Daniel 12-23-2001, 03:18 PM Is your name server running?
Logged in a root type:
/etc/rc.d/init.d/named start
Or you could use WHM to restart it.
Domenico 12-23-2001, 03:51 PM Originally posted by Daniel
I've been happy with the network for the last 8 months so as long as this does not happen again I'm happy. The problem did not bother me the lack of a solution did. But you live and you learn. We all make mistakes just don't let the same mistake burn you twice.:D
And Domenico they have a forum. I don't think anyone knows about it but they do have one.:)
http://burst.net/cgi-bin/UltraBoard/UltraBoard.cgi
Hi Daniel,
I knew about their forum from the first day they started it and I tried posting but it seems I'm one of the very few that has used it.
Why not use better forum software and mail EVERY user that it is online? i posted there sometimes but NOBODY! responded to it now what kind of forum is that?
I don't need forum software to start a discussion with myself.
The forum has been deserted for months now and too me that is the same as having NO forum at all.
Come on BURST and start using the forum and tell all users about it! And when does the remote reboot hardware get installed ???
Domenico
Shiva 12-23-2001, 04:11 PM Originally posted by Daniel
Is your name server running?
Logged in a root type:
/etc/rc.d/init.d/named start
Or you could use WHM to restart it.
Thank you Daniel!
The problem is, that I am only a user and have no root access. HOSTurbo is not interested in hosting... and BurstNET does not answer (and perhaps read) support tickets.
I have no access to my websites now for hours...
allan 12-24-2001, 11:34 AM Originally posted by alchiba
$9175, give or take. Not something you pick up at the local computer shop on the way in to the office. I'm sure the RAM is proprietary too. ;)
That's just for the base router, by the time you add in OS, extra RAM, DS3/OC3 cards, and everyting else you need, you are looking at a price closer to $20-$25k.
Of course what Cisco will tell you is that you do not necessarily need a second router, instead you can just buy spares of the parts you may need to replace.
acer2k 12-25-2001, 04:29 AM I don't use Burst Net..But If I did..and they were down THAT long and it costs me hundreds of dollars..looks like i would have been calling my lawyers, not burst ;)
Arsalan 12-25-2001, 07:53 AM every one has good and bad days... its good to know that atleast the burstNET guys eventually fixed it and came clean instead of hiding behind the bush...
scott2 12-25-2001, 04:59 PM That's just for the base router, by the time you add in OS, extra RAM, DS3/OC3 cards, and everyting else you need, you are looking at a price closer to $20-$25k.
Beginner's question: OS???
Originally posted by scott2
Beginner's question: OS???
Operating system :D
mdrussell 12-25-2001, 06:26 PM Originally posted by scott2
Beginner's question: OS???
OS = Operating System
scott2 12-25-2001, 06:28 PM And you buy the OS separately for a high end router? Are there different operating systems to choose from for a router? I just assumed it would be proprietary to the router and included with the base price. Interesting.
allan 12-25-2001, 07:43 PM Nope, at least with Cisco routers, the OS is separate. This is usually disguised when you buy it from a reseller, because they include the cost of the OS within the final cost, but if you buy directly from Cisco, the OS is itemized independently. You can also buy the OS separately (if you buy a router used, it is not supposed to come with an OS -- just like you cannot transfer Microsoft Licenses).
alchiba 12-25-2001, 11:27 PM Originally posted by acer2k
I don't use Burst Net..But If I did..and they were down THAT long and it costs me hundreds of dollars..looks like i would have been calling my lawyers, not burst ;)
What's with the lawyer thing all the time? Do you actually think a lawyer would take this case? Do you think a lawyer would even dignify this incident with the term "case"? It would cost you thousands to recover hundreds even if somehow you had prayer. Get a grip.
HRBrendan 12-26-2001, 06:01 AM We run virtually an identical router config to them and with the 4 hour part replacement plan it is easily a $25-30k router, and thats without any additional cards. Im curious as to what kind of firewall protection they have in place and if it was effective in stopping the DOS attack at all? If not why? I basically want to know this for my own good, call me greedy... trying to learn from others mistakes. :)
-Brendan
Kylecool 12-26-2001, 09:05 PM Operating System :)
-Kyle
Kylecool 12-26-2001, 09:06 PM Originally posted by HRBrendan
We run virtually an identical router config to them and with the 4 hour part replacement plan it is easily a $25-30k router, and thats without any additional cards. Im curious as to what kind of firewall protection they have in place and if it was effective in stopping the DOS attack at all? If not why? I basically want to know this for my own good, call me greedy... trying to learn from others mistakes. :)
-Brendan
hehehheheh
-Kyle :)
jayglate 12-26-2001, 09:11 PM We have a fully configured standby router ready to be implemented in such a case of router hardware failure. We are working to implement VRRP so both routers share the load.
And I think we run the same router also:
7206VXR with an NPE400 Package
RackMy.com 12-26-2001, 10:14 PM We have a fully configured standby router ready to be implemented in such a case of router hardware failure.This is one of the best ways for network redundancy :) VRRP (or HSRP which is Cisco's version) We do this with all our equipment except end node switches.
BurstNET 12-26-2001, 11:10 PM Cisco HSRP is the way we are going...been planned for awhile...unfortunately not in place before this incident occured :-(
Sean R.
BurstNET
HRBrendan 12-27-2001, 12:00 AM Thats the path we're on too... unfortunatly those routers are not cheap :)
-Brendan
allan 12-27-2001, 12:13 AM Originally posted by HRBrendan
Thats the path we're on too... unfortunatly those routers are not cheap :)
Brendan -- if you are not already running a second router, have you considered pulling in a different vendor? You can get a Riverstone or ImageStream router, with the same throughput as the 7204, for considerably less. As of IOS 12.18T Cisco supports VRRP, as well as their proprietary HSRP. So, you can put in a second router, run VRRP between the two devices and have the same failover capabilities without having to spend the full $25-$30k :D.
RackMy.com 12-27-2001, 12:48 AM Cisco HSRP is the way we are going...been planned for awhile...unfortunately not in place before this incident occured :-( I know, redundancy comes at a high price!
Kylecool 12-27-2001, 01:08 AM Originally posted by RackMy.com
I know, redundancy comes at a high price!
TRUE! True Indeed, but it is an important tool in the hosting industry. ;)
-Kyle
HRBrendan 12-27-2001, 01:28 AM We're gonna stick with cisco, we run all cisco products now and will probably stay that way.
-Brendan
Kylecool 12-27-2001, 01:34 AM Originally posted by HRBrendan
We're gonna stick with cisco, we run all cisco products now and will probably stay that way.
-Brendan
That's good. They are a good brand, and should hopefully keep you up and stable. Smart Choice Brendan Brader! ;)
-Kyle
|