
|
View Full Version : DedNow/NAC Slowdown?
Greetings,
Anyone noticed the major difficulties in reaching the servers at NAC/Dednow? Just about all servers there are so slow that it takes about 7 minutes just to get SSH prompt on any of the servers. Anyone else experiencing the same problem?
ImLagging 06-11-2003, 10:11 AM for now, it just seems as if everything is offline to me.
TotalChoice 06-11-2003, 10:13 AM Its that quality NAC bandwidth that everyone talks about.
alapo 06-11-2003, 10:14 AM Tracing route to nac.net [207.99.0.69]
over a maximum of 30 hops:
1 6 ms 7 ms 5 ms 10.17.144.1
2 8 ms 7 ms 7 ms vl7.aggr1b.lnh.md.rcn.net [207.172.11.132]
3 8 ms 9 ms 9 ms ge3-0.core2.lnh.md.rcn.net [207.172.15.2]
4 13 ms 10 ms 12 ms pos6-0.core1.phdl.pa.rcn.net [207.172.19.8]
5 13 ms 15 ms 17 ms pos6-0.core2.nyw.ny.rcn.net [207.172.19.7]
6 12 ms 14 ms 13 ms fe4-0.border2.nyw.ny.rcn.net [207.172.15.73]
7 13 ms 14 ms 17 ms t3-3-3-3.gbr1.nyc.nac.net [207.99.8.237]
8 27 ms 18 ms 16 ms 1194.at-0-0-0.gbr1.oct.nac.net [209.123.11.10]
9 15 ms 30 ms 15 ms 0095.gi-1-1.msfc1.oct.nac.net [64.21.102.2]
10 16 ms 16 ms 15 ms mail.nac.net [207.99.0.69]
Trace complete.
Thats better than speedy for me...? And I thought I pinged to ezzi quick (20ms to ezzi).
mnett 06-11-2003, 10:15 AM It's really sloooow
alapo 06-11-2003, 10:15 AM How about some traceroutes?
patchwork 06-11-2003, 10:15 AM very poor for me
8 15 ms 16 ms 15 ms linx.ge-0-0-0.gbr1.ltn.nac.net [195.66.224.94]
9 219 ms 234 ms 282 ms 1209.at-3-1-0.gbr1.nyc.nac.net [209.123.11.241]
10 * 687 ms * 1187.at-0-1-0.gbr1.oct.nac.net [209.123.11.150]
11 688 ms 687 ms 703 ms 209.123.182.243
alapo 06-11-2003, 10:17 AM They just took down their ircd. They seem to do it when a large ddos happens. Maybe they are under attack?
ImLagging 06-11-2003, 10:32 AM the main IP for my server seems to be just fine. however, the additional IP's don't seem to be accessable at all. quite weird.
FHDave 06-11-2003, 10:33 AM Originally posted by TotalChoice
Its that quality NAC bandwidth that everyone talks about.
It's the quality data center and bandwidth that everybody has assumed about :)
Good luck.
kickmybutt 06-11-2003, 10:39 AM Here is mine:
REQUEST TIMED OUT
REQUEST TIMED OUT
REQUEST TIMED OUT
REQUEST TIMED OUT
REQUEST TIMED OUT
REQUEST TIMED OUT
REQUEST TIMED OUT
REQUEST TIMED OUT
I love NAC. Problem free 3 hours a day!
patchwork 06-11-2003, 10:43 AM Originally posted by FHDave
It's the quality data center and bandwidth that everybody has assumed about :)
Good luck.
Pre DedicatedNow the network was good, since DN the network quality has gone rapidly downhill, I guess that's what happens when you flood a network with cheapo servers.
.
IGobyTerry 06-11-2003, 10:48 AM Everything seems fine on my end. Did a traceroute from my server at a different DC, and from my house, and it was all fine.
UH-Matt 06-11-2003, 10:49 AM Originally posted by kickmybutt
Here is mine:
REQUEST TIMED OUT
REQUEST TIMED OUT
REQUEST TIMED OUT
REQUEST TIMED OUT
REQUEST TIMED OUT
REQUEST TIMED OUT
REQUEST TIMED OUT
REQUEST TIMED OUT
I love NAC. Problem free 3 hours a day!
:D
ImLagging 06-11-2003, 10:51 AM it seems to be working for me now.
patchwork 06-11-2003, 10:56 AM its up and down, fast and slow
.
FHDave 06-11-2003, 10:57 AM Originally posted by patchwork
Pre DedicatedNow the network was good, since DN the network quality has gone rapidly downhill, I guess that's what happens when you flood a network with cheapo servers.
.
I've never considered NAC to be the best datacenter nor I have considered their bandwidth peering to be the most excellent, pre and post DN. NAC just receive too much praise here, mostly from people that do no know what they are talking about ...
Anyway, does not seem to have any problem from our servers (http://fluidhosting.com/traceroute.php?host=nac.net) at Internap (Boston).
kickmybutt 06-11-2003, 10:57 AM Yea, me too. We are in the "Good 3 hour" time frame now. So, send out all emils and make your apporiporate posts on your forums to let you customers know that they need to hurry and upload new content (if they have any) to their webistes.
You could be a good host and make the index.htm page for them to use....
Here is some code for ya'
<bold> Sorry for the downtime... we use NAC </b>
<a href="http://www.nac.net">NAC</a>
:D :D :D
PhMatt 06-11-2003, 10:59 AM Pre DedicatedNow the network was good, since DN the network quality has gone rapidly downhill, I guess that's what happens when you flood a network with cheapo servers.
The network latency was due to a DOS attack, having NOTHING to do with DedNOW / DCJN. We are not the culprit in this matter, nor are we to blame for any such issues IMHO. We do not own / control network, facilities, UPS, Generator, etc. Perhaps DedNOW has brought more exposure to the network publicly than prior to our launch, given our shear size, and type of service provider that we are, a lot of clients do come from WHT, and as such, make it a more public reaction than previously seen. Since we are a colo company, but staffed onsite, it gives us more presence, agreed, but blaming DedNOW for issues beyond our control is not fair at all.
When I made that post, it wasn't to bash on you guys. But some people can be pretty damn merciless when commenting on some of the well known providers, from left or right.
Oh well...need to get some kind of karma rating for the users to post their comments.
alapo 06-11-2003, 11:08 AM Originally posted by DedicatedNOW
The network latency was due to a DOS attack, having NOTHING to do with DedNOW / DCJN.
Exactly what I thought.
They just took down their ircd. They seem to do it when a large ddos happens. Maybe they are under attack?
patchwork 06-11-2003, 11:29 AM Originally posted by DedicatedNOW
The network latency was due to a DOS attack, having NOTHING to do with DedNOW / DCJN. We are not the culprit in this matter, nor are we to blame for any such issues IMHO. We do not own / control network, facilities, UPS, Generator, etc. Perhaps DedNOW has brought more exposure to the network publicly than prior to our launch, given our shear size, and type of service provider that we are, a lot of clients do come from WHT, and as such, make it a more public reaction than previously seen. Since we are a colo company, but staffed onsite, it gives us more presence, agreed, but blaming DedNOW for issues beyond our control is not fair at all.
Maybe this problem today is not DN's fault, but since DN there has been a lot more network problems than pre DN.
Pre DN I had very few network problems, post DN there has been a constant supply of network problems, ie downtime, packet loss, router/swicher, UPS problems etc... the list seems endless sometimes.
Maybe these problems are not all DN's fault and yes some of them are proberbly temporary growing paids.
ImLagging 06-11-2003, 11:40 AM Originally posted by patchwork
Maybe this problem today is not DN's fault, but since DN there has been a lot more network problems than pre DN.
Pre DN I had very few network problems, post DN there has been a constant supply of network problems, ie downtime, packet loss, router/swicher, UPS problems etc... the list seems endless sometimes.
Maybe these problems are not all DN's fault and yes some of them are proberbly temporary growing paids.
or there could just be all new staff and/or management at NAC. just like when interland buys out a host, the service goes down the tubes. not saying this is it, just tossing around ideas. :)
IGobyTerry 06-11-2003, 11:47 AM Originally posted by FHDave
I've never considered NAC to be the best datacenter nor I have considered their bandwidth peering to be the most excellent, pre and post DN. NAC just receive too much praise here, mostly from people that do no know what they are talking about ...
Anyway, does not seem to have any problem from our servers (http://fluidhosting.com/traceroute.php?host=nac.net) at Internap (Boston).
I'm going to have to agree with you. I think NAC is a good, reliable datacenter, but it's not the best as some seemed to imply. Right now I'm in love in with ThePlanet, the network is so fast (This in the #2 datacenter).
But, hey, to each his own. Some like NAC, some like RackShack... etc.
Aussie Bob 06-11-2003, 11:51 AM Originally posted by inogenius
I'm going to have to agree with you. I think NAC is a good, reliable datacenter, but it's not the best as some seemed to imply . . .
Where exactly are these posts where folks "imply" that NAC is the "best"?
FHDave, thanks for your unbiased personal opinion on NAC. I'm sure we can find a place to file that. :D
IGobyTerry 06-11-2003, 11:58 AM Originally posted by Aussie Bob
Where exactly are these posts where folks "imply" that NAC is the "best"?
FHDave, thanks for your unbiased personal opinion on NAC. I'm sure we can find a place to file that. :D
I'll get back to you on that. I tried doing a quick seach, however it seems WHT's database was pruned. It only goes back to 5/11/03 when I do a search.
Incognito 06-11-2003, 12:11 PM The network latency was due to a DOS attack, having NOTHING to do with DedNOW / DCJN. Since we are a colo company, but staffed onsite, it gives us more presence, agreed, but blaming DedNOW for issues beyond our control is not fair at all. As to those points, I have to somewhat disagree with you, Matthew, as much as I respect you and your service emphasis. You are ultimately responsible for the service to your customer, whether that is impaired by something you do or something else happening within the data center you use.
Now, are you going to have issues such as the one today? Yes, with any provider. Ultimately, DN and PWeb and NAC and all others using either of the three as providers will be judged based on the overall quality of their service. I personally believe the growth has created more issues. However, those issues are then greatly magnified by the percentage of DN users on WHT. It, also, does appear to me that there are some NAC issues needing to be addressed-more than there were 6 months to a year ago. I am sure you, Jay, and others will be attempting to get these resolved, because ultimately they reflect on and impact your businesses.
So, while I take the calm approach asking people not to blow one or a few incidents out of proportion, I do believe there needs to be concern, particularly when a few very well respected hosts have found it appropriate recently to leave NAC.
Frontpage1 06-11-2003, 12:41 PM Here is some ping data comparison for:
Pinging mail.nac.net [207.99.0.69] with 32 data bytes
Start Time: Wed, 11 Jun 2003 12:37:10
ANALYSIS:Target reached by one or more packets.
10 packets transmitted, 10 packets received, 0% packet loss
round-trip (ms) min/avg/max = 25/26/30
Pinging mail.theplanet.com [216.234.234.32] with 32 data bytes
Start Time: Wed, 11 Jun 2003 12:39:35
ANALYSIS:Target reached by one or more packets.
10 packets transmitted, 10 packets received, 0% packet loss
round-trip (ms) min/avg/max = 54/55/57
I would be interested to see what others get. We are pinging from the East Coast USA to these datacenters.
PhMatt 06-11-2003, 12:53 PM Incognito,
Agreeing with your points to a degree, and def not intending to start any "flaming wars here" I only hope to openly talk about issues, and hope that my opinions on the matter are at least understood to some degree by those following this thread. What I was intending originally to clarify in my post, is that we appeared to be the main focal point of the comments being made. I agree 100% DedNOW is solely responsible to its customers, and where we choose to do business, it is our responsibility to our clients that we are offering the very best service that we can, which they deserve.
My points were that we were being blamed for being the cause of such recent issues, rather than being part of a group of those suffering. We are doing the very best that we can, within our means of course for our clients. Without our customers, we are nothing, and that we all know / understand. What we control vs what we do not control was my main concern. We don't run the network, nor have the ability to oversee operations within this facility when it comes to how the network is run, how and what type of backup procedures are in place, etc. The decision is solely up to those who own / run the facility, in this case NAC. It's no different than anybody posting here, that should all of a sudden they grow with enormous leaps / bounds that they would be pinpointed as the root cause of the DC's issues. Sure we'll agree that with expansion / rapid growth there's always going to be times of temp growing pains. What we have done, is when they arise, we take the financial loss of stopping our sales / growth for the benefit / betterment of our clientele. We have and will continue to act upon the best interest of our clients. We too are a customer, and we too have needs which must be met. I won't get into the financial #'s as it's not important, but I can say having to 3 times now cut sales in record breaking months, and halting our own growth has def had an impact on us that many would never believe. We take our losses, weigh our options and choose what the next best step is in our development.
Again, not disputing that we don't have obligations to our customers, because that we'll never deny. All I can say is that we're doing our best to do whatever is necessary to ensure the quality and reliability of our service is maintained & constantly improved upon.
Thank you,
Incognito 06-11-2003, 01:18 PM Matthew:
I never questioned your committment to service as you have firmly stood for that in all three companies where I have observed you as a manager. I am sure you will solve the problems and find it unfortunate that your provider has been coming up short recently. Hopefully, they will find a way to quickly regain their previous standards of performance and to support you better.
AussieHosts 06-11-2003, 01:24 PM Originally posted by Frontpage1
I would be interested to see what others get. We are pinging from the East Coast USA to these datacenters.
Pings/traces from Australia have always been good to theplanet.com and poor to NAC. A difference of around 40 to 70ms, which is more than enough when you've already travelled 150 to get there. :)
Gary
Originally posted by Editor
Pings/traces from Australia have always been good to theplanet.com and poor to NAC. A difference of around 40 to 70ms, which is more than enough when you've already travelled 150 to get there. :)
Gary
well, lets think about this one real quick.. theplanet.com is over in Texas, while NAC is over in NJ. of course you are probably to see lower latency to theplanet.com; Australia is geographically closer to Texas than NJ.
Aussie Bob 06-11-2003, 01:37 PM Doubt you would ever really notice 40 to 70ms. :)
AussieHosts 06-11-2003, 01:43 PM You can generally notice it very clearly Bob. 40-70 is being generous though...it usually exceeds that.
G
Aussie Bob 06-11-2003, 01:49 PM We've got boxes in both DCs and TP is faster from Oz. It's not that noticeable for me. I get about 280 to NAC and around 240 to TP. I'm on Telstra ADSL. Given the vast majority of our client base is USA based, there's not much difference between the 2 there. :)
Doing our little bit for the Aussie trade balance there. ;)
patchwork 06-11-2003, 02:10 PM Originally posted by DedicatedNOW
My points were that we were being blamed for being the cause of such recent issues, rather than being part of a group of those suffering.
This time yes, but most of the other problems have been directly related to the DN expansion.
Originally posted by DedicatedNOW
What we have done, is when they arise, we take the financial loss of stopping our sales / growth for the benefit / betterment of our clientele.
Maybe you should issue refunds or some other kind of compensation to the customers when these problems arise, most of the problems are caused by DN's expansion, but it's not just DN that loose money and sales from the network instability and or downtime.
Maybe you would treat the DN/Pweb section of the NAC network differently if you had to pay customer losses when you are the direct cause of the losses.
In the past 6 months the network has been very unstable at times, and down at other times, sometimes for long periods. Over this period I have never received or been offered compensation of any type, you are gaining from the expansion and your existing customers are paying the price.
.
patchwork 06-11-2003, 02:26 PM Originally posted by Aussie Bob
We've got boxes in both DCs and TP is faster from Oz. It's not that noticeable for me. I get about 280 to NAC and around 240 to TP. I'm on Telstra ADSL. Given the vast majority of our client base is USA based, there's not much difference between the 2 there. :)
Doing our little bit for the Aussie trade balance there. ;)
My results from the UK
C:\>ping theplanet.com
Pinging theplanet.com [216.234.234.68] with 32 bytes of data:
Ping statistics for 216.234.234.68:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 125ms, Maximum = 140ms, Average = 136ms
C:\>ping nac.net
Pinging nac.net [207.99.0.69] with 32 bytes of data:
Ping statistics for 207.99.0.69:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 78ms, Maximum = 94ms, Average = 86ms
When looking at a datacenter I do not just go off the ping times, general stability/uptime are very important factors.
.
missyomar 06-11-2003, 02:33 PM C:\Documents and Settings\Max Tappenden>tracert nac.net
Tracing route to nac.net [207.99.0.69]
over a maximum of 30 hops:
1 * * * Request timed out.
2 16 ms 16 ms 15 ms 62.25.202.144
3 15 ms 15 ms 15 ms lettuce.as5388.net [62.25.202.130]
4 16 ms 17 ms 18 ms 195.50.116.185
5 19 ms 18 ms 18 ms ae0-12.mp2.London1.Level3.net [212.187.131.34]
6 84 ms 85 ms 84 ms unknown.Level3.net [212.187.128.153]
7 83 ms 82 ms 83 ms gigabitethernet5-2.core1.NewYork1.Level3.net [64
.159.17.165]
8 83 ms 83 ms 84 ms williams-level3-oc3.NewYork1.Level3.net [209.244
.160.166]
9 84 ms 85 ms 85 ms nycmny2wcx3-pos9-0.wcg.net [64.200.87.226]
10 83 ms 83 ms 83 ms nycmnyhlce1-oc48.wcg.net [64.200.87.110]
11 95 ms 84 ms 83 ms nycmny2lce1-netaccess-atm.wcg.net [64.200.86.150
]
12 85 ms 88 ms 86 ms 1187.at-0-1-0.gbr1.oct.nac.net [209.123.11.150]
13 87 ms 86 ms 85 ms 0095.gi-1-1.msfc1.oct.nac.net [64.21.102.2]
14 87 ms 86 ms 86 ms mail.nac.net [207.99.0.69]
Trace complete.
C:\Documents and Settings\Max Tappenden>
The first one that errors is just because of the firewall on my router.
mdrussell 06-11-2003, 02:45 PM Perhaps people should sign up to NAC's network ops list, or ask DedicatedNow / Pwebtech to provide one.
I wonder if 90% of the people that posted here suffering problems actually bothered to call NAC, or Pwebtech if they go through them to find out if there was a problem?
mainarea 06-11-2003, 03:10 PM If you're tracing from the east coast or Europe, chances are, your ping times to NAC (located on the east coast) will be lower than your pings to ThePlanet (located in the middle of the US). Ping times aren't everything.
- Matt
PhMatt 06-11-2003, 03:14 PM All right, let's go through the points then one by one:
Maybe you should issue refunds or some other kind of compensation to the customers when these problems arise, most of the problems are caused by DN's expansion, but it's not just DN that loose money and sales from the network instability and or downtime.
To date, we've issued more thank 50k worth of credits and service adds to clients for the 3 major periods of problems within the DedNOW/DCJN. On top of this, we've also halted sales, during those periods, and using low numbers, have lost in the neighborhood of $500,000 - $750,000 recurring revenue annually.
Maybe you would treat the DN/Pweb section of the NAC network differently if you had to pay customer losses when you are the direct cause of the losses.
Point out to me when DedNOW/ Pegasus has been the direct cause of an issue that we have not responsibly taken action for? We don't monitor, nor have access to planning methodology for growth, power, ac, within this facility. Upon being informed of issues, we immediately ceased operations, cutting record sales, to allow for accommodation of our growth to be met. We have SLA's / TOS's that we offer to all clients, and have always fulfilled violation submissions.
In the past 6 months the network has been very unstable at times, and down at other times, sometimes for long periods. Over this period I have never received or been offered compensation of any type, you are gaining from the expansion and your existing customers are paying the price.
Have you ever fulfilled or submitted an SLA violation request? If a DedNOW client, you would have either received Free RRP's, Port monitoring, if credit request submitted, either a credit, or if not submitted, and during certain timeframes, a doubling of additional days to your billing cycle, or moving of billing cycle date for days during outages.
Again, my point is and always was this. We do not control generators, UPS's, AC's, growth and expansion of this facility, network equipment outside of a single router and our switches. I have made my piece with this thread, and offered our companies resolution in these matters. I'm not side-stepping responsibility, and agree that our company is responsible to our clients, and as such, we will take every step necessary to ensure satisfaction of our clientele as best and as quickly as we possibly can.
Thank you,
kicker 06-11-2003, 04:01 PM DedicatedNow has definately gone down hill since we signed up. Packet loss is about 1% higher than the other colocation facilites that I am in and it is having a big impact on download speeds. Download speeds are about 30% slower on average than other colos.
When ever the packet loss numbers get above 5% I try to do about 20-30 traceroutes back to the machines downloading from my server to find the packetloss. Other than an overload NAC OC12 most of the packetloss is hard to pinpoint. Most of the packet loss seems to be just past peering points or inside WCG.
Still, it is hard to justify spending more than UnitedColo and getting worse performance.
kicker 06-11-2003, 04:04 PM Just as a followup here is an example. Current packet loss across all connections of about 6%. Back traces show general problems and packet loss but nothing that is easy to point to in a trouble ticket.
1 66.246.98.1 (66.246.98.1) 0.528 ms 0.565 ms 0.440 ms
2 0860.ge-0-0-0.gbr1.oct.nac.net (209.123.182.242) 1.982 ms 1.866 ms 2.736 ms
3 1187.at-0-1-0.gbr1.nyc.nac.net (209.123.11.149) 4.012 ms 3.367 ms 3.116 ms
4 1188.at-0-1-0.gbr1.ltn.nac.net (209.123.11.242) 69.412 ms 70.217 ms 71.778 ms
5 linx.lon.seabone.net (195.66.226.153) 69.758 ms 70.079 ms 71.402 ms
6 mia1-pal6-racc1.seabone.net (195.22.196.98) 217.921 ms 217.584 ms 215.573 ms
7 200.43.185.249 (200.43.185.249) 347.511 ms 346.652 ms 346.343 ms
Tracing the path to 212.96.100.139 on TCP port 80, 10 hops max
1 66.246.98.1 (66.246.98.1) 0.803 ms 0.575 ms 0.532 ms
2 0860.ge-0-0-0.gbr1.oct.nac.net (209.123.182.242) 3.048 ms 3.163 ms 2.802 ms
3 * 0.so-0-1-0.gbr1.nwr.nac.net (209.123.11.234) 3.191 ms 2.232 ms
1 66.246.98.1 (66.246.98.1) 0.900 ms 0.536 ms 0.404 ms
2 0860.ge-0-0-0.gbr1.oct.nac.net (209.123.182.242) 1.492 ms 1.177 ms 0.946 ms
3 0.so-0-1-0.gbr1.nwr.nac.net (209.123.11.234) 13.514 ms 2.876 ms 2.526 ms
4 1189.at-0-1-0.gbr1.ash.nac.net (209.123.11.194) 9.754 ms 10.784 ms 10.361 ms
5 * res1-avici1-g1-5-3-503.gnaps.net (209.123.24.174) 13.248 ms 8.933 ms
1 66.246.98.1 (66.246.98.1) 0.510 ms 1.035 ms 0.517 ms
2 0860.ge-0-0-0.gbr1.oct.nac.net (209.123.182.242) 3.242 ms 3.251 ms 4.258 ms
3 0.so-0-1-0.gbr1.nwr.nac.net (209.123.11.234) 4.346 ms 4.217 ms 3.770 ms
4 1189.at-0-1-0.gbr1.ash.nac.net (209.123.11.194) 9.660 ms 8.986 ms 9.626 ms
5 sp0-4-ASBNVAAS.broadwing.com (206.223.115.72) 9.766 ms 10.791 ms 10.509 ms
6 216.140.8.77 (216.140.8.77) 12.187 ms 12.239 ms 12.218 ms
7 so-0-1-0.c1.wash.broadwing.net (216.140.8.14) 10.524 ms 9.926 ms 10.192 ms
8 p7-2-0.c1.atln.broadwing.net (216.140.8.22) 22.741 ms 23.177 ms 23.246 ms
9 p3-0-0.a1.atln.broadwing.net (216.140.12.38) 23.249 ms 22.319 ms *
dbbrock1 06-11-2003, 04:17 PM It's going VERY slow for me as well. Takes me about 30 seconds to download our main page which is less than 100KB
Originally posted by kicker
DedicatedNow has definately gone down hill since we signed up. Packet loss is about 1% higher than the other colocation facilites that I am in and it is having a big impact on download speeds. Download speeds are about 30% slower on average than other colos.
When ever the packet loss numbers get above 5% I try to do about 20-30 traceroutes back to the machines downloading from my server to find the packetloss. Other than an overload NAC OC12 most of the packetloss is hard to pinpoint. Most of the packet loss seems to be just past peering points or inside WCG.
To my knowledge there was never any packet loss due to an “overloaded OC-12”. Sure there may have been sporadic higher than normal latency, but not packet loss. And yes, WCG did have some severe network problems within the past 3 weeks, but those are since been rectified. Rather than WCG backhauling NAC’s connection to them via their ATM network, they’ve directly placed NAC on one of their routers. This switchover uncured last week and thins have been running great since then.
Also, some of your packet loss could be due to NAC’s proactive filters against known DoS attacks. Example: they rate-limit ICMP in and out of their network. So if there’s more than X amount of ICMP traffic across the network, then yes, you might see packet loss, but it’s purely cosmetic.
Aussie Bob 06-11-2003, 04:25 PM Originally posted by dbbrock1
It's going VERY slow for me as well. Takes me about 30 seconds to download our main page which is less than 100KB
Your site comes up quick for me. :)
kicker 06-11-2003, 04:39 PM Originally posted by sjs Also, some of your packet loss could be due to NAC’s proactive filters against known DoS attacks. Example: they rate-limit ICMP in and out of their network. So if there’s more than X amount of ICMP traffic across the network, then yes, you might see packet loss, but it’s purely cosmetic.
That is a pretty nice theory except for a couple of cold hard facts.
1. I measure packet loss every 5 minutes by calculating the tcp retransmitted segments over the total number of segments sent during the same period. Packet loss is higher than other colocation facilites.
2. I also measure download speeds for the last 2 hours of downloads and it is *much* lower than other colocation facilities that are running the exact same machine configurations and loads.
If your tech support immediately throws out submitted pings and traceroutes as cosmetic, how exactly I am supposed to show that there is a problem?
Originally posted by kicker
If your tech support immediately throws out submitted pings and traceroutes as cosmetic, how exactly I am supposed to show that there is a problem?
do a tcp trace perhaps.
kicker 06-11-2003, 04:46 PM Originally posted by sjs
do a tcp trace perhaps.
Uhmmmm...
Those traces that I just posted, those were created using tcptrace to port 80 using TCP SYN packets. Not that it makes a difference since the return packets are always going to be ICMP packets that may get filtered.
Originally posted by kicker
Ummmm...
Those traces that I just posted, those were created using tcptrace to port 80 using TCP SYN packets. Not that it makes a difference since the return packets are always going to be ICMP packets that may get filtered.
i think nac limits certian types of syn packets...
Hostname %Loss Rcv Snt Last Best Avg Worst
1. vlan904.msfc1.oct.nac.net 0% 277 277 1 0 1 52
2. 0095.ge-0-0-0.gbr1.oct.nac.net 0% 277 277 0 0 1 25
3. 0.so-0-1-0.gbr1.nwr.nac.net 0% 276 276 6 1 1 35
4. nycmny2wcx1-atm.wcg.net 0% 276 276 2 1 2 17
5. nycmny2wcx2-oc48.wcg.net 0% 276 276 2 1 2 15
6. pos4-0.core1.NewYork1.Level3.net 0% 276 276 2 2 3 15
7. ae0-51.bbr1.NewYork1.level3.net 0% 276 276 2 2 3 51
8. so-0-1-0.bbr1.Washington1.level3.net 0% 276 276 10 8 9 20
9. gige7-1.ipcolo1.Washington1.Level3.n 0% 276 276 9 8 9 36
10. unknown.Level3.net 0% 276 276 11 10 11 16
11. gigabitethernet4-1.dca2a-fcor-rt1.ne 0% 276 276 12 10 11 35
12. vlan47.dca2b-fdisb-sw2-msfc1.netsrv. 0% 276 276 12 10 11 21
13. 164.109.35.20 0% 276 276 11 10 13 75
the above couldn't be better... but this isn't very pretty:
Hostname %Loss Rcv Snt Last Best Avg Worst
1. vlan904.msfc1.oct.nac.net 0% 1808 1808 0 0 0 141
2. 0095.ge-0-0-0.gbr1.oct.nac.net 0% 1807 1807 1 0 1 75
3. 209.123.182.243 1% 1796 1807 0 0 1 17
4. 66.246.xx.x 36% 1164 1807 58 0 27 136
kicker 06-11-2003, 04:53 PM Originally posted by sjs
i think nac limits certian types of syn packets...
So NAC limits every type of packet that could show there is a network problem but magically lets all the other traffic through?
BTW. NAC just has a large network flap. I would post the traces on that but nobody would believe them.
kicker 06-11-2003, 04:59 PM Originally posted by sjs
the above couldn't be better... but this isn't very pretty:
Hostname %Loss Rcv Snt Last Best Avg Worst
1. vlan904.msfc1.oct.nac.net 0% 1808 1808 0 0 0 141
2. 0095.ge-0-0-0.gbr1.oct.nac.net 0% 1807 1807 1 0 1 75
3. 209.123.182.243 1% 1796 1807 0 0 1 17
4. 66.246.xx.x 36% 1164 1807 58 0 27 136
Just happens to be the network I am on. What the heck is that?
FHDave 06-11-2003, 05:00 PM Originally posted by Aussie Bob
FHDave, thanks for your unbiased personal opinion on NAC. I'm sure we can find a place to file that. :D [/B]
Sure, it's unbiased. After visiting 8-10 datacenters (read: carrier grade datacenters) and tasting NAC network in the past for close to two years (even much earlier before DN comes into the picture), I came to that conclusion even years before today :)
Put it this way, NAC was on our top lists when we decided to move datacenter. There are some (unbiased) reasons why we did not end up with NAC :)
BTW, still don't have any problem to NAC from our servers. Going through WCG.
tandem 06-11-2003, 05:25 PM Originally posted by sjs
well, lets think about this one real quick.. theplanet.com is over in Texas, while NAC is over in NJ. of course you are probably to see lower latency to theplanet.com; Australia is geographically closer to Texas than NJ.
That depends on your POV and where the traffic is moving from (eastward or westward?)... :D
Originally posted by kicker
BTW. NAC just has a large network flap. I would post the traces on that but nobody would believe them.
NAC didn't. I think your ISP might have. Please don’t misconstrue your hosting provider, whomever that may be, with NAC. We are not responsible for their infrastructure.
kicker 06-11-2003, 05:46 PM Originally posted by sjs
NAC didn't. I think your ISP might have. Please don’t misconstrue your hosting provider, whomever that may be, with NAC. We are not responsible for their infrastructure.
And don't confuse me with an idiot who can't tell the difference between a local problem and a problem with the network my servers are on.
When I see 50% packet loss for 1-2 minutes and a large number of traces to random networks drop at the first hop only to come back 1-2 minutes latter. Seems like the classic symptoms of a BGP route table reconverging.
So basically your position is this, NAC's network is great, always was great, and always will be great. Any problems that people are seeing with NAC's network, no matter how many people report them, no matter what evidence they may present, are not truly an indication of a problem with NAC's network but are simply evidence that they themselves are deficient. What's it like to operate a network on faith?
tandem 06-11-2003, 05:52 PM Question: Why is it people like sjs (thankfully, they are just a few) falsely assume that people reading WHT are idiots and gullible dorks when it comes to internetworking know-how? :eek:
sjs, you are freaking wrong. Drop your story lines, take your NAC hat off, and start being objective...
kdblog 06-11-2003, 05:55 PM for the record, i'm getting more timeouts than pageloads on both my DN servers, at the moment, it has been slow all afternoon.
patchwork 06-11-2003, 06:24 PM Originally posted by DedicatedNOW
Have you ever fulfilled or submitted an SLA violation request?
So if a server owner doesn't know about some downtime or outage they get nothing?
If the server owner is not available (asleep etc..) and downtime occurs, they get nothing because they didn't fill in some obscure SLA violation request, what ever one of those is!!
Don't you think it would be a better, fairer, more honest business practice to credit ALL customers if downtime/outages happen, regardless of whether the person is aware of the problem.
Or should all your customers setup 24/7 monitoring of everything just so they know about all these network problems?
Should we be submitting an SLA violation request each day just in case something has happened while we were doing other things.
.
Originally posted by kicker
And don't confuse me with an idiot who can't tell the difference between a local problem and a problem with the network my servers are on.
When I see 50% packet loss for 1-2 minutes and a large number of traces to random networks drop at the first hop only to come back 1-2 minutes latter. Seems like the classic symptoms of a BGP route table reconverging.
bgp table reconverging... lol. i didn't see any bgp sessions flap today. you must be confused.
route-server.phx1>show ip route www.nac.net
Translating "www.nac.net"...domain server (206.165.6.12) [OK]
Routing entry for 207.99.0.0/17, supernet
Known via "bgp 3549", distance 200, metric 100
Tag 7911, type internal
Last update from 206.132.117.1 6d15h ago
Routing Descriptor Blocks:
* 206.132.117.1, from 64.212.104.1, 6d15h ago
Route metric is 100, traffic share count is 1
AS Hops 4, BGP network version 608588370
bling bling.
Originally posted by tandem
Question: Why is it people like sjs (thankfully, they are just a few) falsely assume that people reading WHT are idiots and gullible dorks when it comes to internetworking know-how? :eek:
sjs, you are freaking wrong. Drop your story lines, take your NAC hat off, and start being objective...
what on earth are you talking about? i think you have clue++;
FHDave 06-11-2003, 06:48 PM Originally posted by patchwork
Don't you think it would be a better, fairer, more honest business practice to credit ALL customers if downtime/outages happen, regardless of whether the person is aware of the problem.
I assume you are one of DN's customers? I hate it when people comment on somebody's Terms of Service. The fact that you have (a) dedicated server(s) with them means you have already read and agreed to their TOS. So, don't bring this up later on. If you do not agree with their TOS, then look for other providers.
If you are not their customer, then there is no necessity for you to teach other how to run an "honest business". Even Verio (hosting, at least) does not do refund pro-actively. Or do you want to start telling Verio how to run its business?
Or should all your customers setup 24/7 monitoring of everything just so they know about all these network problems?
IMHO, yes! People get cheap servers for DN. So they can certainly spare some peanuts money to setup a 5 minute or even one minute interval monitoring to their servers.
kicker 06-11-2003, 07:32 PM Originally posted by sjs
bgp table reconverging... lol. i didn't see any bgp sessions flap today. you must be confused.
I never said that the routes NAC is announcing to the world flapped. I said that a large number of the routes that you receive from the rest of the world flapped. The symptoms I described are usually the result of a major peer having problems, a hard clearing of the BGP table, or a cut in your core network.
Am I 100% certain of that? No. I only have a small view of your network. But can you give me a better explanation for the symptoms?
But let me paint a pretty picture for you that you may understand. Attached is a graph of average packet loss of my server at Dedicated Now and servers at UnitedColo, RackShack, and ThePlanet. Requests are randomly distributed across all the servers. Guess which color is DedicatedNow. (Hint, it has the highest packet loss).
Now give me a good reason why this isn't just the network sucking like a hoover?
Originally posted by kicker
But let me paint a pretty picture for you that you may understand. Attached is a graph of average packet loss of my server at Dedicated Now and servers at UnitedColo, RackShack, and ThePlanet. Requests are randomly distributed across all the servers. Guess which color is DedicatedNow. (Hint, it has the highest packet loss).
Well as I posted earlier, I was seeing absolutely no loss through NAC to lets say WCG for an example. However, I was seeing a ton of packet loss to DedicatedNow’s router. Perhaps their router was having issues?
Originally posted by sjs
Well as I posted earlier, I was seeing absolutely no loss through NAC to lets say WCG for an example. However, I was seeing a ton of packet loss to DedicatedNow’s router. Perhaps their router was having issues?
here's another example.. i just ran this mtr 1 minute ago:
Hostname %Loss Rcv Snt Last Best Avg Worst
1. vlan904.msfc1.oct.nac.net 0% 213 213 0 0 0 15
2. 0095.ge-0-0-0.gbr1.oct.nac.net 0% 212 212 9 0 1 36
3. 209.123.182.243 1% 211 212 5 0 2 8
4. 209.123.134.100 1% 211 212 6 1 3 7
tandem 06-11-2003, 07:50 PM Originally posted by sjs
what on earth are you talking about? i think you have clue++;
Here are some utterances from you within the past few hours:
>> you must be confused. (= you're an idiot!)
>> it’s purely cosmetic. (= pings ~5 times the usual caused by NAC's hosed core routers are nothing much to worry about!)
>> Australia is geographically closer to Texas than NJ. (= You just have to take my geographical knowledge at face value... unless you know your world map and can prove that I know nothing!)
Originally posted by tandem
Here are some utterances from you within the past few hours:
>> you must be confused. (= you're an idiot!)
>> it’s purely cosmetic. (= pings ~5 times the usual caused by NAC's core routers are nothing much to worry about!)
>> Australia is geographically closer to Texas than NJ. (= You just have to take my geographical knowledge at face value... unless you know your world map and can prove that I know nothing!)
your personal attacks at me are meaningless. please go back to your gaming server or whatever it is you do to pass time.
tandem 06-11-2003, 07:55 PM ... not as meaningless as your BS. :eek:
Originally posted by tandem
... not as meaningless as your BS. :eek:
no bs, just the facts. anyways.
TotalChoice 06-11-2003, 07:59 PM anyways....
:)
Rochen 06-11-2003, 08:36 PM Originally posted by Aussie Bob
We've got boxes in both DCs and TP is faster from Oz. It's not that noticeable for me. I get about 280 to NAC and around 240 to TP. I'm on Telstra ADSL. Given the vast majority of our client base is USA based, there's not much difference between the 2 there. :)
Doing our little bit for the Aussie trade balance there. ;)
Yup, we have just picked up a box in 'The Planet' this evening. All client servers are at NAC and we have our own website and billing hosted off-network at BurstNET. The BurstNET server is being relocated to 'The Planet', due to the below-par service we have received from Burst. 'The Planet' is certainly looking good so far though :)
My pings to 'The Planet' are slightly slower than those to NAC. I can get to NAC in under 100ms and to 'The Planet' at about 150 - 200ms. This is because NAC have their peering point here in the UK at LINX. I can actually get to NAC without travelling over any external bandwidth providers. I fly down my ISP and then pickup NAC's network at LINX, straight on their network all the way to NJ. Is that cool or what? :D I think other UK users may see the same thing :)
Trace to NAC:
Tracing route to nac.net [207.99.0.69]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms xxxxx.rochen.com [xxx.xxx.xxx.xxx]
2 11 ms 7 ms 7 ms xxx.xxx.xxx.xxx
3 9 ms 8 ms 9 ms gsr01-dn.blueyonder.co.uk [62.31.64.225]
4 21 ms 19 ms 19 ms 172.18.20.61
5 19 ms 19 ms 19 ms gat-edi-pos.telewest.net [194.117.136.10]
6 20 ms 19 ms 20 ms tele2-gat-pos.telewest.net [194.117.136.6]
7 20 ms 19 ms 19 ms ge51-linx-gw1.cableinet.net [194.117.154.9]
8 20 ms 21 ms 22 ms linx.ge-0-0-0.gbr1.ltn.nac.net [195.66.224.94]
9 86 ms 86 ms 86 ms 1209.at-3-1-0.gbr2.nyc.nac.net [209.123.11.245]
10 88 ms 88 ms 88 ms 1194.at-0-0-0.gbr1.oct.nac.net [209.123.11.10]
11 88 ms 87 ms 88 ms 0095.gi-1-1.msfc1.oct.nac.net [64.21.102.2]
12 89 ms 89 ms 90 ms mail.nac.net [207.99.0.69]
Trace complete.
blueyonder.co.uk, telewest.net and cableinet.net are all part of my ISP's network.
I've noticed a small drop in speed lately, as a few sites I frequent are located at NAC...Here's my trace.
Tracing route to nac.net [207.99.0.69]
over a maximum of 30 hops:
1 40 ms 1092 ms 20 ms adsl-68-**-***-254.dsl.chcgil.ameritech.net [**.**.***.***]
2 20 ms 20 ms 20 ms dist2-vlan50.chcgil.ameritech.net [67.38.101.3]
3 20 ms 20 ms 20 ms bb2-g5-0.chcgil.ameritech.net [67.38.101.115]
4 20 ms 20 ms 20 ms bb2-p1-0.chcgil.ameritech.net [67.38.101.154]
5 20 ms 20 ms 20 ms bb1-p6-1.eqchil.sbcglobal.net [65.42.70.226]
6 20 ms 20 ms 20 ms chi1-br1-f4-0-99.gnaps.net [206.223.119.41]
7 40 ms 50 ms 40 ms qcy1-avici1-p1-16-1.gnaps.net [199.232.42.97]
8 50 ms 60 ms 50 ms res1-rs1-p6-1.gnaps.net [199.232.44.25]
9 60 ms 61 ms 60 ms ash1-br1-g1-1.gnaps.net [199.232.44.34]
10 50 ms 50 ms 60 ms 0504.ge-0-0-0.gbr1.ash.nac.net [209.123.24.173]
11 60 ms 60 ms 60 ms 0195.at-0-1-0.gbr1.oct.nac.net [209.123.11.138]
12 61 ms 60 ms 60 ms 0095.gi-1-1.msfc1.oct.nac.net [64.21.102.2]
13 60 ms 60 ms 61 ms mail.nac.net [207.99.0.69]
Trace complete.
kicker 06-11-2003, 08:52 PM Originally posted by sjs
Well as I posted earlier, I was seeing absolutely no loss through NAC to lets say WCG for an example. However, I was seeing a ton of packet loss to DedicatedNow’s router. Perhaps their router was having issues?
That is definately a possiblity. Packet loss as high as I am seeing is usually pretty close to the source. But unfortunately nobody at Dedicated Now is seeing any problems. You willing to loan a clue to somebody at DedicatedNow and tell them their circuit isn't up to snuff?
scottc 06-11-2003, 09:06 PM never mind, I misread that post.
|