Web Hosting Talk : Web Hosting Main Forums : Web Hosting : SAVVIS has routing issues?

[IT_Architect]

We've been having off-and-on slow-downs and have traced them to issues within SAVVIS routing. We bought servers from Servstra/LayeredTech/SAVVIS to get into the SAVVIS data center so we wouldn't have these problems. Traceroutes performed on servers outside of the data center result in no errors. Traceroutes performed to servers inside SAVVIS always have 2 to 4 errors, with the timeout set at the default 4 seconds. All errors occur when it hits the SAVVIS network. The routing problems are unrelated to traffic volume. They have been happening since yesterday and still going on at 5 AM EDT. This problem has no doubt been going on for sometime and now I'm beginning to doubt the wisdom of being in the SAVVIS data center. Server-to-server connections between two of our SAVVIS servers inside of the data center on the same Class B, seem fast. Traceroutes done even from the servers in the data center to servers outside the data center manifest the same problem as doing a traceroute to them from the outside.

[MrRadic]

Which Savvis datacenter are you referring to? Our servers are in the Jersey City, NJ datacenter and have had absolutely no problems -- just incredible pings, blazing fast bandwidth .

[IT_Architect]

Originally Posted by MrRadic

Which Savvis datacenter are you referring to? Our servers are in the Jersey City, NJ datacenter and have had absolutely no problems -- just incredible pings, blazing fast bandwidth .

These are the Texas ones.

When I checked originally, I settled here because if was the fastest I could find of any data center. However, I'm seeing, and most likely have been for quite awhile, a problem at the SAVVIS TX center. The border router seems to be OK in this most recent one, but it had problems on the border router earlier as well.

1 1 ms <1 ms <1 ms 10.1.128.1
2 11 ms 9 ms 9 ms My external IP
3 11 ms 9 ms 18 ms 12.244.250.193
4 16 ms 15 ms 13 ms 12.118.112.9
5 25 ms 93 ms 21 ms tbr2-p012401.dtrmi.ip.att.net [12.123.139.142]
6 24 ms 95 ms 22 ms tbr2-cl18.cgcil.ip.att.net [12.122.10.134]
7 25 ms 19 ms 19 ms ggr2-p390.cgcil.ip.att.net [12.123.6.37]
8 23 ms 20 ms 30 ms 192.205.33.154
9 21 ms 23 ms 22 ms dcr2-so-5-0-0.Chicago.savvis.net [204.70.192.46]
10 45 ms 47 ms 43 ms dcr1-so-4-2-0.Denver.savvis.net [204.70.193.221]
11 56 ms 55 ms 64 ms dcr1-so-0-0-0.dallas.savvis.net [204.70.192.94]
12 56 ms 53 ms 52 ms bhr1-pos-12-0.fortworthda1.savvis.net [208.172.131.82]
13 49 ms 61 ms 45 ms 216.39.66.26
14 * 55 ms * 216.39.66.26
15 * * 48 ms 154.205.My.IP.reverse.layeredtech.com [72.36.Server.IP]

This happens with both of our servers in the DC. The servers are configured completely differently, and another one that I don't own or maintain has the same problem. It's just as big a mess when you run traceroute from the server.

[MrRadic]

If you'd like, I can try a tracert from one our servers -- it may just be your ISP, let me know.

[IT_Architect]

Originally Posted by MrRadic

If you'd like, I can try a tracert from one our servers -- it may just be your ISP, let me know.

I PMed you with a server address of one of them. However, it won't be my ISP because it works every place else. It's only SAVVIS TX where I see the problem. When I do a trace to your server, it comes out fine.

[MrRadic]

I also sent you a PM with this --

Tracing route to XXXXX.org [72.36.HIS.IP]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms reliablesite.net [64.237.33.193]
2 <1 ms <1 ms <1 ms 0.te1-1.cr1.ewr1.choopa.net [64.237.32.158]
3 <1 ms <1 ms <1 ms ge-6-21.car2.Newark1.Level3.net [4.79.236.9]
4 <1 ms <1 ms <1 ms ae-1-55.bbr1.Newark1.Level3.net [4.68.99.129]
5 <1 ms 3 ms <1 ms ae-0-0.bbr1.NewYork1.Level3.net [64.159.1.41]
6 1 ms 1 ms 1 ms ge-6-0-0-55.gar3.NewYork1.Level3.net [4.68.97.132]
7 1 ms <1 ms <1 ms dcr6-so-6-1-0.NewYork.savvis.net [4.68.127.206]
8 5 ms 5 ms 7 ms bcs2-so-4-0-0.Washington.savvis.net [204.70.192.1]
9 5 ms 19 ms 5 ms bcs1-so-7-0-0.Washington.savvis.net [204.70.192.33]
10 38 ms 39 ms 19 ms dcr1-so-3-0-0.Atlanta.savvis.net [204.70.192.53]
11 39 ms 19 ms 19 ms dcr2-as0-0.Atlanta.savvis.net [204.70.192.42]
12 40 ms 40 ms 39 ms csr2-ve240.fortworthda1.savvis.net [216.39.64.35]
13 40 ms 38 ms 40 ms 216.39.66.26
14 39 ms 39 ms * bhr1-pos-12-0.fortworthda1.savvis.net [208.172.131.82]
15 40 ms 40 ms 39 ms 154.205.HIS.IP.reverse.layeredtech.com [72.36SERVER.IP]

Trace complete.

[IT_Architect]

Yours is as good as I've gotten. You got bombed only once at the perimeter, and none inside. Most of mine didn't bomb at the perimeter. I just did it twice in a row. One time it bombed twice, and minute later 4 times.

[Cirrostratus]

The results you are seeing are normal. Your traceroutes are trying to hit devices which do not have reverse DNS records setup for them so you will get a failure there and also do not respond to ICMP packet or treat them with a low priority. Once you are inside our network and pass the edge routers we use RFC 1918 IP space which is also blocked on outbound packets so any inbound traceroute will show a * or similar on the first hop in.

$ traceroute -n 72.36.154.xxx
traceroute to 72.36.154.xxx(72.36.154.xxx), 64 hops max, 40 byte packets
1 209.67.208.177 0.348 ms 0.302 ms 0.307 ms <-- Pod 1 Host
2 10.1.3.1 0.495 ms 0.450 ms 0.382 ms
3 216.39.69.49 0.426 ms 0.330 ms 0.407 ms
4 216.39.64.41 0.488 ms 0.375 ms 0.297 ms
5 216.39.64.26 0.945 ms 0.663 ms 0.610 ms
6 216.39.69.238 1.131 ms 1.056 ms 0.970 ms
7 10.1.4.14 0.933 ms 0.879 ms 0.853 ms
8 72.36.154.xxx 0.759 ms 0.714 ms 0.669 ms <-- Your Host

Traceroute to your server from our POD1 network located in the same DC but using a 100% diverse network to your host which is on our POD2 network. You can see that with the -n flag being used there is no errors and shows the IP of the host instead of where you typically see a *.

Use the -n flag which prevents a nslookup of each IP and you will get the same results.

You can also test the througput of the connection by downloading a test file via wget / fetch on your host and it should get 1-1.2MB/s for a 10mb/s or 10-12MB/s for a 100mb/s link.

Thanks,

Jeremy

[IT_Architect]

Jeremy,

>Once you are inside our network and pass the edge routers we use RFC 1918 IP space which is also blocked on outbound packets so any inbound traceroute will show a * or similar on the first hop in.<

Thank you HUGE for that reply. We have a ZABBIX install that we are not done with for this server. This has been huge frustrating because the slow-downs happen at random times and do not follow the normal daily traffic curve. When we need to move a large site to a different data center to server our customers, that's a problem. When the slow downs occur, the server load goes down with it. We were just about close on two more high-end servers from you with mirror and fail-over and are getting concerned with what we see.

I will post back results when I get them,
Thanks again!

[IT_Architect]

Originally Posted by LTADMIN

The results you are seeing are normal. Your traceroutes are trying to hit devices which do not have reverse DNS records setup for them so you will get a failure there and also do not respond to ICMP packet or treat them with a low priority. Once you are inside our network and pass the edge routers we use RFC 1918 IP space which is also blocked on outbound packets so any inbound traceroute will show a * or similar on the first hop in.

I understand what you are saying and that these * may be normal. It's just that I don't get them from the other data centers. I will continue to investigate.

Here is my latest without looking up the IP:
Trace
1 1 ms <1 ms <1 ms 10.1.128.1
2 18 ms 10 ms 8 ms 73.43.source.address
3 11 ms 17 ms 8 ms 12.244.250.193
4 26 ms 15 ms 13 ms 12.118.112.9
5 23 ms 21 ms 25 ms 12.123.139.142
6 22 ms 22 ms 21 ms 12.122.10.134
7 30 ms 103 ms 21 ms 12.123.6.37
8 56 ms 21 ms 23 ms 192.205.33.154
9 22 ms 21 ms 20 ms 204.70.192.46
10 45 ms 43 ms 48 ms 204.70.192.98
11 60 ms 64 ms 53 ms 208.172.129.230
12 55 ms 53 ms 53 ms 208.172.131.82
13 61 ms 47 ms 59 ms 216.39.64.59
14 * * 56 ms 216.39.66.26
15 49 ms * * 72.36.destination address
16 48 ms 54 ms 47 ms 72.36.destination address
Trace complete.

[MrRadic]

Usually a * means a packet was lost due to no-reply. But if Jeremy says that the routers treat those packets with low importance, it could just ignoring them at times -- which also means that during no load times (3am) the routers should almost always respond to each ping.

[IT_Architect]

Originally Posted by MrRadic

Usually a * means a packet was lost due to no-reply. But if Jeremy says that the routers treat those packets with low importance, it could just ignoring them at times -- which also means that during no load times (3am) the routers should almost always respond to each ping.

I suspect that will not be the case unless something changes. I will devise a way to feed ZABBIX for both internal and external routing.

[[MaxX]]

I have the same issue, pings get lost at the border router, although pings to my server itself are NEVER lost

I guess the border router treats pings as low priority as he said.. ( I am in India so these ping times are normal to US servers)

1 <1 ms <1 ms <1 ms 192.168.1.1
2 28 ms 27 ms 25 ms 61.17.201.1
3 53 ms 36 ms 27 ms 202.54.10.62
4 32 ms 31 ms 29 ms 203.197.72.150
5 44 ms 31 ms 31 ms 202.54.2.162
6 252 ms 231 ms 247 ms 202.54.2.130
7 366 ms 311 ms 311 ms 204.70.151.29
8 312 ms 311 ms 311 ms 204.70.193.45
9 297 ms 295 ms 297 ms 204.70.192.1
10 296 ms 311 ms 297 ms 204.70.192.33
11 299 ms 300 ms 309 ms 204.70.192.53
12 298 ms 309 ms 309 ms 204.70.192.42
13 302 ms 301 ms 547 ms 216.39.64.59
14 350 ms 307 ms 299 ms 216.39.66.26
15 300 ms 299 ms * 208.172.131.82
16 299 ms 299 ms 301 ms 72.36.my.ip

[IT_Architect]

Originally Posted by [MaxX]

I have the same issue, pings get lost at the border router, although pings to my server itself are NEVER lost

I've been doing quite a few tests. Here is a test from midnight last night:
1 1 ms <1 ms 1 ms 10.1.128.1
2 12 ms 8 ms 9 ms 73.43.My.IP
3 10 ms 9 ms 9 ms 12.244.250.193
4 17 ms 26 ms 13 ms 12.118.112.9
5 35 ms 22 ms 21 ms 12.123.139.142
6 24 ms 21 ms 21 ms 12.122.10.134
7 22 ms 21 ms 21 ms 12.123.6.37
8 22 ms 21 ms 21 ms 192.205.33.154
9 * 29 ms * 204.70.192.46
10 46 ms 46 ms 44 ms 204.70.193.221
11 55 ms 52 ms 59 ms 204.70.192.94
12 56 ms 52 ms 63 ms 208.172.131.82
13 49 ms 59 ms 48 ms 216.39.66.26
14 * 55 ms * 216.39.66.26
15 * * 48 ms 72.36.Server.IP
Note:
- There are no stars except on the SAVVIS network.
- Check the last three lines

I ran a packet sniffer on an FTP session this morning
17248 My.Server DELL_9400 82 0:05:21.056441 FTP Data Src= 20,Dst= 5006,.A....,S=4062253421,L= 0,A=2774573529,W=33304 TCP Slow Segment Recovery (1.040303 seconds from packet 17,184)
17249 My.Server DELL_9400 82 0:05:21.056716 FTP Data Src= 20,Dst= 5006,.A....,S=4062253421,L= 0,A=2774573529,W=31856
17250 My.Server DELL_9400 70 0:05:21.087577 FTP Data Src= 20,Dst= 5006,.A....,S=4062253421,L= 0,A=2774621313,W= 9412 TCP Slow Segment Recovery (1.038276 seconds from packet 17,186)
17251 My.Server DELL_9400 70 0:05:21.087854 FTP Data Src= 20,Dst= 5006,.A....,S=4062253421,L= 0,A=2774621313,W=13508
17252 My.Server DELL_9400 70 0:05:21.088145 FTP Data Src= 20,Dst= 5006,.A....,S=4062253421,L= 0,A=2774621313,W=21700
17253 My.Server DELL_9400 70 0:05:21.088400 FTP Data Src= 20,Dst= 5006,.A....,S=4062253421,L= 0,A=2774621313,W=17604
17254 My.Server DELL_9400 70 0:05:21.088689 FTP Data Src= 20,Dst= 5006,.A....,S=4062253421,L= 0,A=2774621313,W=25796
17255 My.Server DELL_9400 70 0:05:21.088956 FTP Data Src= 20,Dst= 5006,.A....,S=4062253421,L= 0,A=2774621313,W=29892
17256 My.Server DELL_9400 70 0:05:21.089223 FTP Data Src= 20,Dst= 5006,.A....,S=4062253421,L= 0,A=2774621313,W=33304
17257 My.Server DELL_9400 82 0:05:21.158471 FTP Data Src= 20,Dst= 5006,.A....,S=4062253421,L= 0,A=2774622761,W=33304
17258 My.Server DELL_9400 90 0:05:21.192125 FTP Data Src= 20,Dst= 5006,.A....,S=4062253421,L= 0,A=2774622761,W=33304 TCP Slow Acknowledgement (1.001926 seconds from packet 17,202)
17259 DELL_9400 My.Server 1518 0:05:21.192300 FTP Data Src= 5006,Dst= 20,.A....,S=2774656065,L= 1448,A=4062253421,W=65535
17260 My.Server DELL_9400 90 0:05:21.218485 FTP Data Src= 20,Dst= 5006,.A....,S=4062253421,L= 0,A=2774622761,W=33304 TCP Slow Acknowledgement (1.000211 seconds from packet 17,204)
17261 DELL_9400 My.Server 1506 0:05:21.218647 FTP Data Src= 5006,Dst= 20,.A....,S=2774622761,L= 1436,A=4062253421,W=65535 TCP Retransmission
17262 DELL_9400 My.Server 82 0:05:21.218712 FTP Data Src= 5006,Dst= 20,.A....,S=2774624197,L= 12,A=4062253421,W=65535 TCP Retransmission
17263 DELL_9400 My.Server 1506 0:05:21.218770 FTP Data Src= 5006,Dst= 20,.A....,S=2774625657,L= 1436,A=4062253421,W=65535 Non-Responsive Server
17264 DELL_9400 My.Server 82 0:05:21.218823 FTP Data Src= 5006,Dst= 20,.A....,S=2774627093,L= 12,A=4062253421,W=65535 Non-Responsive Server

However, the trace was not full of this, and the speed would be limited to up stream speed. However, these kinds of errors I didn't get on the non-SAVVIS server. The download went well with not more than the normal errors.

You can also test the througput of the connection by downloading a test file via wget / fetch on your host and it should get 1-1.2MB/s for a 10mb/s or 10-12MB/s for a 100mb/s link.

The download speed of the test file within SAVVIS wasn't that bad. It ran about 10 seconds on average for a 64MB file. It takes about 10 seconds, which indicates about 55%-65% of what it should be. Nothing that would cause slowdowns like we are seeing. This problem appears to the user that it is hung, and then takes off again. Sometimes it's fast, but never for long, and it's slowly getting worse. The server loads are naturally going down because people can't navigate the pages. I'm super glad we move the hotel site out of there. I'm not waiting much longer for the weather site. We've spent untold thousands chasing this problem.

[WO-Jacob]

Hey guys, when you're trying to hide your IP, be tactful about it...

15 * * 48 ms 154.205.My.IP.reverse.layeredtech.com [72.36.Server.IP]

... reverse dns records ... default to displaying in reverse. So, you blocked out the first two octets on the reverse resolve, and the last two on the direct ip... and ... one can put one and the other together to get your IP.

Just figured I would point that out as I'd seen a couple people do it in this thread.