Web Hosting Talk : Web Hosting Main Forums : Dedicated Server : Networking Issues

[r557]

I'm getting calls (complaints) from customers that their websites are from time to time unavailable to clients.

I have troubleshooted down to a single isp, but my dedicated server provider is telling me there's nothing wrong with their partners/their network.

The issues are from Rogers, and they are not available in my location. my server provider advised that if i could get some supporting data, then they may be able to help fix this.

question is, without directly knowing anyone who uses Rogers as their ISP, how can i possible collect traceroute data to submit to my host (iweb.com).

[JTY]

You'll need to instruct your customers on how to perform a traceroute, so they can provide the traceroute to you.

[123Andrew]

http://www.robtex.com/dns/cpe.net.cable.rogers.com.html

Or do As JTY said and tell your clients to give a trace.

[atchoooo]

I found this Rogers Looking glass : http://peering.eng.rogers.com/lg

I thought traffic would be going through TORIX but it looks like it goes through the QIX/RISQ (Quebec Internet Exchange). I will have someone investigate if there could be problems with that specific peering point.

The problem you report could also be DNS server related. I do not know your exact situation but if you use external dns servers it could also be something to look at (there could be problems reaching the dns servers from Rogers).

[r557]

ya, the first response i got from iweb was there wasn't any issues.

i have one confirmed case from rogers, now last night someone from alberta (telus) is having issues.

bell & eastlink here in Nova scotia work great. domain is seabrightmarine.ca.

[atchoooo]

We double checked the connectivity with Rogers and did not find any problem.

At the same time when the customer reports the problem it is working fine for you?

You could try adding the site to a monitoring service that has monitoring servers all over the world, for example Pingdom.Com, and add a 1 minute check on the site and see if Pingdom would report problems.

You could also configure a 1 minute check on your DNS server to see if the DNS server always respond correctly or if it times out.

Monitoring the site and the dns server that way would help identify if it is an issue with the server or with a specific provider.

Also make sure network negotiates in full duplex, sometimes if the network adapter on your server resolves in half duplex it can cause intermittent issues that could be similar to what you are reporting. You can verify this with the following command :

mii-tool -v eth0

You can also run speed and file download tests from and to your server, the following URL can probably help : http://test-bgp.iweb.com/

Regards,

[linuxissues]

Originally Posted by r557

ya, the first response i got from iweb was there wasn't any issues.

i have one confirmed case from rogers, now last night someone from alberta (telus) is having issues.

bell & eastlink here in Nova scotia work great. domain is seabrightmarine.ca.

It appears that you are trying hard to pin this on iweb. Did you get a traceroute report from your customers ? Did you submit that to iweb ?

Not sure if you noticed, but iweb posted 2 responses on this thread and you did not seem to notice/interact with them.

[r557]

Originally Posted by atchoooo

We double checked the connectivity with Rogers and did not find any problem.

At the same time when the customer reports the problem it is working fine for you?

You could try adding the site to a monitoring service that has monitoring servers all over the world, for example Pingdom.Com, and add a 1 minute check on the site and see if Pingdom would report problems.

You could also configure a 1 minute check on your DNS server to see if the DNS server always respond correctly or if it times out.

Monitoring the site and the dns server that way would help identify if it is an issue with the server or with a specific provider.

Also make sure network negotiates in full duplex, sometimes if the network adapter on your server resolves in half duplex it can cause intermittent issues that could be similar to what you are reporting. You can verify this with the following command :



You can also run speed and file download tests from and to your server, the following URL can probably help : http://test-bgp.iweb.com/

Regards,

yes, the network priorities are in full duplex.

i am starting to wonder if there is an issue with my private nameservers with my domain registrar. besides contacting their support channels, what method would you recommend verifying everything is functional?

[hhw]

Originally Posted by r557

yes, the network priorities are in full duplex.

i am starting to wonder if there is an issue with my private nameservers with my domain registrar. besides contacting their support channels, what method would you recommend verifying everything is functional?

Just FYI, Rogers is somewhat prone to DNS issues (or at least have been in the past), so it might not even be anything under your control. I've seen them cache bogus entries for certain records for absolutely no reason whatsoever. You should have someone on Rogers do some lookups to check if that's the case.

[r557]

i should add that i'm also using the service uptimerobot.com to monitor the site and it shows it intermittently being offline from time to time but every it says its offline and i check i can access it no problem.

[atchoooo]

i am starting to wonder if there is an issue with my private nameservers with my domain registrar. besides contacting their support channels, what method would you recommend verifying everything is functional?

You could try to add monitoring on their DNS servers (1 minute on the dns port).

[plumsauce]

This may be related to the turn up of signed roots on May 05/2010.

In anticipation of this, some ISP's enabled EDNS on their caches.

eastlink.ca, a Rogers subsidiary seems to have had a problem when they did this on or about May 01. At that time, the Rogers looking glass did not have the same problem.

The problem could be confirmed directly by using their looking glass server at lg.eastlink.ca, which uses the same dns caches as their clients.

The cause seems to be that EDNS was turned on recently, and they use bind, and they use a version of bind that is listed as EDNS capable, and it is one of the versions that fails to implement the RFC properly when used as a cache. The bind bug is officially acknowledged by isc.org, but not very well publicised. This may have something to do with the fact that the bug directly violates a RFC that has their fingerprints all over it. Whoops.

We cannot confirm that eastlink uses bind, or what version.

What we do know is that changing the packet returned to eastlink, modified as if the were indeed running bind with that bug, immediately cured the problem. In that incident, our client reported that 100 percent of the failures were confirmed to be eastlink.ca users. With the fix, they all reported that the site was again reachable.

In short, either the authoritative server has to return an answer that the failing cache is able to understand properly, or, the isp has to employ a cache that will deal with the returned answer properly. One end or the other is going to have to change dns software.

We expect a lot of this to be happening as isps start turning on EDNS without fully understanding the consequences. Luckily, the eastlink admins forced the issue early, allowing us to get ahead of the curve for our clients.

[plumsauce]

forgot to subscribe