Web Hosting Talk







View Full Version : making link not spiderable by bots


Jaffery
02-04-2006, 03:31 AM
I have a requirement where I have to give link on a page but don't want them to be spiderable by Search engine bots.
Those links are already linked from other pages and on this very pages we want to allow user to go through that link only ...

I don't have to block those pages from search engine completely.. all is that do not want search engine to goto those links from this specific page.

Any idea.. I once heard that using javascript u can make simple link but as I donnno anything about javasctip so I donno hw...
\

dollar
02-04-2006, 03:35 AM
Give this a try: http://www.searchengineworld.com/robots/robots_tutorial.htm

Jaffery
02-04-2006, 09:47 AM
Thanx mate but I have already mentioned that I don't have to completely block those pages from search engine.. Its only that I want links not spiderable from a particular page.

Richard
02-04-2006, 10:04 AM
Let me get this straight. You don't want foo.com to be spidered by search engines accessing Page 1, however anything on Page 2 is fine, even if it includes foo.com aswell? I think you'd be better off just blocking the page itself. I don't think you can block certain links on a page, and have those page links open for crawling from another location.

JayC
02-04-2006, 08:14 PM
If I understand what you're talking about, this might be what you're looking for:
http://www.google.com/webmasters/bot.html#www

That's Google's information on rel="nofollow" which, while Google pioneered it, is now also supported by MSN and Yahoo bots.

NyteOwl
02-05-2006, 06:11 PM
rel=nofollow will prevent a spoder that acknowledges the tag from following it, but the link itself will be included in the spider's sweep which is I think what the OP wants to avoid (I could be wrong of course.)

B33R
02-05-2006, 07:07 PM
rel=nofollow will prevent a spoder that acknowledges the tag from following it, but the link itself will be included in the spider's sweep which is I think what the OP wants to avoid (I could be wrong of course.)That's the impression I got too.

It may be possible to put the link in an external html file - use an iframe or something to display it on the first page and then use robots.txt to block the external html file address so the bots don't visit it.
It would be visible to users but because bots don't visit any of the addresses you block via robots.txt they shouldn't "see" it.

Jaffery
02-06-2006, 09:04 AM
Well.. I think google's guideline for particular link is what I need. Its not that I want to block particular pages not to get spidered at all.. I just want that google or other SE don't count them from a particular page..

Thnx guys..

Only one doubt.. do this way google will not spidered that link from other pages too from where I want it to get spidered ?

JayC
02-07-2006, 08:06 PM
Only one doubt.. do this way google will not spidered that link from other pages too from where I want it to get spidered ?No, that's why I suggested it as matching up with what I thought you were trying to do. The pages would still be indexed as long as there are other links pointing to them. Effectively rel=nofollow means that particular link should be ignored -- it does not mean that the linked page (because it doesn't even have to be on your own site) is completely ignored.

JayC
02-07-2006, 08:10 PM
rel=nofollow will prevent a spoder that acknowledges the tag from following it, but the link itself will be included in the spider's sweep which is I think what the OP wants to avoid (I could be wrong of course.)I'm not clear as to what you mean by "included in the spider's sweep," but Google's word on this approach is that the link is effectively ignored. It's not followed for indexing purposes, it's not included in PageRank calculation, and any anchor text attached to it is not associated with the linked page.