Web Hosting Talk







View Full Version : Best way to make search engines to crall 15000 sub-domains


bugsoft
03-30-2005, 12:30 PM
Hi,

I have a question for SEO experts or someone familiar with this.

I have one domain with about 15000 subdomains.

The master domain is spidered daily/weekly by all the major search engines.

I want to make search engines to spider all the 15,000 subdomains. What's the best way for doing this?

I've added an hidden tag within the master domain that displays a directory with all sub-domains but the search engines didn't bought it.

What's the easiest way to make the search engines to spider and index all those 15,000 sub-domains?

Thank's

Mesum
03-30-2005, 03:21 PM
A sitemap that links to all those subdomains maybe?

Mesum
03-30-2005, 03:24 PM
Also, maybe some kinda script to randomly shows 5 or 6 subdomains at ones at your main site.

dawhb
03-30-2005, 04:58 PM
Maybe you violate the crawler's rules and Search Engines define your subdomains as spamming ones.

Best Regards
Dimitar

wolseley
03-30-2005, 08:19 PM
Don't use a hidden tag. Search engines only follow links which they think a human could - i.e. not links which hidden, overly small or in a text colour the same as the background. Although you are doing this for a semi-legitimate reason, this is usually done in order to try and cheat the search engines and they don't like it.

The best thing to do, as already suggested, is to create a site map. Just have a normal link somewhere on your site which goes to a page with links to the subdomains. As you have so many domains, you are probably best off having several `site map' pags, i.e. have a page which links to 100 subdomains with a `next' link at the bottom which links to the next 100 and so on.

bugsoft
03-31-2005, 07:38 AM
Wha the idden tag is exactly that. A Site Map and it's not actually hidden it's a text in white thus mixes it in background but they stil l where not spidered

Any thoughts?