Web Hosting Talk : Web Hosting Main Forums : SEO / SEM Discussions : stop search engine from indexing domain?

[JayNL]

Hi!

We build websites, and when its CMS implementation time, we'd love to do that online, because we work in a team. But no matter what we try, our temp domains always get indexed in Google (haven't checked yahoo or MSN).

So far, our development domains are armed with:

a robots.txt file that totally disallowes any access to any folder:

User-agent: *
Disallow: /

We added meta tags:

<meta name="robots" content="noindex,nofollow" />
<meta name="expired" content="01-01-2001" />

But what else can we do? Please advice, we had a mad client who noticed a site in development in the search engines, which make us look like a bunch of kiddies that don't know what we're doing, and to be honest: that's correct I have no clue how to stop this, can anybody please help me with this one? Much appreciated.

Jasper

[the_pm]

Why not simply password protect each folder for each client? Problem solved

[JayNL]

Lol that can be done indeed, thank you very much. But without pw protection, is there any other succesful way?

[the_pm]

Don't ever post a public link to a folder, and make sure your folder indices don't show?

[Barti1987]

What engine picked it up?

Peace,

[JayNL]

Google did, as stated in my post. And indeed the_pm I guess I just drop these domains and start over with new, unknown ones, perfectly password protected. Search engines are smart nowadays, they pick up sites within days, sometimes hours...

[3-rx]

also u can close access of SE via Htaccess file

[JayNL]

thnx for all replies! sounds useful, thanks.

[vetwebhosting]

Hate to bring up a old forum thread, but .htaccess is the only way to go if they are ignoring the robots.txt file. I have read in some forums that less well known search engines skip the robots.txt and will crawl the entire site.

Also like what was mentioned before, if you have a link in your site to those folders, the website crawler will crawl those pages. So if you want a truly hidden directory, do not link to it on your main site.

Hope that helps.

[JayNL]

Thanks for your reply mate! This is still an active issue, so no problem that its a bit late. I tried it with .htaccess last week, I also added meta expired tags and redid the robots.txt to just disallow everything (that was incorrect before) and it worked! No more results in at least Google, Live Search and Yahoo!. Thank god!