Results 1 to 10 of 10
  1. #1
    Join Date
    Aug 2005
    Location
    Barcelona, Spain
    Posts
    3,371

    stop search engine from indexing domain?

    Hi!

    We build websites, and when its CMS implementation time, we'd love to do that online, because we work in a team. But no matter what we try, our temp domains always get indexed in Google (haven't checked yahoo or MSN).

    So far, our development domains are armed with:

    a robots.txt file that totally disallowes any access to any folder:

    Code:
    User-agent: *
    Disallow: /
    We added meta tags:

    Code:
    <meta name="robots" content="noindex,nofollow" />
    <meta name="expired" content="01-01-2001" />
    But what else can we do? Please advice, we had a mad client who noticed a site in development in the search engines, which make us look like a bunch of kiddies that don't know what we're doing, and to be honest: that's correct I have no clue how to stop this, can anybody please help me with this one? Much appreciated.

    Jasper
    hi there!

  2. #2
    Join Date
    May 2004
    Location
    Pflugerville, TX
    Posts
    11,222
    Why not simply password protect each folder for each client? Problem solved
    Studio1337___̴ı̴̴̡̡̡ ̡͌l̡̡̡ ̡͌l̡*̡̡ ̴̡ı̴̴̡ ̡̡͡|̲̲̲͡͡͡ ̲▫̲͡ ̲̲̲͡͡π̲̲͡͡ ̲̲͡▫̲̲͡͡ ̲|̡̡̡ ̡ ̴̡ı̴̡̡ ̡͌l̡̡̡̡.__Web Design

  3. #3
    Join Date
    Aug 2005
    Location
    Barcelona, Spain
    Posts
    3,371
    Lol that can be done indeed, thank you very much. But without pw protection, is there any other succesful way?
    hi there!

  4. #4
    Join Date
    May 2004
    Location
    Pflugerville, TX
    Posts
    11,222
    Don't ever post a public link to a folder, and make sure your folder indices don't show?
    Studio1337___̴ı̴̴̡̡̡ ̡͌l̡̡̡ ̡͌l̡*̡̡ ̴̡ı̴̴̡ ̡̡͡|̲̲̲͡͡͡ ̲▫̲͡ ̲̲̲͡͡π̲̲͡͡ ̲̲͡▫̲̲͡͡ ̲|̡̡̡ ̡ ̴̡ı̴̡̡ ̡͌l̡̡̡̡.__Web Design

  5. #5
    Join Date
    Mar 2004
    Location
    USA
    Posts
    4,342
    What engine picked it up?

    Peace,
    Testing 1.. Testing 1..2.. Testing 1..2..3...

  6. #6
    Join Date
    Aug 2005
    Location
    Barcelona, Spain
    Posts
    3,371
    Google did, as stated in my post. And indeed the_pm I guess I just drop these domains and start over with new, unknown ones, perfectly password protected. Search engines are smart nowadays, they pick up sites within days, sometimes hours...
    hi there!

  7. #7
    also u can close access of SE via Htaccess file
    Your Health Encyclopedia
    Medical and health consumer information resources containing comprehensive and unbiased information in patient-friendly language

  8. #8
    Join Date
    Aug 2005
    Location
    Barcelona, Spain
    Posts
    3,371
    thnx for all replies! sounds useful, thanks.
    hi there!

  9. #9
    Join Date
    Oct 2007
    Posts
    30
    Hate to bring up a old forum thread, but .htaccess is the only way to go if they are ignoring the robots.txt file. I have read in some forums that less well known search engines skip the robots.txt and will crawl the entire site.

    Also like what was mentioned before, if you have a link in your site to those folders, the website crawler will crawl those pages. So if you want a truly hidden directory, do not link to it on your main site.

    Hope that helps.

  10. #10
    Join Date
    Aug 2005
    Location
    Barcelona, Spain
    Posts
    3,371
    Thanks for your reply mate! This is still an active issue, so no problem that its a bit late. I tried it with .htaccess last week, I also added meta expired tags and redid the robots.txt to just disallow everything (that was incorrect before) and it worked! No more results in at least Google, Live Search and Yahoo!. Thank god!
    hi there!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •