hosted by liquidweb


Go Back   Web Hosting Talk : Web Hosting Main Forums : SEO / SEM Discussions : stop search engine from indexing domain?
Reply

Forum Jump

stop search engine from indexing domain?

Reply Post New Thread In SEO / SEM  Discussions Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old 10-09-2007, 09:05 AM
JayNL JayNL is offline
#ibuildwebsitesandstuff
 
Join Date: Aug 2005
Location: Barcelona, Spain
Posts: 3,336

stop search engine from indexing domain?


Hi!

We build websites, and when its CMS implementation time, we'd love to do that online, because we work in a team. But no matter what we try, our temp domains always get indexed in Google (haven't checked yahoo or MSN).

So far, our development domains are armed with:

a robots.txt file that totally disallowes any access to any folder:

Code:
User-agent: *
Disallow: /
We added meta tags:

Code:
<meta name="robots" content="noindex,nofollow" />
<meta name="expired" content="01-01-2001" />
But what else can we do? Please advice, we had a mad client who noticed a site in development in the search engines, which make us look like a bunch of kiddies that don't know what we're doing, and to be honest: that's correct I have no clue how to stop this, can anybody please help me with this one? Much appreciated.

Jasper

__________________
hi there!



Sponsored Links
  #2  
Old 10-09-2007, 09:11 AM
the_pm the_pm is offline
Retired Moderator
 
Join Date: May 2004
Location: Pflugerville, TX
Posts: 11,222
Why not simply password protect each folder for each client? Problem solved

__________________
Studio1337___̴ı̴̴̡̡̡ ̡͌l̡̡̡ ̡͌l̡*̡̡ ̴̡ı̴̴̡ ̡̡͡|̲̲̲͡͡͡ ̲▫̲͡ ̲̲̲͡͡π̲̲͡͡ ̲̲͡▫̲̲͡͡ ̲|̡̡̡ ̡ ̴̡ı̴̡̡ ̡͌l̡̡̡̡.__Web Design

  #3  
Old 10-09-2007, 09:13 AM
JayNL JayNL is offline
#ibuildwebsitesandstuff
 
Join Date: Aug 2005
Location: Barcelona, Spain
Posts: 3,336
Lol that can be done indeed, thank you very much. But without pw protection, is there any other succesful way?

__________________
hi there!

Sponsored Links
  #4  
Old 10-09-2007, 10:10 AM
the_pm the_pm is offline
Retired Moderator
 
Join Date: May 2004
Location: Pflugerville, TX
Posts: 11,222
Don't ever post a public link to a folder, and make sure your folder indices don't show?

__________________
Studio1337___̴ı̴̴̡̡̡ ̡͌l̡̡̡ ̡͌l̡*̡̡ ̴̡ı̴̴̡ ̡̡͡|̲̲̲͡͡͡ ̲▫̲͡ ̲̲̲͡͡π̲̲͡͡ ̲̲͡▫̲̲͡͡ ̲|̡̡̡ ̡ ̴̡ı̴̡̡ ̡͌l̡̡̡̡.__Web Design

  #5  
Old 10-09-2007, 05:22 PM
Barti1987 Barti1987 is offline
Web Hosting Master
 
Join Date: Mar 2004
Location: USA
Posts: 4,342
What engine picked it up?

Peace,

__________________
Testing 1.. Testing 1..2.. Testing 1..2..3...

  #6  
Old 10-09-2007, 05:41 PM
JayNL JayNL is offline
#ibuildwebsitesandstuff
 
Join Date: Aug 2005
Location: Barcelona, Spain
Posts: 3,336
Google did, as stated in my post. And indeed the_pm I guess I just drop these domains and start over with new, unknown ones, perfectly password protected. Search engines are smart nowadays, they pick up sites within days, sometimes hours...

__________________
hi there!

  #7  
Old 10-09-2007, 07:56 PM
3-rx 3-rx is offline
Web Hosting Master
 
Join Date: Jan 2005
Posts: 609
also u can close access of SE via Htaccess file

__________________
Your Health Encyclopedia
Medical and health consumer information resources containing comprehensive and unbiased information in patient-friendly language

  #8  
Old 10-12-2007, 07:40 AM
JayNL JayNL is offline
#ibuildwebsitesandstuff
 
Join Date: Aug 2005
Location: Barcelona, Spain
Posts: 3,336
thnx for all replies! sounds useful, thanks.

__________________
hi there!

  #9  
Old 10-15-2007, 01:39 PM
vetwebhosting vetwebhosting is offline
Junior Guru Wannabe
 
Join Date: Oct 2007
Posts: 30
Hate to bring up a old forum thread, but .htaccess is the only way to go if they are ignoring the robots.txt file. I have read in some forums that less well known search engines skip the robots.txt and will crawl the entire site.

Also like what was mentioned before, if you have a link in your site to those folders, the website crawler will crawl those pages. So if you want a truly hidden directory, do not link to it on your main site.

Hope that helps.

  #10  
Old 10-15-2007, 05:52 PM
JayNL JayNL is offline
#ibuildwebsitesandstuff
 
Join Date: Aug 2005
Location: Barcelona, Spain
Posts: 3,336
Thanks for your reply mate! This is still an active issue, so no problem that its a bit late. I tried it with .htaccess last week, I also added meta expired tags and redid the robots.txt to just disallow everything (that was incorrect before) and it worked! No more results in at least Google, Live Search and Yahoo!. Thank god!

__________________
hi there!

Reply

Related posts from TheWhir.com
Title Type Date Posted
UK Police Continue Crackdown on Pirated Content with Torrentz.eu Domain Name Suspension Web Hosting News 2014-05-26 12:30:38
Mistakes To Avoid When Launching A New Website Design Blog 2014-01-10 10:53:50
Why You Should Embrace Google Plus Blog 2013-03-08 09:18:17
ICM Registry Launches .XXX Domain Discount Tied to Search Marketing Credit Web Hosting News 2012-11-01 16:49:31
.XXX Operator ICM Registry Releases More than 1,000 Premium Generic Keyword Domain Names Web Hosting News 2012-10-24 12:25:47


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
Advertisement:
Web Hosting News:
WHT Membership
WHT Membership



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?