Results 1 to 4 of 4
  1. #1
    Join Date
    Jan 2002
    Location
    Colombia, South America
    Posts
    521

    Question Robots.txt file on cPanel shared - how to exclude subdomain?

    I have a domain on a cPanel shared server and have a robots.txt file for it.

    There is a subdomain and I would like to prevent (well behaved) bots from crawling the subdomain.

    The subdomain is secure.example.com and has a Feedback Form, etc.

    Questions: (a) Do I add something to the existing robots.txt file?
    (b) Do I need a new robots.txt file, for the subdomain?
    (c) What lines do I add?

    TIA! Lanny

  2. #2
    Join Date
    Jan 2010
    Location
    San Francisco
    Posts
    1,799
    You would create a file named "robots.txt" with the following contents:

    Code:
    User-agent: *
    Disallow: /
    and place it in the document root of your subdomain.

    More info: http://www.robotstxt.org/robotstxt.html

    [EDIT]: Sorry, jumped the gun a bit. Since this is a cPanel server, a subdomain is added as a subdirectory in your main domain's document root. You should also edit your existing robots.txt of your main domain to exclude this subdirectory, in addition to the new robots.txt for your subdomain.

    So if secure.example.com is in a subdirectory named "secure":

    Code:
    User-agent: *
    Disallow: /secure/
    Last edited by Hsunami; 03-23-2011 at 10:09 AM.

  3. #3
    Join Date
    Jan 2002
    Location
    Colombia, South America
    Posts
    521

    @WickedFactor Thank you!

    Thank you, for all of the information you provided. I am going to add a robots.txt file to the "secure" subdomain and also will add the lines you suggested to the existing robots.txt file. Much appreciated! Lanny

  4. #4
    Join Date
    Jan 2010
    Location
    San Francisco
    Posts
    1,799
    No problem.

Similar Threads

  1. Robots.txt file issue
    By Net_Spy in forum Web Hosting
    Replies: 5
    Last Post: 10-26-2009, 10:45 AM
  2. robots.txt file ?
    By i-background in forum Web Design and Content
    Replies: 2
    Last Post: 05-13-2006, 07:52 PM
  3. robots.txt file and sub domains
    By bohemian in forum Hosting Security and Technology
    Replies: 2
    Last Post: 09-13-2004, 12:55 AM
  4. correct robots.txt file?
    By illusionist in forum Web Hosting Lounge
    Replies: 6
    Last Post: 12-18-2002, 06:17 PM
  5. Warning about robots.txt file
    By scslawin in forum Hosting Security and Technology
    Replies: 6
    Last Post: 05-09-2001, 03:51 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •