Results 1 to 2 of 2
  1. #1
    Join Date
    Jul 2006
    Posts
    42

    robots.txt with wildcard?

    Googlebot is eating up about 30GB bandwidth per month.
    I want to prevent him (and other robots) from spidering:

    http://www.mysite.com/directory1/page1.php

    and all derivatives thereof (e.g. http://www.mysite.com/directory1/pag...riable1=value1)

    Is this possible with robots.txt? How would the code look?

    Will this work?
    User-agent: *
    Disallow: /directory1/page1.php

    or do I need to somehow specify a wildcard, e.g.
    User-agent: *
    Disallow: /directory1/page1.php?*
    Thanks for your help!

  2. #2
    Join Date
    Oct 2003
    Location
    Chattanooga
    Posts
    8,985
    Matt (if this is Matt!),

    You might have better luck on this forum here:
    http://www.webmasterworld.com/forum93/page2.htm

    They seem to be robots.txt gurus
    David
    Web hosting by Fused For businesses with more important things to do than worry about their hosting.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •