hosted by liquidweb


Go Back   Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : prevent wget downloads
Reply

Forum Jump

prevent wget downloads

Reply Post New Thread In Hosting Security and Technology Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old
Junior Guru Wannabe
 
Join Date: Feb 2003
Posts: 34

prevent wget downloads


hi,

how to prevent wget downloads of my website

help required

thank you



Sponsored Links
  #2  
Old
Web Hosting Guru
 
Join Date: Jul 2004
Location: Reporting Live from Marrz
Posts: 254
.htaccess:

Code:
SetEnvIfNoCase User-Agent "^Wget" bad_bot

<Limit GET POST>
   Order Allow,Deny
   Allow from all
   Deny from env=bad_bot
</Limit>

  #3  
Old
Junior Guru Wannabe
 
Join Date: Feb 2003
Posts: 34
I have figured out.. thank you "SupaDucta"

SetEnvIfNoCase User-Agent "^Wget" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.5.3" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.6" bad_bot
<Files ~ "\.(html|pdf|mp3|zip|rar|exe|gif|jpe?g|png|php|jsp) $">
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</files>

Sponsored Links
  #4  
Old
Web Hosting Guru
 
Join Date: Jul 2004
Location: Reporting Live from Marrz
Posts: 254
This line:

Code:
SetEnvIfNoCase User-Agent "^Wget" bad_bot
makes the following two lines unnecessary:

Code:
SetEnvIfNoCase User-Agent "^Wget/1.5.3" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.6" bad_bot
and if you use this:

Code:
<Limit GET POST>
   Order Allow,Deny
   Allow from all
   Deny from env=bad_bot
</Limit>
this blocks Wget completely.

While this:

Code:
<Files ~ "\.(html|pdf|mp3|zip|rar|exe|gif|jpe?g|png|php|jsp) $">
   Order Allow,Deny
   Allow from all
   Deny from env=bad_bot
</files>
would block only the related extensions.

  #5  
Old
Web Hosting Master
 
Join Date: Dec 2002
Posts: 1,300
Just remember that you cannot stop a determined person this way.

Having the source code to wget, I could probably easily substitue IE 5.5 for the user-agent string. This would deter most people though.

Also there is never any need to put php files in your list of non-allowed materials, since they cannot download your source (interpreted by the server first).

__________________
"The only difference between a poor person and a rich person is what they do in their spare time."
"If youth is wasted on the young, then retirement is wasted on the old"

  #6  
Old
Web Hosting Master
 
Join Date: Nov 2001
Posts: 551
You don't need the source code... it is built in:

From the wget manual

Quote:
`-U agent-string'
`--user-agent=agent-string'
Identify as agent-string to the HTTP server. The HTTP protocol allows the clients to identify themselves using a User-Agent header field. This enables distinguishing the WWW software, usually for statistical purposes or for tracing of protocol violations. Wget normally identifies as `Wget/version', version being the current version number of Wget. However, some sites have been known to impose the policy of tailoring the output according to the User-Agent-supplied information. While conceptually this is not such a bad idea, it has been abused by servers denying information to clients other than Mozilla or MS IE. In these cases it may be useful to "fake" the user-agent with this option. In the following example Wget masquerades as Mozilla 4.03 running on Solaris.

wget -U "Mozilla/4.03 [en] (X11; I; SunOS 5.5.1 sun4u)"

Use of this option is discouraged, unless you really know what you are doing.

__________________
--

  #7  
Old
Web Hosting Master
 
Join Date: Dec 2002
Posts: 1,300
Touche

I suppose I could have taken 2 second to read the man pages, heh.

__________________
"The only difference between a poor person and a rich person is what they do in their spare time."
"If youth is wasted on the young, then retirement is wasted on the old"

  #8  
Old
New Member
 
Join Date: Mar 2012
Posts: 2
I am using this

But my Bandwidth Limit Exceeded.
my Internal server Error page hits 2,028,069 and use 2.79 GB
Wget hits 360,006
Googlebot hits 57,130 uses 82.91 GB Bandwidth.

what can i do?

Reply

Related posts from TheWhir.com
Title Type Date Posted
Waxman Offers Hybrid Plan to Keep Broadband Providers Happy While Protecting Net Neutrality Web Hosting News 2014-10-06 14:31:53
Washington Mulls Visa Restrictions to Prevent Chinese Nationals from Attending Popular US Hacking Conferences Web Hosting News 2014-05-27 11:15:49
Google, Microsoft Create Algorithm to Block Child Abuse Search Terms Web Hosting News 2013-11-18 14:59:58
Joomla Adds New Encryption Features, Streamlines Interface with Joomla 3.2 Release Web Hosting News 2013-11-06 15:22:58
OpenDNS Releases Umbrella Web Security Platform with Predictive Detection Web Hosting News 2013-08-12 11:25:29


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
WHT Host Brief Email:

We respect your privacy. We will never sell, rent, or give away your address to any outside party, ever.

Advertisement:
Web Hosting News:
WHT Membership
WHT Membership



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?