Web Hosting Talk : Other Forums : Web Hosting Lounge : Free leeching tools?

[oonth]

I basically have a directory listing of about 20,000 documents which I need to download. I could whip out a VB app to do it but am wondering if there is any stable free tool already available?

Use of it is completely for educational puposes as I have to download the 20K documents, then run a search to find about 100 specific documents that I need from that 20K and print it for our biology department at school.

[oonth]

That was easier than I thought Thanks to Visual Basic and ActiveX

Next question will be I guess once I have these 20,000 documents, what would be the best way to go about searching these? Should I import these and index into MS SQL Server? Each document is about 20KB in size.

[EthicalEpi]

Looks like you've already got the documents downloaded, but for future reference there's an application with a nice front end and filtering features called web reaper that you might find useful if you need to do that again. It's free to download.

Are they word documents? You may be able to use the windows search feature and just use the 'containing text' box then point it at the folder that contains your documents. Not sure how well that would work, but it's worth a try.

[hostito]

what about "wget -R http://www.website.name/folder_of_junk/"?