Hello anyone who is reading this thread, do you have experience with coding web crawler bots?
I am looking for a bot which will crawl certain websites and then save the content into a database. (MYSQL). A search query will then be used to search for certain keywords/information in the database.
Replying because it wont let me PM you (I thought I needed 5 posts to use the PM system? apparently not)
Anyway, I write webcrawling bots all the time. I've created bots to download all the images on a particular site, to pull the artist and track information for music albums from amazon.com, to grab all the email addresses from a particular site, etc. I can do it in PHP, Perl and/or Java, whichever you would prefer. If you get this and are interested, please leave a reply
Basically someone will come on my website type in a search like 'sociology' and it will search the database with all the files with sociology on it. The searches will then be displayed and then the user can click on the search and it will take them to the website with the article/etc on.
Sorry if this sounds confusing, but i am in abit of a rush.
I got your PM (apparently mine is working fine now), anyway unfortunately I could not reply: "Aaron Luckie has exceeded their stored private messages quota and can not accept further messages until they clear some space."
So I sent my reply as an email instead. Hopefully you received it