We have a big issue with people using bots to grab our dynamic information and when one of our customers releases new info the site gets flooded with these scrapers downloading all the new content. We've tried everything from banning IPs but this bites us because they use big IP pools and then we ban real users. We tried setting cookies on the clients computer to make sure they're legit but they started using cookies in their scripts. Now, they are using proxies so the IP is not the same so it makes it very hard to even track if it's the same person but you can tell because the searches are in secuence and very fast. Any ideas on how to get around this?
He have customers(airlines) that have dynamic fares so customers can search for flights. People write scripts(bots) to search every day, every time, and get all the flights so they can sell/use them with something else. This uses a lot of resources and needs to be stopped.
There is a good article in this months issue of information security about screen scrappers and how to protect your site. Google Information Security Magazine and you should ficnd thel ink off of techtarget.com not sure if you can view the article on line but i will try to get the info from my copy.