What Specifications Should I be Seeking for Hosting Web Tools?
I need to purchase a new dedicated server which will serve to power many web tools we offer that scrape content from many different types of websites - APIs, Search Engines, Directories, etc.
The problem is the amount of queries that come through and the time it takes to process. What would your recommendations be for a server that could properly handle many multiple scraping/content-grabbing operations quickly without slowing down the process due to hardware issues?
Well without knowing the actual scripts in detail it's hard to recommend an exact solution, but if they are getting popular now they are most likely not going to stop anytime soon. Since you are constantly scraping content and serving it I will assume that you have many reads and writes going on at the same time, which would lead to a recomendation in a SCSI Raid array (do a search on the forums for tons of info about RAID and great links). I would also go with a SMP system (dual processor) either opteron or Xeon with 2GB of ram. This is just a guess as to what you need though.
The link in my sig shows the tools I'm referring to. Maybe that can help.
Thanks for your input - my developer thought we didn't need something this powerful (Xeon, 2 processors, etc.), but I'm of the opinion that every fraction of a second shaved off a request will amount to a lot of time over thousands of requests.
You would think I would have at least looked at your sig...feels like a monday
After checking out the links in your sig your developer is most likely right. A single xeon processor would probably do, but personally I would still opt for a machine like a mentioned because it will help you out with growing pains
Interesting stuff you're running there, very different from what I've seen before.
Balancing over two machines may do the job, it all depends, is your traffic consistently balanced or does it spike at certain times, if it spikes a dual CPU would benefit you more, whereas if it's consistently busy, two machines would probably be better suited.
█ Dan Kitchen | Technical Director | Razorblue
█ ddi: (+44) (0)1748 900 680 | e: [email protected]
█ UK Intensive Managed Hosting, Clusters and Colocation.
█ HP Servers, Cisco/Juniper Powered BGP Network (AS15692).