I am having a very difficult time deciding which route to take to host my next project on - mainly because anticipating what is required is often the hardest part. I'll start by describing the site and what I am looking to achieve. The site is a custom web application built on Drupal (Version 5.7, MySQL 5, PHP 5). The site must parse and update anywhere from 500-10000 RSS feeds and remote csv files hourly, if not more frequently. The number will start off small and expand based on user needs (but will never exceed far past that maximum). Minimal time and performance benchmarking has been done but what can't be simulated is having hundreds/thousands of users (perhaps) on the site at once while this is going on. My experience with custom MySQL applications is limited, and I don't really know what it is capable of, or what it would take to strain it, etc.
My budget is limited and I began looking into VPS solutions because I hosted my last Drupal site across two VPS's (one for MySQL, and one for everything else). The performance was amazing (digg effect did nothing) but the site didn't require much at all. Also, configuring and securing the VPS's was a nightmare and I hope to never have to do that again. So I look for a managed / configured solution. I also looked at all the new "grid" and "cloud" services but they either seemed too complex, too expensive, or just simply dressed up shared hosting. So it's hard to tell...
Scalability, performance, and basically not having to worry about configuration and security are my main interests (whose aren't...). I want to basically never have to worry about the server being powerful enough to run very smoothly (grow as needed). So with all of this in mind, I don't know which route I should take, whether it be VPS or not, multiple VPS, a machine just for the updating, single managed, grid, etc, etc etc.
I'm sure I forgot something and will be posting again soon..
Thank you very much for anyone who either took the time to read this or to who responds..