Web Hosting Talk : Web Hosting Main Forums : Web Hosting : Scalable Solutions?

[Hiro_Nakumora]

We will be launching our website/application (LAMP) this May and the web developers are currently looking at our server requirements and costs.

Our developers have suggested purchasing a server (estimated at $10,000), with searches estimated to take 5+ seconds depending upon the load and processing power. I would like to reduce search times as much as possible and also believe we will outgrow a single server quickly as we are a travel site and:
1) Our database is going to be huge (e.g. 30,000 products to search through).
2) Users can perform complex searches (duration, price, location, features etc)
3) We convert uploaded videos to flash and resize photos etc

As such, I am leaning towards renting a dedicated server for the time being so that we can move to a more easily scalable solution when needed. I have been reading all weekend and have checked out multiple services, however my lack of technical knowledge lets me down a bit.

Are there any services out there which take care of the scaling (processing power, bandwidth and storage) on the back without our needing to do anything differently to what we would do if we were to use a single dedicated server?

I believe Mosso do this, however their "per request" metric is silly and I have read reports of their servers being slow at times which is no good as we need searches and video conversions to be processes as fast as possible.

Lastly, 80% of our users (to begin with) will be based in Australia, and upon searches being performed we need to request data from Aussie servers. As such, 200ms either way adds 800ms to each search (Aus user->our US server -> Aus servers -> our US server -> Aus user), so any solutions with minimal latency between the server and Australia would be preferable. Unfortunately I do not believe there are any solutions available here in Australia.

I am a bit out of my depth, but want to make sure we go with the best solution with improved response times and scalability being catered for, so thanks heaps in advance for any help/advice.

[cristibighea]

There are few really scalable solutions out there, such as GoGrid, Amazon's EC2, but they have limitations and are not fully matured yet, so they aren't really good for mission critical applications.
You could either start off with one high-end server or, get a couple of servers and turn it into a cluster so you can withstand a high amount of searches.
Also, why do you actually need to query Aus servers from US? Is there no way to cache the data on your US servers to minimize the amount of requests?

[Hiro_Nakumora]

I see. Maybe a dedicated server and then a dedicated cluster is the way to go for now then (in which case we would go with an Australia based dedicated server).

Initially we planned on caching the data and doing an update nightly. Unfortunately some of the APIs we are connecting to are just not configured in a way that makes that easy and it was taking far to long to grab, translate and store some of the data in our database (e.g not providing rates per night based on duration, but rather the total price for a stay from date to date).

As such, we now shoot off the stay duration and they return the price and availability for the duration. Bit of a pain but we have to work with whats available. The benefit of this though is that we provide users with the actual price/availability in the search results, not what was cached the night before (we were going to grab the data again when they clicked the view the product in more detail).

[astutiumRob]

the obvious thing to do would be to lose the US server entirely from the equation, and run the service close to the data/audience !

[Hiro_Nakumora]

Yeah, when using a dedicated server, it will most definitely be based in Australia.

The problem is that there are no cloud/grid type solutions here in Australia, so clustering ourselves seems to be our only option. Also servers/bandwidth are far more expensive here in Australia.

I guess that we could do the file converting (media) with a US based service and store this media in the US, while keeping the rest of the site/database in Australia with a clustering type setup to spread the load over multiple machines to improve performance.

We plan to expand globally within a year or so though (US, EU, NZ, CA), so it would be good to know what solutions are available for when this time comes, however the marketplace will probably have evolved by then and our situation will be more complex (possibly with servers in multiple locations).

[Henrik]

Originally Posted by cristibighea

There are few really scalable solutions out there, such as GoGrid, Amazon's EC2, but they have limitations and are not fully matured yet, so they aren't really good for mission critical applications.
You could either start off with one high-end server or, get a couple of servers and turn it into a cluster so you can withstand a high amount of searches.
Also, why do you actually need to query Aus servers from US? Is there no way to cache the data on your US servers to minimize the amount of requests?

I second this. You'll be able to scale your budget with your system resource needs going with a service like Amazon (sic).

There are of course other issues like data integrity etc etc, depending on which line of business you're in. But, these "in the cloud"-services can actually be a good alternative for you.

Let us know what you went for and how it's going?