I dont have much knowledge in this area and was wondering if you could answer a few of my questions or set me in the right direction to research it. Ive read a few documents on the beowulf cluster, but im not sure if that is the propper solution for the application im looking to use. I need a cluster/load balanced solution for low latency applications. Some of the questions I need answered are:

1. Can I purchase theese pre-setup. If so where?

2. What knowledge is needed to maintain them?

3. What am I looking at price wise for 3-4 nodes and the load balancing hardware. Also how scalable are they?

Thank you for the help.

check out LVS -

http://www.linuxvirtualserver.org


http://www.linuxvirtualserver.org/Documents.html


http://www.linuxvirtualserver.org/HighAvailability.html

Originally posted by TheWalrus
...I need a cluster/load balanced solution for low latency applications.
....

Can you clarify "low latency" definition?

Peter Kinev.

I think low latency was a most likely not what i was looking for, since Its not compliation. Im looking to do game server hosting, and the only way I see that its can be done profitably in my eyes is with a cluster solution.

Also, I would like to migrate our web hosting services to such a setup.

So, you are looking for a clustering solution to handle expected level of load first and availability second.

Thus your intuitive approach to split load and application roles among few physical nodes is fundamentally correct approach. In a very simple physical model you will have a database layer (back end) on one node and application layer (server + application itself) on the second server. Now, I am not an expert in gaming, you can get a next level of granularity if in your expectation one or both layers need to have some additional improvement. But I would start with two nodes simple model first and analyze real load before to make an extra effort. However, your configuration and installation of two-nodes cluster must be done with the expectation that you need an additional load split.

At this time you have to define a configuration of two nodes (memory size, CPU, etc).

Regards,
Peter Kinev.

you sure your game server/daemon will support cluster configuration/set-up? As load will be balanced between different nodes (low-price servers) an application (in your case, a game server daemon) must support it as well....

Well, would I be able to have the application migrate to the node with the least CPU usage? Not actualy break the process down, but rather migrate the entire process to the least utilized node?

As I said, I am not a game expert. But, my consideration in system selection and configuration would be focused along these lines.
- separate database and application to run on different nodes
- application node(s) should be capable to process 32bits application in SMP fashion
- application better be capable to run on different CPUs (bounded)
- I may choose to have more than one application node with a load balancing between two of them
- I will wait for introduction of SMP/clustered MySQL in the end of this year (if I can) http://www.infoworld.com/article/04/04/14/HNmysqlcluster_1.html

Peter Kinev

Thanks for the help peter. At this point I think my best bet is to A. hire a consultant or be get some books and learn this stuff myself.

I am glad you see it this way - it's a very good assessment. The true clustering solution is not a simple issue.

Peter.

Ive quickly learned that. The good news is im not looking to do it at a bargain... The bad news is finding someone I know that they know what they are doing.

Actually a high availability cluster is what I think im looking for, ive yet to find a rock solid definition of what exactly it can handle tho. Also I need a solution in the windows enviorment as well as the NIX enviorment.

http://www.linuxvirtualserver.org/whatis.html that as pasted by an earlier poster is what looks like a viable solution for myself.

Originally posted by TheWalrus
Actually a high availability cluster is what I think im looking for, ive yet to find a rock solid definition of what exactly it can handle tho. Also I need a solution in the windows enviorment as well as the NIX enviorment.
I am not sure about it - from what I see, you need to solve performance issue first and availability second. Sometimes they may have two conflicting resolutions.

To mix two clustering environments (windows and Linux) in one place and for the same reason is not a good idea. You must make your decision before you will start implementation. IMHO, Linux base cluster is much easier, cheaper and more reliable.

Peter Kinev

When I said linux and windows clustering, I meant 2 seperate clusters.

Originally posted by TheWalrus
When I said linux and windows clustering, I meant 2 seperate clusters.
Yeh, I understood. But, why do you want it?

Peter.

Certain applications only run under windows unfortunately. If I had it my way I would run it completely under *nix, as I agree that it is a superior solution in every aspect.

OK. One thing you can be certain - it's to select Linux for database. IMHO, I like PostgreSQL much more than MySQL. But, as I said in the end of the year they will have a solution (we need to see) for MySQL clustering. Your development effort if you will start now may very well drag you till MySQL availability time. So, you need to start working on an application cluster configuration.

Peter Kinev.

Originally posted by TheWalrus
Well, would I be able to have the application migrate to the node with the least CPU usage? Not actualy break the process down, but rather migrate the entire process to the least utilized node?
Not really. To migrate an existing process, you would need an SSI cluster (i.e. Mosix), since you need to maintain the process state. (Beowulf and load-balancing systems are 'job queue' oriented, so they move to another box by relaunching the job -- any resume abilities would need to be application-level.) Most game server processes are so large (100MB or so) that SSI load-leveling will not move an existing process until it starts swapping. Not that I blame it -- even over dedicated gig-E, that's no small hiccup.

Load balanced / fault tolerant web hosting is more viable, but you will still want expert assistance before dropping that kind of coin.

You will get tons of suggestions and advices. Not all of them (from my experience on this forum actually almost all of them) will be qualified as "recommended". The one thing seems to me is indisputable - you will need an expert help.

Peter Kinev.

Yes, I agree that I will need expert help =D. The question is where to get it at this point. Can anyone point me in the direction of someone that can tell me exactly what type of cluster I would need, and someone that can set it up and maintain it? Im willing to make the investment (in equipment and consultation as long as I know it is worth it).