Anyone here dabbled into fully redundant cluster based hosting? I'd love to hear opinions, software suggestions, etc. Basically when i say fully redundant, i mean 1. no major issues with the clustering, 2. i can walk over to any server with a baseball bat and there wont be more then 30 seconds of downtime, and 3. new servers can be added at any time and start sharing the load immediatly.

Anyone know of what software is used to do this, if its moderatly priced, or rediculously priced (it'd be annoying to make a cluster if you're paying $200/mo per server for the software). I know hostonfly uses a cluster based configuration, any others that people know of, and any commercial software to do so?

Suggestions are welcome :D, im more curious then serious at this point, but its been bugging my mind awhile now :).

noone has insight on this? Need this be moved to the dedicated-server forum? :eek: thought i woulda at least gotten *some* response!

They've got to be the way of the future, Myles. I find the whole area fascinating but way out of reach for what our business provides etc. Are you playing around with this concept as part of your offerring? :)

Originally posted by Aussie Bob
They've got to be the way of the future, Myles. I find the whole area fascinating but way out of reach for what our business provides etc. Are you playing around with this concept as part of your offerring? :)

I indeed hope to be, the concept is simply amazing to me, and i believe provided with the proper configuration, Priority Colocation would have everything needed to set the rest up seeing as we manage our own servers/space directly, etc. Managing multiple CPanel servers for resellers is a tad annoying, the idea of being able to simply plug another server into the cluster whenever the load got higher then desired just floats my boat! :D

It's a nice idea, but it runs into problems with state replication.

If you can restrict your application space to database-backed transactions, it's possible, but even then it is only practical if you have 10+ systems.

This is one of the markets for diskless server blades.

Originally posted by porcupine
Managing multiple CPanel servers for resellers is a tad annoying, the idea of being able to simply plug another server into the cluster whenever the load got higher then desired just floats my boat! :D
It does more than rocks my boat mate!!!! :D:agree:

We do local clustering, local LB and GLB for some customers. All of them are W2K systems and we run Foundry ServerIrons on the hardware side. Also, all our GLB equipment is redunant (no sense doing LB/Clustering if you have a single point of failure).

We don't charge tens of thousands to do it, but it's not dirt cheap.

Not sure if that helps!

Mike

Hey Mike,

Any idea on unix/linux clustering? And does your setup *require* the Foundry ServerIrons hardware, or is it just what you recommend?

Hey porcupine,

HA (High Availability) is a very involved topic so I will try and keep it as concise and precise as I can (we all know that won't happen ;) ).


Depending on the OS you are using you have a number of different solutions, both hardware and software. Personally, I like going the route of a hardware load-balancer as the prices have dropped drastically since the dot-com fallout (doesn't seem anybody wants to pick them up anymore).


For strict load-balancing you technically do not need a SAN (Storage Area Network) for clustering it is a must (share same redundant data source). If you plan on getting a SAN, wait for the iSCSI line of products to come out (I believe the first version's standards were recently ratified), will save you bundles of money and avoid a number of headaches (my 2 cents).

If you are running Linux here are two pretty good links to get some information:

Red Hat - Piranha (http://www.redhat.com/software/advancedserver/technical/piranha.html)
Holy Grail of Linux HA (http://www.linux-ha.org/)


If you are running Windows 2K or .Net you need to purchase the Advanced Server version in order to load-balance/cluster (if you want to use the OS's built in HA).

If you want to go 3rd party there are companies like Veritas (http://www.veritas.com/) and Legato (http://www.legato.com/) that we have used with pretty good results.

Again, my personal favorite is the hardware load balancer (almost always doesn't matter what OS you are using, merely does port based availability and SNMP health checks) from companies like F5 (http://www.f5.com) , Foundry Networks (http://www.foundrynetworks.com) , and Radware (http://www.radware.com) .

If you want HA on the cheap look on ebay for some of HP's (rebadged Intel) products (specifically the 7000 and 8000 series).

If you are looking into GCR (Global Content Redirection), which it doesn't sound like you are, try out the 9000 series, very cheap for what it does.

We personally use a mix of hardware products and an in house GCR product that we developed for our ASP/MSP clients.

Hopefully, some of this info helps out in some way. I didn't want to get too specific, just wanted to help guide you down the right path as you said you were just curios.

Have fun!

Humm, yes, this information is interesting, im just wondering if it can be hacked up to make a redundant CPanel/WHM cluster, sounds like it may be possible...

Just curious, if you've got a SAN, setup, wouldn't that give you a single point of failure (even if it is running RAID)? Also, wouldn't this be a potential mega-bottleneck, if you've got 10+ servers hammering for data off it?

Look into Linux Virtual Server & Heartbeat & Mon



Two Routers (one master plus one slave) Servers running:

LVS to do the actual load balancing

Heartbeat manage the master/slave servers (i.e. when the master goes down the slave will automatically take over)

Mon
To monitor web/db/etc servers and automatically add and remove them from the lvs pool. (i.e. when a server goes down mon is responsible for removing it from lvs)


add in as many servers running the application you wish to load balance -- these can be added and removed at will...



The company I work for, Sparklit [Sparklit.com] has been running a two similar setups (one in vancouver, one in seattle) with about 25 servers. We have had zero downtime since implementing this setup.

Look into it...

Originally posted by funkee
Look into Linux Virtual Server & Heartbeat & Mon

That's the right solution to the wrong problem. How do you add systems to the cluster dynamically? You have to copy all the user files across -- how long does that take?. What happens when someone is using a cgi script which modifies files on disk? You have to distribute those changes to all the other systems -- what happens if the system crashes before it has finished doing that?. What happens when two cgi scripts try to modify the same file at the same time? And whatever do you do when there are too many files for each system to have a copy of everything?

Set up servers to a fully redundant array (scsi/fibre).
Set up a bunch of servers that all boot off drives in the array.
Those servers will also be reading data from the array (fibre will result in better performance).
Get a load balancer.

You can seriously build this for under $5,000 (Not including the "bunch of servers") - depending on how much disk space you need ($5000 should get you a load balancer and around 200gb of fibre channel RAID5)

I'm actually becoming quite experienced in the fibre array world, very cool stuff.

We're eperimenting with using scripts on two machines in the data centre to monitor each others services, if one machine goes down the other will assume its IP address.

At the same time, user mail and documents are copied using rsync (We need to find a way of doing this everytime a new file is ftp'd or new mail arrives... at the moment it's done nightly - any ideas ppl?)

As for MySQL, we have a master-slave, slave-master setup, but the backup server only accepts connections if the master is down (can't have INSERT/UPDATE queries hitting both servers at the same time)

Its something we're still experimenting with, the goal is to provide protection against hardware failure.

i dont know much about clustering but - i know there is a way to "mirror" serveral hard drives together and make them all look like 1 hard drive. so if a change is made on 1 of them, they all get changed.

Originally posted by clockwork
Set up servers to a fully redundant array (scsi/fibre).
Set up a bunch of servers that all boot off drives in the array.
Those servers will also be reading data from the array (fibre will result in better performance).
Get a load balancer.

You can seriously build this for under $5,000 (Not including the "bunch of servers") - depending on how much disk space you need ($5000 should get you a load balancer and around 200gb of fibre channel RAID5)

I'm actually becoming quite experienced in the fibre array world, very cool stuff.

Doesen't that make a single point of failure? Unless of course that is you had 2 servers with fiberwire raid 5 scsi arrays, and used redhat to raid 1 over the two servers (is that even possible, crazy idea)....

BTW, all the servers load all the information off the remote arrays, even boot? Do the servers have any drives what-so-ever? And doesen't this become a bottleneck really fast? I've never had a server boot off a drive that wasn't local, interesting concept to say the least :).

I finally found a thread about this....woohoo! :D

I, too, agree that this stuff is the shiznit. Blades....clusters....muahaha.

I would love to find out how to bridge systems together to use all of their processing power for a certain task. I've read all kinds of stuff about Beowolf (sp?) and other types of configuration stuff you can do, but never got anything in English (it was in English....just not my English ;) ).

I thought that having hot-swap harddrives and power supplies was an awesome idea....I can't imagine being able to walk over and plug in another server to the configuration and having it get to work immediately. *drools*

:D

Blades? Ick! I want to do this with a lot of power for a little money, or our standard setup with less management/licensing costs :D, i dont want to use those unreasonably expensive blade systems.

Hot-Swappable equipment is cool, but not really the same as a cluster :D

In order to start unix-based mass hosting cluster system, you have to:
1. setup redundant data storage accessible from any server in rw mode. it's not a problem, you can find a lot hardware and software of solutions on the net. but usually it costs a lot.
2. setup load balancing for webservers farm. for this task you can find a lot of good solutions (even free).
3. setup similar configuration for all webservers. if you plan to use Apache, it's a problem because you can't store vhosts config in usual apache structure - it takes a lot of RAM (more than 1000 virtual hosts is a big problem).
4. setup replicated SQL server(s). it's not a problem....


We have successfully solved all problems using our own software.
Now we are happy with our worldwide cluster-based hosting system.:cool:

you sell that software? :D

Originally posted by HostOnFly
In order to start unix-based mass hosting cluster system, you have to:
1. setup redundant data storage accessible from any server in rw mode. it's not a problem, you can find a lot hardware and software of solutions on the net. but usually it costs a lot.
2. setup load balancing for webservers farm. for this task you can find a lot of good solutions (even free).
3. setup similar configuration for all webservers. if you plan to use Apache, it's a problem because you can't store vhosts config in usual apache structure - it takes a lot of RAM (more than 1000 virtual hosts is a big problem).
4. setup replicated SQL server(s). it's not a problem....

That's a good start; how about

5. Provide transparent distributed file locking with fcntl semantics,
6. Provide coherant memory-mapped files, and
7. Allow process migration in case of server failure?

(A lack of 5 & 6 will cause some software to break; a lack of 7 will simply reduce your system's availability.)

We don't sell our cluster-based hosting software. But we offer access to resources of our global cluster system (kind of reselling).
Also we are open for partnership with colo companies.

Interested in a toronto cluster? :D

Originally posted by cperciva


That's a good start; how about

5. Provide transparent distributed file locking with fcntl semantics,
6. Provide coherant memory-mapped files, and
7. Allow process migration in case of server failure?

(A lack of 5 & 6 will cause some software to break; a lack of 7 will simply reduce your system's availability.)


you can avoid or implement 5 & 6 depends on your exact needs, it's not too hard.
process migration in case of server failure - it's too hard to implement and absolutely not required for 99,99999% of applications (network latency & packet loss on the user end will cause much more incidents).

>Interested in a toronto cluster?
Why not? ;)

we'll have to discuss that sometime :)

nobody is gonna mention beowulf?

Originally posted by funkee
Look into Linux Virtual Server & Heartbeat & Mon



Two Routers (one master plus one slave) Servers running:

LVS to do the actual load balancing

Heartbeat manage the master/slave servers (i.e. when the master goes down the slave will automatically take over)

Mon
To monitor web/db/etc servers and automatically add and remove them from the lvs pool. (i.e. when a server goes down mon is responsible for removing it from lvs)


add in as many servers running the application you wish to load balance -- these can be added and removed at will...



The company I work for, Sparklit [Sparklit.com] has been running a two similar setups (one in vancouver, one in seattle) with about 25 servers. We have had zero downtime since implementing this setup.

Look into it...

Yeah I've used LVS and it rocks, www.rxlist.com is using a setup I put together, 4 web servers, 1 file server, 1 load balancer (yes ithat's not exactly redundant but this was mainly for LOAD balancing than the redundancy).

To avoid problems with sessions I just set the load balancer to keep sessions tied to a particular server for an hour (or was it 12 hours) and then I wrote a handful of scripts that checks each server for availability each minute, if a server fails it gets taken outta the loop, if it comes back later it goes back in the loop.

The servers boot their own individual OS's and then run apache over the network.

If I could find a control panel that could scale to a couple thousand web sites without dying a horrible death I'd probably spring for a nice netapp and put a dozen or so P4 listeners in front of it, with a pair, or even a trio, of load balancers, using a diskless listener scenario so OS updates, etc and maybe a load balanced mysql setup in the back happen on only one server. Plesk for damn sure can't do this, Ensim "might" it seems a little more graceful with a full server of web sites.

Another idea I had was have one web server running the control panel, and then have the listeners not running the CP, have the load balancer send all CP requests to that server and then the other systems would only have to run apache and sendmail.

It'd be nice from an uptime standpoint.