hey guys just wanted to get your thoughts on what do you think of hosts that offer fail over or load balancing where a web site is on more than a single server...and also if any of the hosts here offer such service? any kind thoughts if this concept can do any good for the host? many thanks in advance for sharing your thoughts!:D

Well it can prevent nagging customers calling every 30 seconds to report to you that they can't access their anime site at 4AM. :D It's also good to keep MySQL hogs away. Are you looking for a host that offers this or ... ?

feather,

mirroring failover is *very very* hard, especially for webservers. it is doable with shared scsi disks ($5k+ in hw costs) or via software. the software solution implementation is very very hard (we did some work on that for a client) and unless the host that offers it has kernel developers on staff, i wouldnt trust their ability to implement it. it is *not* going to be cheap.

load balancing is not possible for mysql and hard for apache (transaction state). it is more practical to split clients between servers.

if you have specific questions, feel free to drop me a pm.

cheers,
paul

well u did seem to make it look very hard i was planning to do this myself for my new hosting company which i would be starting out very soon hopefully and i have ppl who can pull this off very easily as they have done in the past and yesh! i agree with you its very expensive but sumthin i just wanted to do *alot* more than just a fail over and load balancing so i m hoping everything will work out pretti soon!

edit: :D

hostopia.com is a well-known company that uses an infrastructure like this.

A setup like you describe is relatively easy with just static content; when you add in dynamic content (PHP, CGI, etc.) it gets messy, fast. You're looking at a pretty significant investment in capital to do it right.

All I can say is... good luck. :)

feather,

if you have people that found a way to do this, i want to talk to them (and so do a lot of vcs).

there is no possible way you can create:
1. an active/active cluster for applications that are not aware of synchronization between several instances of itself (eg mysql)
2. an active/active cluster for applications that require a transaction to have a node affinity
3. a failover cluster for applications that do not serialize state to disk

(there are points that i missed, but its 2 am and those are the most obvious ones).

your money is better spent on better connectivy and more reliable (and redundant) hardware. with a well managed server, those are the points of failure.

alternatively, talk to ibm - they have been working on single image systems for 10 years now (they still dont have it done though =).

paul

and also if any of the hosts here offer such service? Hi Feather, AWH has invested heavily in infrastructure to achieve our load-balanced, server farm environment. (If I'm allowed to link to it, more info on our set up is here (http://forums.alwayswebhosting.com/announcement.php?s=&forumid=18))

Other than some initial bumps, its working great! Even with our setup, our prices are still rather low-ball. Its going to take some volume for you to recoup your costs (it did for us at least). :D

I agree with Paul. We tried to implement this for our own site, to prevent future problems should there be any. I was working with 2 of my close friends (seasoned kernel developers at berkeley BTW) with little success. We couln't afford the downtime so we put the project on the backburner for now.

Heck if you meet someone(s) who can handle this with confidence, I'd definitely want to get in touch with them....

Best
Tim

Originally posted by rusko
feather,

if you have people that found a way to do this, i want to talk to them (and so do a lot of vcs).

there is no possible way you can create:
1. an active/active cluster for applications that are not aware of synchronization between several instances of itself (eg mysql)
2. an active/active cluster for applications that require a transaction to have a node affinity
3. a failover cluster for applications that do not serialize state to disk

(there are points that i missed, but its 2 am and those are the most obvious ones).

your money is better spent on better connectivy and more reliable (and redundant) hardware. with a well managed server, those are the points of failure.

alternatively, talk to ibm - they have been working on single image systems for 10 years now (they still dont have it done though =).

paul

alwaysweb,

interesting. i have a few questions:

1. are you load balancing mysql?
2. how do you ensure session affinity?
3. are you using shared storage for /home?

anyuone can split everything up into a mysql, web and mail server - that is not load balancing.

paul

I think the best way to do this, would be to setup two servers:

serv01.you
serv01repl.you

and do replication between the two every hour or so... Maybe 2 hours, and always backup serv01repl.you to a second drive... That way, if serv01 goes down, you can easily switch to serv01repl

I dunno though. I don't know much about load balancing... I didn't even think about it until now...

Hi Rusko,

1. we aren't load balancing MySQL. The headaches/complexity of doing just don't seem to be worth it.

2/3. Exactly right, we have a shared /home. As well, there is a shared /home/tmp for use by session variables, etc. (If I missed your affinity question, please explain further what you mean by that term)

Our web serving is load-balanced -- We have a Netscaler 9000 series Switch (http://www.netscaler.com/product/sa_gateway_datasheet.html) doing our 'intelligent' load-balancing -- And web/DNS traffic is distributed across multiple 'www' servers, running the same httpd.conf, and accessing the data across the /home store. This distributes the CPU/apache load, and also ensures that if apache fails on one server or if the server goes down, the other servers can sustain the web load with no degradation in service to the end user.


I'm trying to remember the name of the web hosting company I ran into in the past that has 100% guarantees on everything. I believe they were balancing everthing, including mysql and user data, with hot failover, etc. I remember it being ridiculously expensive, if I'm not mistaken. If I could only remember the name...

FreeBSD is working on a shared environment where you can have several servers that all work and act like one. Probably won't be done for a couple years though, if that :) I'm certainly not holding my breath while waiting.

leif: what you are referring to is a single image system. yes, i dont think fbsd will have it done any time soon, id say ibm is a bit closer.

alwaysweb: it is not possible to run an active/active mysql setup, not just 'hard'. im assuming you use nfs for the shared mountpoints? if so, how do you deal with applications that dont to posix fcntl locking (and do, say, flock instead)?

paul

Rusko, not possible, you're probably right. If you want to PM me, I can tell you more about our use of nfs... ;)

Originally posted by rusko
feather,

mirroring failover is *very very* hard, especially for webservers. it is doable with shared scsi disks ($5k+ in hw costs) or via software. the software solution implementation is very very hard (we did some work on that for a client) and unless the host that offers it has kernel developers on staff, i wouldnt trust their ability to implement it. it is *not* going to be cheap.

load balancing is not possible for mysql and hard for apache (transaction state). it is more practical to split clients between servers.

if you have specific questions, feel free to drop me a pm.

cheers,
paul

1) Problem #1 solved: Network attached storage. Netapps go for cheap these days.

2) Don't load balance mysql then. Use replication and toss a beefy server at it. Also, it's not very hard to keep state with modern load balancers (might need to enable route caching depending on your architecture). If you're even seriously considering this type of move, then surely you can lay out the capital to buy a cisco CSS, alteon, or foundry serveriron on the used market. All very fine devices and cheap if purchased used.

We used to have this setup but found that not many people wanted it so we stopped.

We used FreeBSD + FreeVrrpd (http://www.bsdshell.net/hut_fvrrpd.htm ) and two-way MySQL replication.

FreeVrrpd let us have one IP and set a backup server for it which would take over that IP if the main server became unavailable.

We setup circular replication between the MySQL servers and gave users the IP being controlled by FreeVrrpd.

Our setup didn't involve copying user files but we had plans for this using FAM (http://oss.sgi.com/projects/fam/) and rsync but we never got around to trying this.

Most of Googles servers were (as of about 6 months ago, probably hasn't changed) FreeBSD/MySQL, maybe someone should ask them how they do it :)

leif,

their databases are not coherent. try connecting to different servers adn searching for the same thing - you are going to get different results. while complete coherency is not a requirement for them, it is a must for your average user who, say, runs a forum.

paul

Originally posted by Leif
Most of Googles servers were (as of about 6 months ago, probably hasn't changed) FreeBSD/MySQLActually Google runs on Linux (just read the site or the press).

Originally posted by Leif
maybe someone should ask them how they do it :) As much as many people would like, any detailed (i.e. useful) information on Google's operations is confidential.

There's 2 things about load balancing that will determine the level of complexity / money you need to spend on it.

First off, if you have an ultra critical site that you are trying to load balance and need up to date mysql data on it, then you will need to spend a ton of cash.

If you just want to be able to claim 100% uptime with load balanced servers (yep... it would be a good marketing tactic I would think) then you can easily load balance and just not worry about mysql data. (Make sure you write this fact into the TOS though).

I would think for most shared hosting accounts, the latter would be acceptable. If a site is up and serving day-old data out of the DB, that's better than not serving any data at all! :D

Has anyone used CSS with JRun on app servers
Any comments on performace hicups or JRun cache issues?

-asif


Originally posted by manmythlgnd
1) Problem #1 solved: Network attached storage. Netapps go for cheap these days.

2) Don't load balance mysql then. Use replication and toss a beefy server at it. Also, it's not very hard to keep state with modern load balancers (might need to enable route caching depending on your architecture). If you're even seriously considering this type of move, then surely you can lay out the capital to buy a cisco CSS, alteon, or foundry serveriron on the used market. All very fine devices and cheap if purchased used.

It isn't as expensive as some may say. Just not supported via any control-panel system by any means as i have yet to see one thats "shared" aware.

I run many Oracle Portal 9iAS systems with 2-16 servers all behind Cisco local directors. Portal is built to work in a tiered structure with RDBMS stored content and even the database is clusterable via Real Application Clusters.

You need a solution that is supported, designed to be scalable and something you understand. If you have to ask here, i'm afraid clustering/failover is a bit out of your understanding heh

the point: unless the applications are aware of an active-active setup or you control the applications, cant do it.

Originally posted by turboweb

I run many Oracle Portal 9iAS systems with 2-16 servers all behind Cisco local directors. Portal is built to work in a tiered structure with RDBMS stored content and even the database is clusterable via Real Application Clusters.



I guess you aren't using this setup for hosting (only) :D? Does this setup guarantee you 100% uptime?

And more in general, having 100% server uptime may be a good selling point (was considering it myself too), but with that you will be targeting a niche which isn't big enough to cover the cost. Normal hosting clients will not pay you the higher fees, just because their server is always up compared to a few minutes downtime at other host. Of course downtime is bad, but you only need to use it for mission critical stuff (if you are selling products from your site, just a few minutes downtime might lose you a 200 dollar sale), but 99% of the people you can reach (since you still have to find this special group of customers) won't need it, and the other 1% will already have a solution.
And after all, if you are running a mission-critical site, would you like it if your site is on the same server as the site for the "Annual Hackers Meeting" (for excample). No, you get your own setup, which you can protect yourself, and don't have to allow people on the same server who are not so serious about you are...

Just my 2 (euro) cents!

If you want a cheap version of this you could try "Round Robin" DNS. ..

I had a nice thread about this a while ago, called What is needed for clustering/load balancing (http://www.webhostingtalk.com/showthread.php?s=&threadid=110424), which turned out in a pretty helpfull thread (as always on WHT :D).
You might want to do a search on clustering (http://www.webhostingtalk.com/search.php?s=&action=showresults&searchid=1169403&sortby=lastpost&sortorder=descending) or on Load balancing (http://www.webhostingtalk.com/search.php?s=&action=showresults&searchid=1169405&sortby=lastpost&sortorder=descending), because are some interesing threads on this board.
Success!

Jacco

blikje,

that thread is informative, but it does not answer the original question in this thread. you wanted to split the load between a bunch of servers and the data was written to a single machine that would be a single point of failure. not having to mirror data took out all of the complexity.

in this thread, the starter inquired about failover, which does involve mirroring writable data to a remote machine. as was pointed out, it can be done in an active/passive configuration if you use mysql.

paul

Originally posted by rusko
blikje,

that thread is informative, but it does not answer the original question in this thread. you wanted to split the load between a bunch of servers and the data was written to a single machine that would be a single point of failure. not having to mirror data took out all of the complexity.

in this thread, the starter inquired about failover, which does involve mirroring writable data to a remote machine. as was pointed out, it can be done in an active/passive configuration if you use mysql.

paul

True, you're right :D Let's see what the thread starter has to say after reading this, he hasn't responded yet :D

check out linux virtual server, then you can move all backend (dynamic) to NFS.....

for MySQL you can do something similar with LVS/failover (check out MySQL replication)... probably you will need some raid set-up for redundancy

wow...where did this thread come from after such a long time? anyhow well i might have asked the rong question overall but this is the setup i had in mind..

1. Main server directs traffic to a web server which is faster {checks every 30 seconds to see which web server has faster response time}

2. A box with the control panel and the mail

3. A MySQL Server

4. A web server

5. 2nd web server {both mirror the same web content content}

extra can be added incase needed...all the boxes have raid1 or raid5....

operating system is freebsd and this setup works fine with h-sphere

edit: no i m not a host nor i am offering this plz no need to blast me for this!:D

Originally posted by feather

1. Main server directs traffic to a web server which is faster {checks every 30 seconds to see which web server has faster response time}

that seems counter productive...

well its just an added feature so one server doesnt have to much load and top of that if it happens to be one of the web server goes down it will not direct the traffic toward that server....

you will not be able to do active/active with mysql. period. nfs on latest stable fbsd is broken (no client-side locks). for that matter, it is broken on linux as well. (not actually nfs, lockd/nlm).

active/active with session affinity on a webserver cluster is definitely doable. that would be the only component in the system that could qualify as ha under the config you suggested.

paul

well watever u said i have no clue as i m not the person who is going to do it...last time i checked the setup i had listed above everything seem fine running that way...

edit: :D

feather,

what i meant to say in plain english is that there is no failover (aka high availability) in the setup you described. the 2 web servers are the only instance of load balancing.

had you mentioned that you would be happy with such a setup, most people posting in the thread would have said that you just need a control panel that supports that and would have pointed you to hsphere.

the tough part would be making db servers highly available/load balanced. you do not seem to require that.

paul

Just thought I would toss this into the thread since it came up on /. today:

http://developers.slashdot.org/developers/03/04/30/1331227.shtml?tid=

Summary: make a RAID of databases (as opposed to disks) :)

interesting, i havent seen that one yet. however, this is most likely not all that great - you cant provide real-time coherency without changes to the way mysql serializes write accesses.

paul

yep, just as i thought:


You just have to ensure that all database accesses are performed through JDBC.


this means that this can only be used if everyone you are hosting is using java (jdbc more precisely) to access the databases. yeah right. it is not hard to introduce an external serialization reparse point, its a matter of coding it more than anything else.

what *i* want to see is this approach ported to the mysql client libraries. now *that* would be very useful.

paul

rusko....what i was looking to do was take off load n give a bit more uptime than usual incase of hardware failure...mostly all the host here have everything in one box...so having 2 web servers pushing the same web content certainly wud push the web site out so ppl can see it and it wud be lil faster than usual even if the control panel and mysql server is down... you would still be able to view the site {web content}... 2nd all the boxes would be setup with raid1 or raid5 so incase of hard drive failure a mirroring hard drive can take over... nuthin is ever enuff but certainly this helps to some extent...i am not sure how many ppl you would put on this setup but i dont see a reason y this shudnt be enuff...

the web servers can give abt 4000GIG before they wud crash...
MySQL least can put up with 800 GIG
Control panel and mail shud do fine
and the main server wont have any load so everything shud work fine...

there is always more things to do but i am not looking to do them now...but if you see a problem in this setup or you feel sumthin will get messy plz do let me know as suggestions n help is always welcome! :)

In order to make a solution truly resiliant you ideally want replicated systems on two different networks... I've seen network connections go down far more often than the actual hardware.

^^ that is very true...i had some thinng like this in mind but really doesnt work out as it costs so much u cant really deal with the costs of it...or else who wudnt wanna do it.....thas you can have blend of providers where if one has a problem things work out fine

no, the problem is that you cant ensure coherent real-time replication over a wan.

paul

I don't think this is what most of you are looking for. I just wanted to say, as far as a hardware HA load-balancing solution. I owe many thanks to F5. I'm just saying we have been totally happy with these hardware loadbalancers from F5 for years now.

We recently upgraded some of our old f5 products to the lastest models. Just purchased 2 of these.

http://www.f5.com/f5products/bigip/LB520/

take care

I'm a newb here @ WHT.. take it easy on me.