Any one providing high availability hosting. For example hosting where you have a F5 load balancer or even data base hosting with high availability, for example My SQL with www.high-availability.com product ?. It looks like big name hosting companies (Rackspace etc) offer these on a dedicated setup but not on a shared environment. I use www.maximumasp.com and their VPS-LB product but they don't provide the same on database level.

Don't you think their is a market for it ?

Jay

Originally posted by jay23
Any one providing high availability hosting. For example hosting where you have a F5 load balancer or even data base hosting with high availability, for example My SQL with www.high-availability.com product ?. It looks like big name hosting companies (Rackspace etc) offer these on a dedicated setup but not on a shared environment. I use www.maximumasp.com and their VPS-LB product but they don't provide the same on database level.

Don't you think their is a market for it ?

Jay

The market is probably there, but the setups with high availability and/or load balancing are often so customized to fit the needs of a specific customer, that it is hard to offer shared solutions.

We do dedicated high availability / load balanced solutions ourself, but we haven't done them for shared hosting yet. If you really need load balancing for example, chances are that you can afford a set of dedicated servers, and that you can really use all the power from these servers.

I agree. But at the same time I see the need for affordable HA like what www.maximumasp.com offers

Jay

Not wanting to go off topic but 2 issues I can't resolve when using load balancing/directors in a shared environment.

The only way to use LVS in a shared hosting co that I can work out is to use a good file server separate to the web servers but you then face the risk that the file server becomes the single point of failure be it hardware, NFS failures or just a network bottleneck. Can't quite work out the best solution here, I've looked at a few file systems for a file server, Network block device, coda etc etc.

Second issue was to use mysql replication to provide high availability but the issue there is that you can't always ensure that customers only write to the master db server. Always thought that there should be some sort of proxy mysql that sits on each web servers and passes writes to a master db and reads from a pool of slaves.

Originally posted by john@domaincity
Not wanting to go off topic but 2 issues I can't resolve when using load balancing/directors in a shared environment.

The only way to use LVS in a shared hosting co that I can work out is to use a good file server separate to the web servers but you then face the risk that the file server becomes the single point of failure be it hardware, NFS failures or just a network bottleneck. Can't quite work out the best solution here, I've looked at a few file systems for a file server, Network block device, coda etc etc.


There are many solutions to that problem:

* Use rsync replication to have the files on all server
* Use a distributed file system
* Use shared SCSI
* etc.

For distributed file systems you could look at Lustre, AFS, OpenAFS, Coda, GFS, etc.


Second issue was to use mysql replication to provide high availability but the issue there is that you can't always ensure that customers only write to the master db server. Always thought that there should be some sort of proxy mysql that sits on each web servers and passes writes to a master db and reads from a pool of slaves.

Your proxy would then be a SPOF.

You can do one of the following:

* Make your programs do the right thing
* Patch PHP (for example) to do the right thing
* Use mirrored mysql proxies

Or you could remove the problem of having to send updates only one place by using for example EAC for MySQL (other software packages exists that does the same thing). Or you could upgrade to a different type of DMBS such as Oracle.

It sounds to me like you could benefit from hiring in a consultant to help you with the design of your systems.

Originally posted by jks
* Use rsync replication to have the files on all server
* Use a distributed file system

For distributed file systems you could look at Lustre, AFS, OpenAFS, Coda, GFS, etc.

Looked at all of these and like the concepts. My problem is that we're the 30th largest host in the UK (so big but not massive) but we've never had enough tech staff. That means that every system has needed to be quick to install, manage and when the s%!t hits the fan it's got to be quikck to fix or find problems. All of the above just seemed a little too complex for our level of ability.

I've spent 12 months researching systems and software. If the package wasn't working and ready to go 30 mins after I started (although I read the manuals 2 or 3 times at the start) then I'm not really interested. :-)

It sounds to me like you could benefit from hiring in a consultant to help you with the design of your systems.

If you know one that meets with the budget and has the experience to work for a very demanding client then you know where I am.

Originally posted by john@domaincity
Looked at all of these and like the concepts. My problem is that we're the 30th largest host in the UK (so big but not massive) but we've never had enough tech staff.

Hire more tech staff then :-)

That means that every system has needed to be quick to install, manage and when the s%!t hits the fan it's got to be quikck to fix or find problems. All of the above just seemed a little too complex for our level of ability.

There's an old saying in the industry,that goes something like this:

Fast, Good, Cheap... pick any two!

You're basically looking for something that is:

* Distributed without SPOFs
* Scales to meet performance demands
* Can be setup in 30 minutes
* Requires little or no maintenance
* Problems should be easy to find, and quick to fix
* Cheap

This is not really realistic today.


If you know one that meets with the budget and has the experience to work for a very demanding client then you know where I am.

I don't know what your budget is, nor do I know which kind of system it is you want to apply this to... so I can't really help you out right here.

Feel free to send me a private message with an indication of your budget and needs, and I'll see if it's something we can help you with.

Not exactly what you want, but it's similar:

http://availanet.com High-Availability Email

Richard

Originally posted by jks
You're basically looking for something that is:
snip...
* Cheap

This is not really realistic today.
[/B]

Cheap is a bonus but it was never really a requirement.

This is the core component of a reasonable web platform. http://changedfiles.org/
It's dead simple.

Using this it would be easy to have a file server or any other server have a reasonably 'real time' mirror. Only problem I found was that it wouldn't work with NFS or the vserver kenel that we used.

Originally posted by john@domaincity
Cheap is a bonus but it was never really a requirement.

Okay, well - if you can throw lots of money at it -- a solution would be easy to find.


This is the core component of a reasonable web platform. http://changedfiles.org/
It's dead simple.

Using this it would be easy to have a file server or any other server have a reasonably 'real time' mirror. Only problem I found was that it wouldn't work with NFS or the vserver kenel that we used.

Well, I have tried the above... several different implementations of the same thing exists. It can be made to work with the vserver kernel too actually..

If you can live with "less than real time", what's the problem with rsync then?

It's quite easy to setup, no maintenance needed... works like a charm for me.

Combined with something like the changedfiles patch you mention, it's quite good actually.

Moved to Technical & Security Issues

Really looking to have the file system accurate to within 30 mins. Rsync has always been too slow to do a sync of a large file system. One of our main servers has 1000 sites on it and about 8Gb of site files but that takes 2+ hours.

It's not the sort of process you really want to be running every hour :-)

Originally posted by john@domaincity
Really looking to have the file system accurate to within 30 mins. Rsync has always been too slow to do a sync of a large file system. One of our main servers has 1000 sites on it and about 8Gb of site files but that takes 2+ hours.

It's not the sort of process you really want to be running every hour :-)

Who said you had to rsync the whole filesystem every time? :-)

Use something like the changedfiles patch (a nicer interface for that exists for FreeBSD) and then only rsync the actual file that was changed (or directory).

I manage a system with about 15 webservers in our server room that does exactly this. Works just fine there!

Due to special needs, we have it setup so that changed files are "queued" and then every 15 minutes rsyncs are started. Currently it uses a flat structure, but a hierachial structure is just as possible if we want to extend to a lot of servers.

I was going to use changedfiles for rsync but then discovered that vserver and nfs prevented changedfiles from recognising the file changes. :-(

The other mental barrier that I had was 3 web servers and 1 file server with changedfiles mirroring to a backup file server or 3 web servers each with their own local copy of the files. Last option seemed safer but the problem then boils down to customers having to get php/perl scripts to write to the www master server and not the www slave servers.

Every idea I come up with has good plus points for reliability and simplicity but there's always something to bugger it up :-)

Originally posted by john@domaincity
I was going to use changedfiles for rsync but then discovered that vserver and nfs prevented changedfiles from recognising the file changes. :-(

Just use fam instead. It works on NFS for sure... and with the DNotify support it is directly supported by your vserver kernels without any patching (I'm assuming you're using 2.4 kernels).


The other mental barrier that I had was 3 web servers and 1 file server with changedfiles mirroring to a backup file server or 3 web servers each with their own local copy of the files. Last option seemed safer but the problem then boils down to customers having to get php/perl scripts to write to the www master server and not the www slave servers.


Used layered file systems (NFS on local rsync replication fs). Use multi-way replication.

Or just educate your users to never write to files from scripts. Setup FTP servers and the like to always write to the main server.

Well done that man!!

I'd looked at fam a long time ago but had another visit and found http://fileschanged.sourceforge.net/

Very slick :-)

Originally posted by john@domaincity
Well done that man!!

I'd looked at fam a long time ago but had another visit and found http://fileschanged.sourceforge.net/

Very slick :-)

Ehm, that program just uses FAM.

It's a much better idea to write your own FAM client. To ensure that you do not get a bunch of rsync's started up at the same time... also makes it possible to aggregate updates.