Results 1 to 32 of 32
  1. #1
    Join Date
    Feb 2004
    Posts
    1,226

    suggestion for backup solution

    hello

    I'd like to have opinions about the idea I had to backup my servers

    I have 4 servers and I want to backup them one 1 (exclusive for backups)

    My idea is to create folders like
    /home/server01/home/ (the first "/home" is just because this partition will be the biggest one)
    /home/server02/home/
    ...
    /home/server01/mysql/
    /home/server02/mysql/
    ...
    /home/server01/ftpd/
    /home/server02/ftpd/
    ...
    /home/server01/mails/
    /home/server02/mails/
    ...
    /home/server02/namedb/
    /home/server03/namedb/


    well... basically it's that

    The more complicated thing is that, in case I have a problem in one of the main servers, I want to be able to run the domains on the backup server (one of the nameservers of the domains will be on this backup server, so I'll need only to monitor others servers with nagios, for example, and change the DNS entry in case some of them has a problem... and wait for the propagation to the caches)

    to accomplish this, my idea is to create the struct I told above and:
    [I'M CONSIDERING THERE'S NO DUPLICATED USERNAMES ON THE SERVERS]
    - merge the httpd.conf (shouldn't be difficult)
    - find a way to make mysql consult more than one directory to find the databases (any suggestions?)
    if it's not possible, i'll put them all on the same dir, but I'd rather having them separed...
    then I'll need to "merge" the main database (no idea how)
    - I didn't think how to do with ftpd (specially because one server uses pureftpd and others proftpd) and mail yet...

    what must be considered is that this server will only be called in emergency... and probably for less than 4 hours, so it doesn't need to be "perfect"...
    I think that if apache+mysql works it's fine... if possible it could at least queue the mails to be delivered when the main server is back (even if the user can't check them when it's down)

    --/--

    or maybe I should take another approach, like setting one service for each server, on different IPs?
    like
    backup server has usable IP 66.0.0.1 to .6
    I could then open mysql for server01 on 66.0.0.1, for server02 on 66.0.0.2... and httpd for server01 on 66.0.0.1, and so on...

    what do you think?

    thanks

  2. #2
    Join Date
    Nov 2005
    Location
    Maidstone, Kent
    Posts
    92
    something similar to what i have been debating my self.

    one (of probably many) issues that you didnt mention is: what about when a visitor accesses the site while its failedover to backup server and does something that updates to the database. When the main server comes back online you have data to copy back the other way. When do you do that ? and can you just copy it back or does it need merging? the database on the backup may have been say an hour old ? so copy back may delete data. On the other hand if you try and merge the data, you have issues with auto inc fields ! because on main server it allocated say id 123 to a row after the database was last copied to the backup. backup server in turn allocates the same auto inc field value of 123 ! BOOM.

    One option (harsh) is to restrict write access to the database on the backup. This will no doubt screw some functionality (depending on your application this could be devistating or acceptable for a short period).

    Just some extra food for thought .... I would be very interested in a real working solution
    TME Solutions - eCommerce Web Designers - Visit our Web Design Portfolio

  3. #3
    Join Date
    Feb 2004
    Posts
    1,226
    Quote Originally Posted by tmesolutions
    something similar to what i have been debating my self.

    one (of probably many) issues that you didnt mention is: what about when a visitor accesses the site while its failedover to backup server and does something that updates to the database. When the main server comes back online you have data to copy back the other way. When do you do that ? and can you just copy it back or does it need merging? the database on the backup may have been say an hour old ? so copy back may delete data. On the other hand if you try and merge the data, you have issues with auto inc fields ! because on main server it allocated say id 123 to a row after the database was last copied to the backup. backup server in turn allocates the same auto inc field value of 123 ! BOOM.

    One option (harsh) is to restrict write access to the database on the backup. This will no doubt screw some functionality (depending on your application this could be devistating or acceptable for a short period).

    Just some extra food for thought .... I would be very interested in a real working solution
    yes
    i think restricting the write access is the best solution in this case, specially because it's a backup (which is expected to be at least some hours old)

  4. #4
    Join Date
    Nov 2005
    Location
    Maidstone, Kent
    Posts
    92
    agreed.. also a note about the DNS: I presume the backup server is going to be a slave DNS server so it automatically copies updates from primary... so you cant update the secondary :/ so possibly have to change it to a primary if it fails over to the secondary. Then update the ip.. then when moving back make it a secondary again (i have thought on this a bit.. its all a real pain isnt it)

    On top of all that, im a windows user and I have user accounts for each domain.. so i would have to duplicate them too i guess sigh
    TME Solutions - eCommerce Web Designers - Visit our Web Design Portfolio

  5. #5
    Join Date
    Feb 2004
    Posts
    1,226
    Quote Originally Posted by tmesolutions
    agreed.. also a note about the DNS: I presume the backup server is going to be a slave DNS server so it automatically copies updates from primary... so you cant update the secondary :/ so possibly have to change it to a primary if it fails over to the secondary. Then update the ip.. then when moving back make it a secondary again (i have thought on this a bit.. its all a real pain isnt it)

    On top of all that, im a windows user and I have user accounts for each domain.. so i would have to duplicate them too i guess sigh
    in fact they will be "independent"
    the main server DNS will point to the main server IP, so do the backup server DNS
    if the backup server can't reach the main server (for example, checking with nagios), then it will issue a "DNS change" to point to its own IP
    it's the best way I found

  6. #6
    Join Date
    Feb 2004
    Posts
    1,226
    I think the "one server per IP" approach will be easier, specially for mysql (guess it's hard to merge)
    do you agree?
    httpd is easier and wouldn't need a new IP, but since the IP will be used by mysql, it will make things more organized to put each server-related services on specific IPs

    well
    I didn't try that, although I'm thinking/planning a lot... but I can't antecipate any problems (but I'm sure some will happens, but maybe it's not too hard to solve )

    I didn't search too much, but the first result I found on google to easily protect mysql from write is something like:
    /sbin/lidsconf -A -s /usr/local/mysql/libexec/mysqld -o /usr/local/mysql -j READONLY
    (since I'll use FreeBSD, I don't know how to get this... maybe setting read-only permission (444) on the mysql databases will work?)

  7. #7
    Join Date
    Feb 2004
    Posts
    1,226
    the other way I can think is to put "Y" only in the Select_priv row of mysql.db table... but then you'd need to have another backup of this database

  8. #8
    Join Date
    Nov 2005
    Location
    Maidstone, Kent
    Posts
    92
    probably best to experiment. Let me know how it goes plz
    TME Solutions - eCommerce Web Designers - Visit our Web Design Portfolio

  9. #9
    Join Date
    Jun 2003
    Location
    World Wide Web
    Posts
    581
    Lem0nHead, your idea is excellent. The issue is complicated only because it handles four servers at a time, so we need to replicate the processes for all the four servers into one.
    Basically, I think it should be just an extension of clustering in fault tolerance. Apache failover is doing the same thing. While handling mysql, if we make sure all servers are running the same version of mysql same as this backup server, I think you can run everything from /var/lib/mysql

    Inorder to handle ftp for different servers, you might probably need to include two different configuration files from /home/server01/etc/passwd/vhosts, /home/server02/etc/passwd/vhosts and so on. I guess, this could be included in the main ftp file.

    Now its down to email. I think you will need to definitely run four different mail servers for this. Each listening on different port numbers and likely do a 'port redirection' using packet filtering methods in NAT.

    for eg:

    Redirection rule:

    rdr on tl0 proto tcp from 192.0.2.1 to 24.65.1.13 port 25 \
    -> 192.168.1.5 port 8001
    Packet before the rdr rule is processed:

    Source address: 192.0.2.1
    Source port: 4028 (arbitrarily chosen by the operating system)
    Destination address: 24.65.1.13
    Destination port: 25
    Packet after the rdr rule is processed:

    Source address: 192.0.2.1
    Source port: 4028
    Destination address: 192.168.1.5
    Destination port: 8001

    similary, you could use services running on 8002,8003 for different servers.

    I have only put this up for experimenting because it looks possible by theory. I think port forwarding is the way to go if can identify the ip's to which the requests are coming ( if running two sets of base directories for a service is impossible - due to the way server is programmed )
    SupportExpertz.com - the name says it all!
    Managed Cloud Servers
    Server Management and Monitoring
    24x7 outsourced customer support

  10. #10
    Join Date
    Feb 2004
    Posts
    1,226
    maybe I shouldn't set up FTP...
    it won't be so easy (specially with proftpd and pureftpd on some servers) and I think it's not totally necessary... since the idea is that the files be like read-only...

    now for MySQL... i didn't think about that
    the MySQL version of the servers isn't necessary the same

    maybe an approach would be like I said, to run them on different IPs (even if on same port, so PHP scripts for example and connect easilty)... then different versions could be run

    I'll probably start trying to set this up on thursday

  11. #11
    Join Date
    Jun 2003
    Location
    World Wide Web
    Posts
    581
    Ya. I'd appreciate if you could keep us updated on how it goes. It looks like an interesting prospect if everything falls into place.
    SupportExpertz.com - the name says it all!
    Managed Cloud Servers
    Server Management and Monitoring
    24x7 outsourced customer support

  12. #12
    Join Date
    Feb 2004
    Posts
    1,226
    Quote Originally Posted by visiondream3
    Ya. I'd appreciate if you could keep us updated on how it goes. It looks like an interesting prospect if everything falls into place.
    my server just got ready now, so IŽll start it on sunday or monday
    IŽll update it as soon as I can test how some ideas work

  13. #13
    Join Date
    Feb 2004
    Posts
    1,226
    I just got to start it tomorrow
    made some configs, but not enough to things work
    I'll probably keep working on it in the weekend

  14. #14
    Join Date
    Feb 2004
    Posts
    1,226
    http with SSL and scripts for translating the config are ready
    mysql will be the next (probably other day)

  15. #15
    Join Date
    Feb 2004
    Posts
    1,226
    got some time now
    named tested and working... some scripts to replace paths and merge named.conf files

    now "just" mysql is left
    probably the most difficult one...

  16. #16
    Join Date
    Nov 2005
    Location
    Maidstone, Kent
    Posts
    92
    still interested to know how it pans out.. and how effective your backup server is. ie, if any functionality is sacrefised (like write to mysql etc)

    do you think mysql replication is an option ?
    TME Solutions - eCommerce Web Designers - Visit our Web Design Portfolio

  17. #17
    Join Date
    Feb 2004
    Posts
    1,226
    I'm thinking about not replicate... just to backup it from time to time
    but I'll wait for some suggestions while trying to solve this problem...
    the problem is that some servers run mysql 4.0 and others 4.1
    so I'll probably need at least 2 mysql servers running...

  18. #18
    Join Date
    Nov 2005
    Location
    Maidstone, Kent
    Posts
    92
    still... u could use replication couldnt you? even if you are running multiple mysql instances all replicating from seperate mysql servers
    TME Solutions - eCommerce Web Designers - Visit our Web Design Portfolio

  19. #19
    Join Date
    Feb 2004
    Posts
    1,226
    Quote Originally Posted by tmesolutions
    still... u could use replication couldnt you? even if you are running multiple mysql instances all replicating from seperate mysql servers
    yes
    you probably can, but it's not my focus now
    I'm trying to make at least them to work

    I just though about another problem... probably there will not be possible to run different mysql servers...
    you could bind them on different IPs, no problems
    but scripts usually connects to "localhost"... so it will not work
    I'll probably need to use just one mysql... and merge the databases

    waiting some suggestions...

  20. #20
    Join Date
    Nov 2005
    Location
    Maidstone, Kent
    Posts
    92
    ouch.. hmm.. problems
    1) obviouse... but what if 2 servers have the same database
    2) not sooo easy to merge the users database.. and again conflicts ?
    also.. just thought, u cant just do a file copy... some types of mysql database dont always dump to disk all of the time (partly held in ram) i believe.

    what about running X virtual machines on your backup box ? so will be like having each server separate, but in fact all on 1 backup machine
    TME Solutions - eCommerce Web Designers - Visit our Web Design Portfolio

  21. #21
    Join Date
    Feb 2004
    Posts
    1,226
    Quote Originally Posted by tmesolutions
    ouch.. hmm.. problems
    1) obviouse... but what if 2 servers have the same database
    2) not sooo easy to merge the users database.. and again conflicts ?
    also.. just thought, u cant just do a file copy... some types of mysql database dont always dump to disk all of the time (partly held in ram) i believe.

    what about running X virtual machines on your backup box ? so will be like having each server separate, but in fact all on 1 backup machine
    hm...
    1) they won't... the DB has "username_dbname" format... and I never allow 2 customers with the same username on different servers
    2) no conflicts again for the same reason (at least on the usernames... but merging mysql 4.0 with 4.1 may cause some real problem)
    I know I can't copy the files... but there's a program called mysqlhotcopy that can do it pretty well

    for the virtual machines... yes, this would probably be the best way to do that...
    but I have no experience (and, in fact, no idea) doing so
    do you have?

  22. #22
    Join Date
    Nov 2005
    Location
    Maidstone, Kent
    Posts
    92
    nope sorry. I do know that there is a free virtual server software available for linux though. I have used one in windows but that will be totally different. Basically it allows you to simulate any number of pcs within a single pc... very clever. I would check out google
    TME Solutions - eCommerce Web Designers - Visit our Web Design Portfolio

  23. #23
    Join Date
    Feb 2004
    Posts
    1,226
    well... the only trouble I'm seeing is mysqld
    that's the reason I'm reluctant to change everything (forgetting about
    doing everything in one serve and doing it in more than one - even if
    they're virtual)

    so here's another idea I had:
    I run all the needed versions for mysql... one on each IP, in the
    default port (3306)
    on localhost I open a special service on port 3306 and implement some
    basic mysql connection/authentication on this service
    then, based on which username is given to connect, this service would
    redirect (ie, act like a proxy) the connection to the real mysql
    server
    it could also even filter the UPDATE, INSERT and other commands that
    would change the database...
    this is looking like a very promissing way to do it... and I don't
    think it's hard at all
    (maybe some code can be stole from GPL mysql server)

  24. #24
    Join Date
    Nov 2005
    Location
    Maidstone, Kent
    Posts
    92
    sounds overly complex to me ! although once they auth u wouldnt need to filter anything i dont think.. just keep forwarding the same connection to the same real server.
    TME Solutions - eCommerce Web Designers - Visit our Web Design Portfolio

  25. #25
    Join Date
    Feb 2004
    Posts
    1,226
    Quote Originally Posted by tmesolutions
    sounds overly complex to me ! although once they auth u wouldnt need to filter anything i dont think.. just keep forwarding the same connection to the same real server.
    filtering update commands would simulate a read-only mysqld, that's what we wanted on first place
    better yet, without needing to change the database

  26. #26
    Join Date
    Jun 2003
    Location
    World Wide Web
    Posts
    581
    Lem0nHead, I thought for a moment but even if we keep 'localhost' same, it looks like we will need a different port or unix socket for reading multiple data directories. If nothing works you could go back to the first and basic option. On extreme emergency, run it on different mysql servers at different data directories and then use a handy script to search the /home of that server and convert all 'localhost:port' to 'localhost:requiredport'. Wild idea though; you might as well bring the server backup before this script finish combing the servers home directory
    SupportExpertz.com - the name says it all!
    Managed Cloud Servers
    Server Management and Monitoring
    24x7 outsourced customer support

  27. #27
    Join Date
    Feb 2004
    Posts
    1,226
    Quote Originally Posted by visiondream3
    Lem0nHead, I thought for a moment but even if we keep 'localhost' same, it looks like we will need a different port or unix socket for reading multiple data directories. If nothing works you could go back to the first and basic option. On extreme emergency, run it on different mysql servers at different data directories and then use a handy script to search the /home of that server and convert all 'localhost:port' to 'localhost:requiredport'. Wild idea though; you might as well bring the server backup before this script finish combing the servers home directory
    but this idea of the "special service" (on my other post) should take care of this, redirecting to the correct mysqld

  28. #28
    I'm not sure why you would need virtual machines... you do know that you can run multiple instances of MySQL on a server? You can use different sockets and different port numbers.

    As far as free virtualization software, there's User Mode Linux (common but slow) and the more complex Xen. Xen run's modified kernels (NetBSD, Linux, FreeBSD) at near full speed -- compared to the commercial vmware that is slightly faster than UML (which is sloooow).

    There's also a cool virtual project for Windows called CoLinux which will let you run Linux inside of windows at near full speed as well. I'm not sure how far they've progressed lately though.

  29. #29
    Join Date
    Feb 2004
    Posts
    1,226
    Quote Originally Posted by HostingEnthusiast
    I'm not sure why you would need virtual machines... you do know that you can run multiple instances of MySQL on a server? You can use different sockets and different port numbers. (...)
    yes, that's possible
    but when you have many scripts trying to connect to "localhost:3306" (since the idea is not to change the files/scripts, because it should work as a backup), there will be a problem

  30. #30
    Join Date
    Jun 2003
    Location
    World Wide Web
    Posts
    581
    That exactly is our problem. different sockets and different port numbers means function calls in the scripts (php, perl etc) need to be modified. Virtualization is fine, but how do we pick and differentiate data. More than one data directory will be a great feature to have, ofcourse if it does not slow down the server too much. With a straightforward approach, this switching will obviously slow down the mysql requests.
    SupportExpertz.com - the name says it all!
    Managed Cloud Servers
    Server Management and Monitoring
    24x7 outsourced customer support

  31. #31
    I probably misunderstood... I thought the suggestion was that you run a MySQL proxy-type server on 3306 and forward to a virtual server... I suggested running it on a different port to relieve the need of a virtual server. Sorry for the confusion.

  32. #32
    Join Date
    Feb 2004
    Posts
    1,226
    I'm still trying to find a solution to that...
    one problem I'm seeing now is that, by MySQL protocol ( http://www.redferni.uklinux.net/mysq...-Protocol.html ) the server says it's version right when the connection is stabilished... and the client will use this version to communicate

    but we don't know yet to which server we will need to connect
    so probably it would be need to "translate" protocols versions when proxying a different version
    that probably wouldn't be easy

    --/--

    while I think about that, my other idea (if everything fails), is to ask users to configure mysql host as "mysql.domain.com" instead of "localhost"
    so I'll be able to resolve this host to the correct IP...

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •