Results 1 to 2 of 2
-
11-18-2006, 01:37 PM #1Web Hosting Master
- Join Date
- Aug 2004
- Location
- Shanghai
- Posts
- 1,475
How to setup a Xen VPS Server using LVM, RAID1 and dtc-xen
This howto is pretty long, because explaining in details. This is just in order to make it easy to understand. If you got some troubles, don't fear to get in touch with me, I'll be happy to give a hand.
Note: this howto does NOT explain how to setup a Debian. You MUST have basic knowledge of Unix and especialy Debian administration to deal with it.
A / Setting-up a Debian with LVM and RAID1
1. Why?
This howto explains how to have a working setup using LVM and RAID1 ready to host Xen VPS.
RAID is something you HAVE to use if you care a bit about your customer's data. RAID1 in software is the cheapest way to do mirroring, and it makes it easy to fit in a 1U server: that will save on costs, and you will be able to give back the saved money to your customers.
When using Xen, all of your VPS will have to use a separate disk. There are 3 ways to give a disk to a Xen VPS. The first one (which is the most commonly (miss)used) is a file loopback. This is a very poor setup, as it's slow, and as the file-loopback descriptors are limited in numbers in the kernel (even if that number can be set to a higher value, it's not something you want to do). The second way is to use a NFS server. This is a very good idea, as with a NFS server you will be able to do live migration of your VMs from one physical server to another. But this is a very expensive setup, and you will have to use a gigabit LAN network with many servers. The last way is the one we use: we use LVM to give an access to a physical partition of the server. LVM allow resize of the partition on-the-fly (you will just need to shutdown the server for a very short time to do it), and it's as fast as a normal partition. Moreover, LVM partitions are not limited in number.
You will need LVM to give a (fast) physical partition to your VPS, but Linux don't know how to boot on LVM, so you will need to use a small RAID1 partition on which you will boot. Our setup uses a 500 MB to 1GB / (root) partition, so it's more easy to understand how it works (i.e: the /boot is still in /boot and not in / at the grub stage), but some might prefer to use a separate /boot partition that will be able to be even smaller (only few megs for this /boot partition would be enough). Both setup work, it's more a mater of tastes.
Let's start the tutorial now!
2. Booting
We always use the network sarge installer, but a normal full CD or DVD is ok. If you use amd64 arch and newer hardware, we advice you to use the daily built debian installer (with the lastest kernel).
First, boot the Debian Installer (DI). On the syslinux prompt, you need to type "expert26" in order to boot with kernel 2.6 and have the installer ask more questions.
Do the begining of the setup as usual (keyboard, region, timezone, etc.), and don't forget to ask to load the RAID and the LVM modules. Then you should arrive at the partition setup screen.
3. Creating the physical partitions
You should now create the partitions for RAID1. First, create a small partition at the begining of each disks. They will be used for the / (root partition), and it doesn't have to be bigger than 800 MB. Set the boot flag on each disks for this partition, and select "use as", then "physical device for RAID".
Then select the free space on each disk, and create a partition that should take all the remaining space but 5BG, and then select to use them as RAID device like you just did for the (small) first partition. You should leave 5GB unused for 2 reasons. First, the end of the disks are often where you find bad sectors. Second, when a HDD needs to be replaced (because it's broken for example), it's hard to find exactly the same type. If you buy a replacement disk that is few MB smaller, and can't create a RAID1 partition of exactly the same size, then you are stucked.
Now you should have 2 disks, with each 1 small partition of 800 MB with the boot flag on, and a big one with all the rest of the space but 5GB. All partition marked to be used for RAID1. It's time to go in "finish and write to disk", and then reboot (yes, without the partition being mounted!). Linux loads the partition tables ONLY at boot time, so you really need to reboot. Note that this seems to be true only for IDE devices, newer hardware using SATA or SCSI don't seem to need a reboot.
4. Creating 2 RAID1 devices
Restart the installer, do like you did before, and reach the partitionning utility. You should now see the partitions you created.
Go in the RAID utility menu of the partition screen (on top of the menu). Select the option to create RAID device, select RAID1, say you want to use 2 disks, and no spare disk (so: create -> raid1 -> 2 -> 1). Select the first and the third entry (the small boot partition on each disks). Do it again with the 2 remaining partition (that big one) on each disk. Leave the RAID1 menu. It should start to do the copy of the first disk to the 2nd (RAID1 is done at device level, not filesystem, it never knows if the disks are empty, in fact it doesn't care at all of the content of the datas itself).
Now, back at the partman screen, you should see the 2 partitions on each disks, plus 2 RAID1 devices.
5. Creating the LVM devices
Select the 2nd RAID1 device (the one using the big partition). Say you want to use it as a LVM device, then go back to the main screen.
It's now time to create the LVM partitions for your xen management operating system (the domO). Go on the top entry of the partition screen (partman), and go in the LVM manager. Go in the "volume group" management, and create the LVM array. It will normaly detect the LVM partition that you have set on the big raid device. It will prompt you for a name, call it "lvm1" as this is the name that will use DTC-Xen (otherwise you will have to do some more configuration later), and select the (only) available device you just created that is in fact the big RAID1 device.
Exit the volume group creation menu, and now go in the "logical volume" creation. Create 5 logical volumes. It will ask you for a name, just call it "usr" for the /usr partition and so on. A partition scheme like this one should be ok:
PHP Code:name size
usr 3GB
var 1GB
tmp 400MB
root 400MB
swap 512MB
6. Selecting type (ext3) and mount points
Exit the LVM utility, and go back to the main partition screen. You should now see the 2 raid1 device, the 5 logical volumes you did, and the 2 physical partitions on each disks. It's time for you to mount the partitions.
First, select the small RAID1 device. Say you want to use it as ext3, and that the mount point is / (the root partition), using ext3 fs.
Then select the "usr" logical volume say you want to use it as ext3, and select the mount point. Do the same for /var, /tmp and /root (for /root you will need to type the full path of the mount point). Select the "swap" logical volume and say you want to use it as swap space.
7. You are good to go!
All your partitions are now created and the mount point is selected for all of them. Just go in "Finish and write change to disk", it will format them all and do the rest of the install as usual.
One last thing: the DI will warn you again that Linux doesn't load partition tables on the fly, but at boot time only. Just ignore the message this time: it's true it can't load the partition tables, but it knows about the changes you did with LVM and RAID1 arrays.
8. Important thing
When doing this, you HAVE to follow the order:
* Create the RAID partitions on both disks
* Reboot
* Create the LVM physical volume
* Create the LVM logical volumes
* Mount the / on the first RAID1 device
* Select the moint point for the LVM volumes
If you don't, you will screw everything.
B / Upgrade your kernel to Xen
1 - Downloading and unpacking the kernel and Xen
Go in /usr/src and download the lastest Xen stable sources. Currently (at the time of writting), you can do:
PHP Code:wget http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/xen-3.0.2-2-src.tgz
Unpack the xen sources:
PHP Code:tar -xvzf xen-3.0.2-2-src.tgz
PHP Code:wget ftp://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.29.tar.bz2
ln -s ../linux-2.6.16.29.tar.bz2 xen-3.0.2-2/linux-2.6.16.tar.bz2
PHP Code:cd xen-3.0.2-2
PHP Code:KERNELS ?= linux-2.6-xen
PHP Code:KERNELS ?= linux-2.6-xen0 linux-2.6-xenU
apt-get install make gcc libc6-dev zlib1g-dev python python-dev python-twisted bridge-utils iproute libcurl3 libcurl3-dev bzip2 module-init-tools transfig tgif libncurses5-dev patch libvncserver-dev libsdl-dev libjpeg62-dev bcc bin86
This will download as well the things needed to compile Xen with HVM support (VMX), the hardware assisted emulation that can run Windows (for example). Now, just type:
PHP Code:make world
3 - Customizing the kernel
The way to configure your dom0 kernel is easy:
PHP Code:cd linux-2.6.16-xen0
make menuconfig
PHP Code:cd linux-2.6.16-xenU
make menuconfig
As per default, many things has to be changed. Namely, you need to have support for iptables, but also physdev match (if you want to use the anti-spoof facility of Xen), QoS (all the options), pf_key and other routing facilities. Note that you might need to compile all the modules needed by the Xen domU in your dom0, I have noticed once that a domU kernel module didn't work if it was not compiled in the dom0 as well, which shouldn't be the case, but this is what I could experience. If you want to satisfy all of your customers, best is to compile all the network options you can, and if you want to be able to access to the file systems of guest OS, it can be nice to have support for things like NetBSD fs, NTFS and others. The dom0 is the most important kernel to configure, as there is always the possibility to customize the domU kernel later, when your box is ready and in the data center (no need for a hardware reset for your domU kernel).
Once you are satisfied, go on the toplevel folder of the xen sources, and type make install
4 - Grub
Edit grub, and set the first option to be something like this:
PHP Code:title Xen 3.0 / XenLinux 2.6
kernel /boot/xen-3.0.2-2.gz dom0_mem=262144
module /boot/vmlinuz-2.6.16.29-xen0 root=/dev/md0 ro console=tty0
## ## End Default Options ##
title Debian GNU/Linux, kernel 2.6.8-2-386
root (hd1,0)
kernel /boot/vmlinuz-2.6.8-2-386 root=/dev/md0 ro
initrd /boot/initrd.img-2.6.8-2-386
boot
Double, and even tripple check that your kernel version number correspond to what you have just compiled (and that should be now located in your /boot).
If not done before, you can install grub on both disks:
PHP Code:grub --no-floppy
root (hd0,0)
setup (hd0)
root (hd1,0)
setup (hd1)
C / Setup dtc-xen so you can manage your server with DTC control panel
1 - Setup a DTC control panel
This wont be detailed here. Note that you can do this inside one of the VMs of your Xen Server, that's no problem.
2 - Naming conventions
Your Xen Server dom0 MUST have a node number like nodeXXXX.your-domain.com, with XXXX being with 4 to 5 characters. At GPLHost, we use the phone number prefix for the first 2 or 3 characters, and then the last 2 increase. If you will have only one xen server, 00001 is fine. We will use 1212 in this example, and my-host.com as the domain name, so this will be: node1212.my-host.com, and 1.2.3.4 as the IP of the dom0 of your xen server.
3 - Preparing the dom0
Your dom0 MUST be able to find 1.2.3.4 when it tries to resolve node1212.my-host.com. Best is to setup /etc/hosts the following way:
PHP Code:1.2.3.4 node1212.my-host.com node1212
4 - Installing the replacement libc6, dtc-xen package and the rest
You might have noticed that when starting your server you saw a message asking you to move /lib/tls away. This is because you don't have a libc6 that is Xen aware. No problem, GPLHost has one ready for you. First, add one of the GPLHost repository to your dom0 in your /etc/apt/sources.list:
deb ftp://ftp.gplhost.com/debian stable xen # US main ftp
deb ftp://ftp.gplhost.sg/debian stable xen # Singapore mirror (for asian users)
deb ftp://ftp.gplhost.fr/debian stable xen # Paris mirror (for european users)
Do a apt-get update, then apt-get upgrade to get the xen-aware libc6. Note that Etch has already a libc6 that work for Xen (it's called something like libc6-xen...), and that amd64 arch don't need the update.
Then it's time to:
PHP Code:apt-get install dtc-xen
5 - Adding the Xen Server to the DTC panel, and provisionning some VPS
Now that the dtc-xen package is setup, you can connect the dtc panel to it. Go in the general config screen of dtc, click on the vps server tab, and enter the following in the feilds:
PHP Code:node1212.my-host.com / XXXXXXX / YYYYYYYYY
6 - Provisionning some IPs, VMs and editing in the package manager
Next, you should click on "edit ip addresses" and add some VPS IPs to your xen servers. Add as much as you can, and they will be able to be taken by your customers when they register with DTC.
Then you should go in the Package manager and create some VPS package. Nothing special to say here, exept that you need to create some renewal package as well in order to enable recurring possibility.
7 - Preparing the server for hosting NetBSD and CentOS
You need to download all the packages necessary to setup a CentOS distrib. Do it this way:
PHP Code:cd /usr/src
mkdir centos3
rpmstrap --verbose --download-only centos3 /usr/src/centos3
PHP Code:cd /boot
wget ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-daily/netbsd-4/200609170000Z/i386/binary/kernel/netbsd-XEN3_DOMU.gz
gzip -d netbsd-XEN3_DOMU.gz
mv netbsd-XEN3_DOMU netbsd-XENU
wget ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-daily/netbsd-4/200609170000Z/i386/binary/kernel/netbsd-INSTALL_XEN3_DOMU.gz
gzip -d netbsd-INSTALL_XEN3_DOMU.gz
mv netbsd-INSTALL_XEN3_DOMU netbsd-INSTALL_XENU
That's about it now. Good luck!GPLHost:>_ open source hosting worldwide (I'm founder, CEO & official Debian Developer)
Servers & our leading control panel and our Xen VPS hosting, which are already included in Debian and Ubuntu
Available in: Kuala Lumpur, Singapore, Sydney, Seattle, Atlanta, Paris, London, Barcelona, Zurich, Israel
-
11-18-2006, 05:08 PM #2LORD OF THE RINGS
- Join Date
- Dec 2005
- Location
- Internet
- Posts
- 1,352
Thanks for sharing the information