How to setup a Xen VPS Server using LVM, RAID1 and dtc-xen
This howto is pretty long, because explaining in details. This is just in order to make it easy to understand. If you got some troubles, don't fear to get in touch with me, I'll be happy to give a hand.
Note: this howto does NOT explain how to setup a Debian. You MUST have basic knowledge of Unix and especialy Debian administration to deal with it.
A / Setting-up a Debian with LVM and RAID1
This howto explains how to have a working setup using LVM and RAID1 ready to host Xen VPS.
RAID is something you HAVE to use if you care a bit about your customer's data. RAID1 in software is the cheapest way to do mirroring, and it makes it easy to fit in a 1U server: that will save on costs, and you will be able to give back the saved money to your customers.
When using Xen, all of your VPS will have to use a separate disk. There are 3 ways to give a disk to a Xen VPS. The first one (which is the most commonly (miss)used) is a file loopback. This is a very poor setup, as it's slow, and as the file-loopback descriptors are limited in numbers in the kernel (even if that number can be set to a higher value, it's not something you want to do). The second way is to use a NFS server. This is a very good idea, as with a NFS server you will be able to do live migration of your VMs from one physical server to another. But this is a very expensive setup, and you will have to use a gigabit LAN network with many servers. The last way is the one we use: we use LVM to give an access to a physical partition of the server. LVM allow resize of the partition on-the-fly (you will just need to shutdown the server for a very short time to do it), and it's as fast as a normal partition. Moreover, LVM partitions are not limited in number.
You will need LVM to give a (fast) physical partition to your VPS, but Linux don't know how to boot on LVM, so you will need to use a small RAID1 partition on which you will boot. Our setup uses a 500 MB to 1GB / (root) partition, so it's more easy to understand how it works (i.e: the /boot is still in /boot and not in / at the grub stage), but some might prefer to use a separate /boot partition that will be able to be even smaller (only few megs for this /boot partition would be enough). Both setup work, it's more a mater of tastes.
Let's start the tutorial now!
We always use the network sarge installer, but a normal full CD or DVD is ok. If you use amd64 arch and newer hardware, we advice you to use the daily built debian installer (with the lastest kernel).
First, boot the Debian Installer (DI). On the syslinux prompt, you need to type "expert26" in order to boot with kernel 2.6 and have the installer ask more questions.
Do the begining of the setup as usual (keyboard, region, timezone, etc.), and don't forget to ask to load the RAID and the LVM modules. Then you should arrive at the partition setup screen.
3. Creating the physical partitions
You should now create the partitions for RAID1. First, create a small partition at the begining of each disks. They will be used for the / (root partition), and it doesn't have to be bigger than 800 MB. Set the boot flag on each disks for this partition, and select "use as", then "physical device for RAID".
Then select the free space on each disk, and create a partition that should take all the remaining space but 5BG, and then select to use them as RAID device like you just did for the (small) first partition. You should leave 5GB unused for 2 reasons. First, the end of the disks are often where you find bad sectors. Second, when a HDD needs to be replaced (because it's broken for example), it's hard to find exactly the same type. If you buy a replacement disk that is few MB smaller, and can't create a RAID1 partition of exactly the same size, then you are stucked.
Now you should have 2 disks, with each 1 small partition of 800 MB with the boot flag on, and a big one with all the rest of the space but 5GB. All partition marked to be used for RAID1. It's time to go in "finish and write to disk", and then reboot (yes, without the partition being mounted!). Linux loads the partition tables ONLY at boot time, so you really need to reboot. Note that this seems to be true only for IDE devices, newer hardware using SATA or SCSI don't seem to need a reboot.
4. Creating 2 RAID1 devices
Restart the installer, do like you did before, and reach the partitionning utility. You should now see the partitions you created.
Go in the RAID utility menu of the partition screen (on top of the menu). Select the option to create RAID device, select RAID1, say you want to use 2 disks, and no spare disk (so: create -> raid1 -> 2 -> 1). Select the first and the third entry (the small boot partition on each disks). Do it again with the 2 remaining partition (that big one) on each disk. Leave the RAID1 menu. It should start to do the copy of the first disk to the 2nd (RAID1 is done at device level, not filesystem, it never knows if the disks are empty, in fact it doesn't care at all of the content of the datas itself).
Now, back at the partman screen, you should see the 2 partitions on each disks, plus 2 RAID1 devices.
5. Creating the LVM devices
Select the 2nd RAID1 device (the one using the big partition). Say you want to use it as a LVM device, then go back to the main screen.
It's now time to create the LVM partitions for your xen management operating system (the domO). Go on the top entry of the partition screen (partman), and go in the LVM manager. Go in the "volume group" management, and create the LVM array. It will normaly detect the LVM partition that you have set on the big raid device. It will prompt you for a name, call it "lvm1" as this is the name that will use DTC-Xen (otherwise you will have to do some more configuration later), and select the (only) available device you just created that is in fact the big RAID1 device.
Exit the volume group creation menu, and now go in the "logical volume" creation. Create 5 logical volumes. It will ask you for a name, just call it "usr" for the /usr partition and so on. A partition scheme like this one should be ok:
This should leave most of the LVM space free to be used by your all your VPS. 5GB for /usr is enough so you can compile Xen in /usr/src.
6. Selecting type (ext3) and mount points
Exit the LVM utility, and go back to the main partition screen. You should now see the 2 raid1 device, the 5 logical volumes you did, and the 2 physical partitions on each disks. It's time for you to mount the partitions.
First, select the small RAID1 device. Say you want to use it as ext3, and that the mount point is / (the root partition), using ext3 fs.
Then select the "usr" logical volume say you want to use it as ext3, and select the mount point. Do the same for /var, /tmp and /root (for /root you will need to type the full path of the mount point). Select the "swap" logical volume and say you want to use it as swap space.
7. You are good to go!
All your partitions are now created and the mount point is selected for all of them. Just go in "Finish and write change to disk", it will format them all and do the rest of the install as usual.
One last thing: the DI will warn you again that Linux doesn't load partition tables on the fly, but at boot time only. Just ignore the message this time: it's true it can't load the partition tables, but it knows about the changes you did with LVM and RAID1 arrays.
8. Important thing
When doing this, you HAVE to follow the order:
* Create the RAID partitions on both disks
* Create the LVM physical volume
* Create the LVM logical volumes
* Mount the / on the first RAID1 device
* Select the moint point for the LVM volumes
If you don't, you will screw everything.
B / Upgrade your kernel to Xen
1 - Downloading and unpacking the kernel and Xen
Go in /usr/src and download the lastest Xen stable sources. Currently (at the time of writting), you can do:
I've noticed that the kernels from www.cl.cam.ac.uk are most of the time more accurate than the ones from xensource.com, and that even the same file on both location where different. Strange...
Unpack the xen sources:
tar -xvzf xen-3.0.2-2-src.tgz
Then, as you want to have the most up-to-date kernel with the latest bug-fix and patches, you need to download by hand the Linux kernel, and make a symlink to it in the Xen sources, otherwise Xen 3.0.2-2 will download the oldest kernel 2.6.16 and you want the newest. Here's how:
2 - Preparing the system to compile the Xen kernel
Now edit the toplevel Makefile in the xen source tree. The first line says to compile only one kernel, I do not advice you to do it, it's best to customize the domU kernel with some options you wont need for dom0 and vice-versa. So change the line:
KERNELS ?= linux-2.6-xen
To something like:
KERNELS ?= linux-2.6-xen0 linux-2.6-xenU
Then it's time for you to get all the things you need to compile. Here's my one time apt-get command:
This will download as well the things needed to compile Xen with HVM support (VMX), the hardware assisted emulation that can run Windows (for example). Now, just type:
and this will compile everything for you.
3 - Customizing the kernel
The way to configure your dom0 kernel is easy:
You can do the same for domU:
Add all the hardware drivers that you need as usual, and don't forget to check for the support of RAID1 and LVM2 (which is called device mapper in the kernel).
As per default, many things has to be changed. Namely, you need to have support for iptables, but also physdev match (if you want to use the anti-spoof facility of Xen), QoS (all the options), pf_key and other routing facilities. Note that you might need to compile all the modules needed by the Xen domU in your dom0, I have noticed once that a domU kernel module didn't work if it was not compiled in the dom0 as well, which shouldn't be the case, but this is what I could experience. If you want to satisfy all of your customers, best is to compile all the network options you can, and if you want to be able to access to the file systems of guest OS, it can be nice to have support for things like NetBSD fs, NTFS and others. The dom0 is the most important kernel to configure, as there is always the possibility to customize the domU kernel later, when your box is ready and in the data center (no need for a hardware reset for your domU kernel).
Once you are satisfied, go on the toplevel folder of the xen sources, and type make install
4 - Grub
Edit grub, and set the first option to be something like this:
As you see, you should set the xen kernel BEFORE the ## ## End Default Options ##. This way, the debian system wont overwrite it. Note that my experience showed that something lower than 192MB for the dom0 is to avoid, and 256MB seems to be more than enough.
Double, and even tripple check that your kernel version number correspond to what you have just compiled (and that should be now located in your /boot).
If not done before, you can install grub on both disks:
C / Setup dtc-xen so you can manage your server with DTC control panel
1 - Setup a DTC control panel
This wont be detailed here. Note that you can do this inside one of the VMs of your Xen Server, that's no problem.
2 - Naming conventions
Your Xen Server dom0 MUST have a node number like nodeXXXX.your-domain.com, with XXXX being with 4 to 5 characters. At GPLHost, we use the phone number prefix for the first 2 or 3 characters, and then the last 2 increase. If you will have only one xen server, 00001 is fine. We will use 1212 in this example, and my-host.com as the domain name, so this will be: node1212.my-host.com, and 184.108.40.206 as the IP of the dom0 of your xen server.
3 - Preparing the dom0
Your dom0 MUST be able to find 220.127.116.11 when it tries to resolve node1212.my-host.com. Best is to setup /etc/hosts the following way:
18.104.22.168 node1212.my-host.com node1212
And check that /etc/hostname has node1212.
4 - Installing the replacement libc6, dtc-xen package and the rest
You might have noticed that when starting your server you saw a message asking you to move /lib/tls away. This is because you don't have a libc6 that is Xen aware. No problem, GPLHost has one ready for you. First, add one of the GPLHost repository to your dom0 in your /etc/apt/sources.list:
Do a apt-get update, then apt-get upgrade to get the xen-aware libc6. Note that Etch has already a libc6 that work for Xen (it's called something like libc6-xen...), and that amd64 arch don't need the update.
Then it's time to:
apt-get install dtc-xen
Answer to it's questions. Best is to pickup a randomly generated password for the SOAP server and cut-past it to the debconf screen when it asks for a password (dtc can do it for you in many screen, like the email tab of the client interface). Make sure you keep this password somewhere, as it will be crypted and you will need to enter it in your DTC panel interface.
5 - Adding the Xen Server to the DTC panel, and provisionning some VPS
Now that the dtc-xen package is setup, you can connect the dtc panel to it. Go in the general config screen of dtc, click on the vps server tab, and enter the following in the feilds:
node1212.my-host.com / XXXXXXX / YYYYYYYYY
where XXXXXXX is the SOAP password you have just entered durring the setup, and YYYYYYYYY is just the location of the Xen server to show up when somebody will register (something like "foo data center in bar country" is fine). Note that if there is more than one server with the exact same location, the panel will show only one entry in the registration popup, which is rather nice.
6 - Provisionning some IPs, VMs and editing in the package manager
Next, you should click on "edit ip addresses" and add some VPS IPs to your xen servers. Add as much as you can, and they will be able to be taken by your customers when they register with DTC.
Then you should go in the Package manager and create some VPS package. Nothing special to say here, exept that you need to create some renewal package as well in order to enable recurring possibility.
7 - Preparing the server for hosting NetBSD and CentOS
You need to download all the packages necessary to setup a CentOS distrib. Do it this way:
rpmstrap --verbose --download-only centos3 /usr/src/centos3
Then for NetBSD, you only need to download the NetBSD normal and install kernel for xenU:
You can replace XEN3 by XEN2 depending on what you use. Also, the URL showed here are only examples, you might wont be able to find them as those are the daily archives (often rotated), but it should be pretty easy to find.
That's about it now. Good luck!
GPLHost:>_ open source hosting worldwide (I'm founder, CEO & official Debian Developer)
Servers & our leading control panel and our Xen VPS hosting, which are already included in Debian and Ubuntu
Available in: Kuala Lumpur, Singapore, Sydney, Seattle, Atlanta, Paris, London, Barcelona, Zurich, Israel