How to create multiple Xen virtual machines with LVM backing

This tutorial shows how to create multiple Xen images under CentOS 5, using LVM logical volumes directly as backing store.

This means that it's easy to mount, manipulate and resize Xen filesystems directly in dom0 when the VM is offline. Furthermore, using LVM volumes for filesystems and swap means there is lower overhead than filesystem-based backing.

Summary

We will install a CentOS 5 system with Xen, leaving plenty of unallocated LVM space.
We will make another CentOS 5 installation within a VM, as a master image
We will see how to separate out the "partitions" within this master image, and make image copies into new logical volumes
We will boot new VMs using these logical volumes

Step 1: Prepare your base install

(If you have an existing CentOS 5 installation, with some spare LVM space, then go to step 1a)

Setting up Xen under CentOS 5 is very straightforward. At installation time you simply select the "Virtualization" option, and the installation will have a Xen kernel and the necessary Xen utilities.

When setting up a new system from scratch, you should take the opportunity to choose a suitable partitioning scheme.

If you have two hard drives and wish to use them mirrored, then create
- /dev/hda1 and /dev/hdb1: 256MB
- /dev/hda2 and /dev/hdb2: rest of disks
- /dev/md0 as /dev/hda1 mirrored with /dev/hdb1
- /dev/md1 as /dev/hda2 mirrored with /dev/hdb2
Otherwise, just create /dev/hda1 (256MB) and /dev/hda2 (rest of disk)
Set the first partition as "ext3" mounted on /boot. This is because grub can't boot from LVM volumes
Set the second partition as "LVM physical volume"
Within LVM, create a volume group called "dom0" and create logical volumes within it:
- dom0_root 4GB (ext3, mount on /)
- dom0_swap 1GB (swap)
- dom0_var 10GB (ext3, mount on /var)

These sizes are just guidelines. You can also set up LVM volumes for /usr or /home as you wish. Just make sure you leave plenty of unused space within your volume group, because this is what we'll use for backing store for virtual machines. Also, make sure /var is at least 4GB, as we're going to use 3GB of it in the next section.

Remember that you can always grow these partitions later using lvresize and resize2fs.

Set up the base installation however you're comfortable. For example if this is a desktop or laptop system, you might wish to have Gnome in your dom0; if it's a lights-out server then you may wish to leave those out.

Reboot into your new OS. Login, su to root, and check that xm list works and shows your dom0 instance.

(Note: this is not a tutorial on how to use Xen. If you are new to xen, search for documentation on xm, virt-manager and virt-install)

Step 1a: Adding Xen to an existing CentOS system

If you have a suitable base install but simply didn't select Virtualization, you can add it later by

yum groupinstall Virtualization
service xend enable
reboot

Your system should reboot with a xen kernel running. Use xm list to check that everything is working.

Step 2: Create a master VM image

We will now install CentOS 5 as a virtual image within CentOS 5. Because anaconda assumes that it's connected to a real device, we will end up with a single image file which contains a partition table and three partitions. We'll tease these out later.

If you have a CentOS 5 DVD ISO image available on your base system, mount it as a loopback and share it using HTTP:

mkdir /var/www/html/centos
mount -o loop /path/to/CentOS-5.1-i386-bin-DVD.iso /var/www/html/centos
/etc/init.d/httpd start

If you have a real physical DVD-ROM, then mount it under /var/www/html/centos or symlink it there.

In either case, the install URL will be http://192.168.122.1/centos

(If you can't do this, then give an install URL of a local mirror, e.g. http://www.mirrorservice.org/sites/mirror.centos.org/5/os/i386 - but this will use quite a lot of Internet bandwidth)

Now start up virt-manager:

virt-manager

This gives a graphical utility to allow you to create a new VM. Call it "centos5_template". Give it a file backing, /var/lib/xen/images/centos5_template, of size 3072MB.

The installation will now proceed within a window.

Now you need to be careful to partition the virtual disk correctly. The message will say "The partition table on device xvda was unreadable", which is fine. Say Yes to initialize this virtual "drive".

Choose "Create custom layout". You'll see an empty partition table, with free space of 3100MB (cylinder 1 to 396)

New: mount point /: type ext3: size 2048MB: fixed size
New: mount point /var: type ext3: size 512MB: fixed size
New: type swap: fill to maximum allowable size

This should give you:

/dev/xda1: /
/dev/xda2: /var
/dev/xda3: swap

(Make a note if this isn't the order you get)

Note that we do NOT want to set up LVM here. We'll create logical volumes in the host system when we copy this snapshot.

When partitioning is complete, set up the target to use DHCP to pick up an IP address dynamically. If you want your final VMs to have static IPs you can set this up after you've cloned each one.

Now continue as per a normal install. Since we're making a basic 2GB image, disable "Desktop - Gnome" so that only a basic set of packages is installed (this actually uses about 800MB). We can add extra packages within our target VMs later as required.

Once installation is complete, boot into your VM. Bring it up to date as follows:

yum upgrade
yum clean packages
halt -p

Before halting you can also install any other packages that you wish to have available in all machines which are cloned from this image.

Step 3: Create LVM logical volumes for a new VM

Now we want to create a new virtual machine. To do this, create three logical volumes inside your main volume group. Use "vgscan" to check your volume group name; it may be VolGroup00 if you took the CentOS default. I'm going to assume now you created it as dom0.

lvcreate --size 2048M --name scratch1_root dom0
lvcreate --size 512M --name scratch1_var dom0
lvcreate --size 512M --name scratch1_swap dom0

You're free to make the scratch1_root and scratch1_var logical volumes larger than these sizes if you wish. The scratch1_swap LV can be any size you like.

Typy lvscan and you should now see logical volumes called /dev/dom0/scratch1_* as above.

Preparing the swap space is easy:

mkswap /dev/dom0/scratch1_swap

Now, our LVM image above consists of a single file which contains a partition table, a / partition, a /var partition and the swap partition (the latter we're not going to use; it was just to provide swap space during the installation)

A little work is now required to tease out these partitions and copy them into the logical volumes we created above.

MAKE SURE THE TEMPLATE VM IS NOT RUNNING AT THIS POINT. Use xm list to confirm, or check in virt-manager.

We will use fdisk on the image file to find out the offsets and sizes to the partitions it contains:

# fdisk -lu /var/lib/xen/images/centos5_template.img
last_lba(): I don't know how to handle files with mode 81ed
You must set cylinders.
You can do this from the extra functions menu.

Disk /var/lib/xen/images/centos5_template.img: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes

                                   Device Boot      Start         End      Blocks   Id  System
/var/lib/xen/images/centos5_template.img1   *          63     4192964     2096451   83  Linux
/var/lib/xen/images/centos5_template.img2         4192965     5237189      522112+  83  Linux
/var/lib/xen/images/centos5_template.img3         5237190     6345674      554242+  82  Linux swap / Solaris

The start and end offsets are in sectors (512 bytes), whilst the size is given in blocks (1K). Confirm that the first two partitions are slightly smaller than the ones we created using lvcreate. (2048M = 2097152K; 512M = 524288K)

Now calculate the size of each in blocks:

bc
4192964-63+1         # 4192902
5237189-4192965+1    # 1044225

And perform the actual image copy into the logical volumes:

dd skip=63 count=4192902 if=/var/lib/xen/images/centos5_template.img \
    obs=4096K of=/dev/dom0/scratch1_root
dd skip=4192965 count=1044225 if=/var/lib/xen/images/centos5_template.img \
    obs=4096K of=/dev/dom0/scratch1_var

(The first copy takes 2 mins 20 secs on a laptop with a 5400rpm 2.5in drive; a desktop system with SATA drive should be quicker than that)

For a quick check that everything is OK:

e2fsck /dev/dom0/scratch1_root
e2fsck /dev/dom0/scratch1_var

Both should be shown as "clean". If you made the logical volumes larger than the template partitions, now is a good time to grow the filesystems to fit:

e2fsck -f /dev/dom0/scratch1_root
resize2fs /dev/dom0/scratch1_root
e2fsck -f /dev/dom0/scratch1_var
resize2fs /dev/dom0/scratch1_var

You can, if you wish, mount these logical volumes and browse or update the filesystems. Just remember to unmount them before starting Xen on them.

Step 4: Create a Xen config file

Type "uuidgen" to get a unique ID for the system. Then create a Xen config file using vi /etc/xen/scratch1

Base it on the following template:

name = "scratch1"
uuid = "f0e47dd6-e919-407f-8fd1-5384f0dd93cd"
maxmem = 512
memory = 256
vcpus = 1
bootloader = "/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vfb = [ ]
disk = [ "phy:/dev/dom0/scratch1_root,xvda1,w",
         "phy:/dev/dom0/scratch1_var,xvda2,w",
         "phy:/dev/dom0/scratch1_swap,xvda3,w" ]
vif = [ "mac=00:16:3e:dd:93:cd,bridge=virbr0" ]

For the vif, choose random values for the last three bytes of the MAC address (the last 3 bytes of the uuid would be good ones to choose)

If you want a graphical console then use vfb = [ "type=vnc,vncunused=1" ]

Step 5: Start your cloned virtual machine

You can use xm to start and attach a console in one go:

xm create -c scratch1

Or you can start it using virt-manager, although if your VM has no graphical console then you'll need to connect to the text console separately:

xm console scratch1

You're advised to edit the hostname in /etc/sysconfig/network and /etc/hosts right away, otherwise it can be very confusing which machine you're talking to.

Hit Ctrl-] to detach the text console.

Step 6: Scripted install

None of the above work is particularly onerous, but if you're going to create a lot of VMs this way then you may wish to write a simple script to automate it.

Here is a sample which you can customise easily. I install it as /root/bin/clone-vm. Using this script you can create a new virtual machine in about 3 minutes.

Note that it doesn't perform much error checking, in particular it doesn't check for logical volume sizes which are too small.

#!/bin/sh

volgroup="dom0"

echo -n "VM name:"
read vmname
if echo "$vmname" | grep "[^a-z0-9_]"; then
  echo "Name can only contain alphanumeric chars and underscore"
  exit 1
fi
echo -n "/ filesystem size in MB (2048):"
read root_size
[ -z "$root_size" ] && root_size=2048
echo -n "/var filesystem size in MB (512):"
read var_size
[ -z "$var_size" ] && var_size=512
echo -n "swap size in MB (512):"
read swap_size
[ -z "$swap_size" ] && swap_size=512

echo
echo "VM name:   $vmname"
echo "/ size:    $root_size"
echo "/var size: $var_size"
echo "swap size: $swap_size"
echo
echo "Press Enter to continue, or ctrl-C to abort"
read

set -e # abort on errors

lvcreate --size ${root_size}M --name ${vmname}_root $volgroup
lvcreate --size ${var_size}M --name ${vmname}_var $volgroup
lvcreate --size ${swap_size}M --name ${vmname}_swap $volgroup

mkswap /dev/$volgroup/${vmname}_swap
echo "Copying template..."
dd skip=63 count=4192902 if=/var/lib/xen/images/centos5_template.img \
  obs=4096K of=/dev/$volgroup/${vmname}_root
dd skip=4192965 count=1044225 if=/var/lib/xen/images/centos5_template.img \
  obs=4096K of=/dev/$volgroup/${vmname}_var

echo "Resizing..."
e2fsck -f /dev/$volgroup/${vmname}_root
resize2fs /dev/$volgroup/${vmname}_root
e2fsck -f /dev/$volgroup/${vmname}_var
resize2fs /dev/$volgroup/${vmname}_var

uuid="$(uuidgen)"
mac="$(echo $uuid | perl -ne 'print "$1:$2:$3" if /(\w\w)(\w\w)(\w\w)$/')"

cat <<EOF >/etc/xen/$vmname
name = "$vmname"
uuid = "$uuid"
maxmem = 512
memory = 256
vcpus = 1
bootloader = "/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vfb = [ ]
disk = [ "phy:/dev/$volgroup/${vmname}_root,xvda1,w",
         "phy:/dev/$volgroup/${vmname}_var,xvda2,w",
         "phy:/dev/$volgroup/${vmname}_swap,xvda3,w" ]
vif = [ "mac=00:16:3e:$mac,bridge=virbr0" ]
EOF

echo "Complete. Start using: xm create -c $vmname"

Step 7: Mounting the template image

If you wish to edit the template image before cloning it again, one way is simply to start the template VM, make your changes, then halt.

A quicker option is simply to mount the template VM image file. However this is slightly fiddly, due to the fact that it contains partitions. You need to calculate the offsets into the image when mounting them.

fdisk -ul /var/lib/xen/images/centos5_template.img
63 sectors: 63*512 = 32256 bytes
start offset 4192965 sectors: 4192965*512 = 2146798080 bytes

Use these values as follows:

mount -o loop,offset=32256 /var/lib/xen/images/centos5_template.img /mnt
mount -o loop,offset=2146798080 /var/lib/xen/images/centos5_template.img /mnt/var
# do some work
umount /mnt/var
umount /mnt

It's very important that the VM is not running when you do this. To make accidents less likely you may wish to move /etc/xen/centos5_template to some other directory. For example, you could create /etc/xen/disabled/ and move it there.

Future improvements

Link to Xen documentation.

Using offsets into a single partitioned disk image is icky. I would have preferred to pass three separate LVs to the installer, but I couldn't work out how to get anaconda to accept xvda1, xvda2, xvda3 as a valid "partitioned" disk. Perhaps this is possible using kickstart.