Blog

2011
- 12
- 11
- 10
- 09
- 08
- 07
- 06
- 05
- 04
- 03
- 02
- 01

2010
- 12
- 11
- 10
- 09
- 08
- 07
- 06
- 05
- 04
- 03
- 02
- 01

2008
- 12
- 11
- 10
- 09
- 08
- 07
- 06
- 05
- 04
- 03
- 02
- 01

2007
- 12
- 11
- 10
- 09
- 08
- 07
- 06
- 05
- 04
- 03
- 02
- 01

2006
- 12
- 11
- 10
- 09
- 08
- 07
- 06
- 05
- 04
- 03
- 02
- 01

2005
- 12
- 11
- 10
- 09
- 08
- 07
- 06
- 05
- 04
- 03
- 02
- 01

Posted 2007/01/23

Installing Xen on Lister (Dell Poweredge 2950)

This machine was bought to provide a mechanism to quickly deploy Linux instances with different software configurations for testing purposes (in our experience many middleware toolkits don’t play well with others). This post is a rough guide to installing Xen on the box, mainly for our own documentation purposes but maybe it will help someone. The hardware is a Dell Poweredge 2950, all the hardware in this box works with a Linus kernel. The base install was Debian Etch.

Hardware

The basic chassis is a Dell Poweredge 2950. We opted for 4GB of RAM, 2 dual core Xeons and 6 x 300GB SAS HDs. Because we had heard of problems with the Broadcom Nics with Xen we bought an additional Intel network card. A picture of the guts is online here.

00:00.0 Host bridge: Intel Corporation 5000X Chipset Memory Controller Hub (rev 12) 00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 2 (rev 12) 00:03.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 3 (rev 12) 00:04.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 4-5 (rev 12) 00:05.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 5 (rev 12) 00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 6-7 (rev 12) 00:07.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 7 (rev 12) 00:10.0 Host bridge: Intel Corporation 5000 Series Chipset Error Reporting Registers (rev 12) 00:10.1 Host bridge: Intel Corporation 5000 Series Chipset Error Reporting Registers (rev 12) 00:10.2 Host bridge: Intel Corporation 5000 Series Chipset Error Reporting Registers (rev 12) 00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev 12) 00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev 12) 00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev 12) 00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev 12) 00:1c.0 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 1 (rev 09) 00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09) 00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2 (rev 09) 00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 (rev 09) 00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9) 00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller (rev 09) 00:1f.1 IDE interface: Intel Corporation 631xESB/632xESB IDE Controller (rev 09) 01:00.0 PCI bridge: Intel Corporation 80333 Segment-A PCI Express-to-PCI Express Bridge 01:00.2 PCI bridge: Intel Corporation 80333 Segment-B PCI Express-to-PCI Express Bridge 02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID controller 5 04:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c3) 05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12) 06:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01) 06:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge (rev 01) 07:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1 (rev 01) 07:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E2 (rev 01) 08:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c3) 09:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12) 0a:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (rev 06) 10:0d.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)

Installing Xen and Dom0 Kernel

XEN_TARGET_X86_PAE=y is important if you have >4GB RAM. First we make all.

make XEN_TARGET_X86_PAE=y world make install

Now configure your own version of the Dom0 kernel to match your hardware.

make XEN_TARGET_X86_PAE=y linux-2.6-xen0-config CONFIGMODE=menuconfig KERNELS="linux-2.6-xen0" make XEN_TARGET_X86_PAE=y -j5 linux-2.6-xen0-build make linux-2.6-xen0-install depmod 2.6.16.33-xen0

Edit /boot/grub/menu.lst for Xen.

# Xen kernel title Xen 3.0.4 / XenLinux 2.6 root (hd0,0) kernel /xen.gz dom0_mem=262144 module /vmlinuz-2.6-xen0 root=/dev/sda5 ro max_loop=255 # Linus Kernel title Kernel.org 2.6.16.33 root (hd0,0) kernel /vmlinuz-2.6.16.33 root=/dev/sda5 ro

Reboot into dom0.

Setting Up a DomU

First build the domU kernel, I did not use any modules to make life a bit easier when managing the doms.

make linux-2.6-xenU-config CONFIGMODE=menuconfig KERNELS="linux-2.6-xenU" make XEN_TARGET_X86_PAE=y -j5 linux-2.6-xenU-build make linux-2.6-xenU-install

Make a virtual disk to install to.

pvcreate /dev/sda3 vgcreate vg /dev/sda3 lvcreate -L4096M -n master_debian-etch vg mkfs -t ext3 /dev/vg/master_debian-etch

Bootstrap Debian Etch.

mount /dev/vg/master_debian-etch /mnt/tmp debootstrap --arch i386 etch /mnt/tmp/ http://debian.blueyonder.co.uk/ cat /etc/apt/sources.list > /mnt/tmp/etc/apt/sources.list cp -rp /lib/modules/2.6.16.33-xenU /mnt/tmp/lib/modules/ chroot /mnt/tmp/ /bin/sh apt-get update apt-get install localeconf locales ssh vim dnsutils sudo telnet bzip2 make gcc less nmap g++ dpkg-reconfigure locales rm /var/cache/apt/archives/* mv /lib/tls/ /lib/tls.disabled

Edit /etc/fstab.

/dev/hda1 / ext3 defaults 1 2 /dev/hda2 none swap sw 0 0 none /dev/shm tmpfs defaults 0 0

Edit /etc/network/interfaces. I don’t know why it is eth1 instead of eth0 in this case. The thing to do though is ifconfig -a as this lists all available interfaces.

# loop auto lo iface lo inet loopback # public auto eth1 iface eth1 inet dhcp

Edit /etc/hosts.

127.0.0.1 localhost.localdomain localhost

Change both /etc/init.d/hwclock.sh and /etc/init.d/hwclockfirst.sh to ‘exit 0′ when they run. Then snapshot the file system to save a copy of this vanilla Etch install. By using snapshots this basic install can be reused by multiple doms, it is important when creating the clones that you make sure they wont run out of room or ‘bad things can happen’. In this case an additional 4GB has been allocated.

umount /mnt/tmp lvcreate -s -L4096M -n clone_debian-etch_dom01 /dev/vg/master_debian-etch lvcreate -L1024M -n swap_dom01 vg mkswap /dev/vg/swap_dom01

Last thing is to write a config file to start the dom. In this case the dom is restricted to 512MB RAM and is forced to not use the first CPU core (being reserved for dom0).

name="dom01" kernel="/boot/vmlinuz-2.6-xenU" root="/dev/hda1" memory=512 disk=['phy:/dev/vg/clone_debian-etch_dom01,hda1,w','phy:/dev/vg/swap_dom01,hda2,w'] vcpus=3 cpus="^0" vif=[ 'mac=00:16:3e:00:01:01, bridge=xenbr0' ] dhcp="on"

Start the dom with ‘xm create -c /etc/xen/dom01-confic.sxp’. That’s it for this how to.