nico
/
www.nico.schottelius.org
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
www.nico.schottelius.org/blog/kvm-vms-with-cdist-at-local...

352 lines
12 KiB
Markdown
Raw Permalink Blame History

[[!meta title="KVM Virtual Machines managed with cdist and sexy @ local.ch"]]
## Introduction
This article describes the KVM setup of [local.ch](http://www.local.ch), which is
managed by [[sexy|software/sexy]] and configured by [[cdist/software/cdist]].
If you haven't so far, you may want to have a look at the
[[Sexy and cdist @ local.ch|sexy-and-cdist-at-local.ch]]
article before continuing to read this one.
## KVM Host configuration
The KVM hosts are Dell R815 with CentOS 6.x installed. Why Dell? Because they
offered a good price/value combination. Why CentOS? Historical
reasons. The hosts got a minimal set of BIOS tuning to support the VM performance:
* Enable the usual virtualisation flags (don't forget to enable the IOMMU!)
* Change the power profile to **Maximum Perforamnce**
Furthermore, as the CentOS kernel is pretty old (2.6.32-279) and
conservatively configured, the kernel needs the following
command line option to enable the IOMMU:
amd_iommu=on
Not enabling this option degrades the performance.
In our case, enabling it reduced the latency of the
application running in the VM by a factor of 10.
One big design consideration of the the KVM setup at local.ch is to make the
KVM hosts as independent as possible and sensibly fault tolerant. That said,
VMs are stored on local storage and hosts are always redundantly connected
to two switches use [LACP](https://en.wikipedia.org/wiki/Link_aggregation).
## KVM Host Network Configuration
[[!img kvm-setup-local.ch-overview.png alt="Overview of KVM setup at local.ch"]]
As can be seen in the picture above, every KVM host is connected to two
**10G Arista switches (7050T-52-R)** using LACP. Besides being capable
of running 10G, the Arista switches are actually pretty neat for the Unix geek,
because they are Linux based with a
[FPGA](https://en.wikipedia.org/wiki/Field-programmable_gate_array)
attached. Furthermore you can easily
gain access to a shell by typing **enable** followed by **bash**.
The Arista switches are connected together with 2x 10G links, over which LACP+MLAG
is configured. This gives us the ability to connect every KVM host with LACP to two
**different** switches: They use MLAG to synchronise their LACP states.
On the KVM host, the network is configured as follows:
The dual Port 10G card (Intel Corporation 82599EB) is bonded together into bond0.
[root@kvm-hw-inx01 network-scripts]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
Aggregator ID: 3
Number of ports: 2
Actor Key: 33
Partner Key: 30
Partner Mac Address: 02:1c:73:1b:f5:b2
Slave Interface: eth4
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 68:05:ca:0b:5b:6a
Aggregator ID: 3
Slave queue ID: 0
Slave Interface: eth5
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 68:05:ca:0b:5b:6b
Aggregator ID: 3
Slave queue ID: 0
The following configuration is used to create the bond0 device:
[root@kvm-hw-inx01 network-scripts]# cat ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
BONDING_OPTS="mode=802.3ad"
ONBOOT=yes
MTU=9000
[root@kvm-hw-inx01 sysconfig]# cat network-scripts/ifcfg-eth4
DEVICE="eth4"
NM_CONTROLLED="yes"
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
[root@kvm-hw-inx01 sysconfig]# cat network-scripts/ifcfg-eth5
DEVICE="eth5"
NM_CONTROLLED="yes"
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
The MTU of the 10G cards has been set to 9000, as the Arista switches support
[Jumbo Frames](https://en.wikipedia.org/wiki/Jumbo_frame).
Every VM is attached to two different networks:
* PZ: presentation zone (for general traffic) (10.18x.0.0/22 network)
* FZ: filer zone (for NFS and database traffic) (10.18x.64.0/22 network)
Both networks are seperated using the VLAN tags 2 (pz) and 3 (fz), which result
in **bond0.2** and **bond0.3**:
[root@kvm-hw-inx01 network-scripts]# ip l | grep bond
6: eth4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000
7: eth5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP
139: bond0.2@bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP
140: bond0.3@bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP
To keep things simple, the two vlan tagged (bonded) interfaces are added to a bridge each,
to which the VMs are attached later on. The configuration looks like this:
[root@kvm-hw-inx01 network-scripts]# cat ifcfg-bond0.2
DEVICE="bond0.2"
ONBOOT=yes
VLAN=yes
BRIDGE=brpz
[root@kvm-hw-inx01 network-scripts]# cat ifcfg-brpz
DEVICE=brpz
TYPE=Bridge
ONBOOT=yes
DELAY=0
NM_CONTROLLED=no
MTU=9000
This is how a bridge looks like in production (with about 70 lines stripped):
[root@kvm-hw-inx01 network-scripts]# brctl show
bridge name bridge id STP enabled interfaces
brfz 8000.024db29ca91f no bond0.3
tap13
tap73
[...]
brpz 8000.02f6742800b2 no bond0.2
tap0
tap1
[...]
Summarised, the network configuration of a KVM host looks like this:
arista1 arista2
| |
[eth4 + eth5] -> bond0
|
|
/ \
bond0.2 bond0.3
/ \
brpz brfz
\ /
tap1 tap2
\ /
VM
## VM configuration
The VM configuration can be found below **/opt/local.ch/sys/kvm**
on every KVM host. Every VM is stored below
**/opt/local.ch/sys/kvm/vm/<vm name>** and contains the following
files:
[root@kvm-hw-inx03 jira-vm-inx01.intra.local.ch]# ls
monitor pid start start-on-boot system-disk vnc
* monitor: socket to the monitor from KVM
* pid: the pid of the VM
* start: the script to start the VM (see below for an example)
* start-on-boot: if this file exists, the VM will be started on boot
* system-disk: the qcow2 image of the system disk
* vnc: socket to the screen of the VM
With the exception of monitor, pid and vnc are all files generated by cdist.
The start script of a VM looks like this:
[root@kvm-hw-inx03 jira-vm-inx01.intra.local.ch]# cat start
#!/bin/sh
# Generated shell script - do not modify
#
/usr/libexec/qemu-kvm \
-name jira-vm-inx01.intra.local.ch \
-enable-kvm \
-m 8192 \
-drive file=/opt/local.ch/sys/kvm/vm/jira-vm-inx01.intra.local.ch/system-disk,if=virtio \
-vnc unix:/opt/local.ch/sys/kvm/vm/jira-vm-inx01.intra.local.ch/vnc \
-cpu host \
-boot order=nc \
-pidfile "/opt/local.ch/sys/kvm/vm/jira-vm-inx01.intra.local.ch/pid" \
-monitor "unix:/opt/local.ch/sys/kvm/vm/jira-vm-inx01.intra.local.ch/monitor,server,nowait" \
-net nic,macaddr=00:16:3e:02:00:ab,model=virtio,vlan=200 \
-net tap,script=/opt/local.ch/sys/kvm/bin/ifup-pz,downscript=/opt/local.ch/sys/kvm/bin/ifdown,vlan=200 \
-net nic,macaddr=00:16:3e:02:00:ac,model=virtio,vlan=300 \
-net tap,script=/opt/local.ch/sys/kvm/bin/ifup-fz,downscript=/opt/local.ch/sys/kvm/bin/ifdown,vlan=300 \
-smp 4
Most parameter values depend on output of sexy,
which uses the cdist type **__localch_kvm_vm**,
which in turn assembles this start script.
The above script may be useful for one or more of my readers,
as it includes a lot of tuning we have done to KVM.
## Automatic startup of VMs
The virtual machines are brought up by an init script located at
***/etc/init.d/kvm-vms***. As every VM contains its own startup script
and is marked whether it should be started at boot, the init script
is pretty simple:
basedir=/opt/local.ch/sys/kvm/vm
broken_lock_file_for_centos=/var/lock/subsys/kvm-vms
case "$1" in
start)
cd "$basedir"
# Specific VM given
if [ "$2" ]; then
vm_list=$2
else
vm_list=$(ls)
fi
for vm in $vm_list; do
vm_base_dir="$basedir/$vm"
start_script="$vm_base_dir/start"
# Skip start of machines which should not start
if [ ! -f "$vm/start-on-boot" ]; then
continue
fi
echo "Starting VM $vm ..."
logger -t kvm-vms "Starting VM $vm ..."
screen -d -m -S "$vm" "$start_script"
done
touch "$broken_lock_file_for_centos"
;;
As you can see, every VM is started in its own
[screen](http://www.gnu.org/software/screen/) - so if screen decides to
hang up, only one VM is affected.
Furthermore screen supports only a limited number of windows it can server.
The process listing for a running virtual machine looks like this:
root 64611 0.0 0.0 118840 852 ? Ss Mar11 0:00 SCREEN -d -m -S binarypool-vm-inx02.intra.local.ch /opt/local.ch/sys/kvm/vm/binarypool-vm-inx02.intra.local.ch/start
root 64613 0.0 0.0 106092 1180 pts/22 Ss+ Mar11 0:00 /bin/sh /opt/local.ch/sys/kvm/vm/binarypool-vm-inx02.intra.local.ch/start
root 64614 2.9 2.2 9106828 5819748 pts/22 Sl+ Mar11 5221:41 /usr/libexec/qemu-kvm -name binarypool-vm-inx02.intra.local.ch -enable-kvm -m 8192 -drive file=/opt/local.ch/sys/kvm/vm/binarypool-vm-inx02.intra.local.ch/system-disk,if=virtio -vnc unix:/opt/local.ch/sys/kvm/vm/binarypool-vm-inx02.intra.local.ch/vnc -cpu host -boot order=nc -pidfile /opt/local.ch/sys/kvm/vm/binarypool-vm-inx02.intra.local.ch/pid -monitor unix:/opt/local.ch/sys/kvm/vm/binarypool-vm-inx02.intra.local.ch/monitor,server,nowait -net nic,macaddr=00:16:3e:02:00:7f,model=virtio,vlan=200 -net tap,script=/opt/local.ch/sys/kvm/bin/ifup-pz,downscript=/opt/local.ch/sys/kvm/bin/ifdown,vlan=200 -net nic,macaddr=00:16:3e:02:00:80,model=virtio,vlan=300 -net tap,script=/opt/local.ch/sys/kvm/bin/ifup-fz,downscript=/opt/local.ch/sys/kvm/bin/ifdown,vlan=300 -smp 4
## Common Tasks
The following sections show you how to do regular maintenance
tasks on the KVM infrastructure.
### Create a VM
VMs can easily be created using the script **vm/create-vm** from the sysadmin-logs repository
(local.ch internally), which looks like this:
sexy host add --type vm $fqdn
sexy host vm-host-set --vm-host $vmhost $fqdn
sexy host disk-add --size $disksize $fqdn
sexy host memory-set --memory $memory $fqdn
sexy host cores-set --cores $cores $fqdn
mac_pz=$(sexy mac generate)
mac_fz=$(sexy mac generate)
sexy host nic-add $fqdn -m $mac_pz -n pz
sexy host nic-add $fqdn -m $mac_fz -n fz
sexy net-ipv4 host-add "$net_pz" -m "$mac_pz" -f "$fqdn"
sexy net-ipv4 host-add "$net_fz" -m "$mac_fz" -f "$fz_fqdn"
echo "Updating git / github ..."
cd ~/.sexy
git add db
git commit -m "Added host $fqdn"
git pull
git push
# Apply changes: first network, so dhcp & dns are ok, then create VM
cat << eof
Todo for apply:
sexy net-ipv4 apply --all
sexy host apply --all
Start VM on $vmhost: ssh $vmhost /opt/local.ch/sys/kvm/vm/$fqdn/start
eof
### Delete a VM
Run the script **remove-host**, which essentially does the following:
* Remove various monitoring / backup configurations
* Detect if it is a VM, if so
* Stop it
* Remove it from the host
* Add mac address to the list of free mac addresses
* Delete host from the networks
* Delete host from sexy database
### Move VM to another server
To move one VM to another host, the following steps are necessary:
* sexy host vm-host-set ... # to new host
* stop vm
* scp/rsync directory from old host to new host
* sexy host apply --all # record db change
* start vm on new host
[[!tag cdist localch net sexy unix]]
Reference in New Issue View Git Blame Copy Permalink