Start KVM virtual machines from the console with virsh

Here’s what I see when I run virsh and type list:

# virsh
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
   'quit' to quit

virsh # list
 Id Name                 State
----------------------------------

I’m in the middle of working from the command line in a remote ssh session on a KVM virtual machine host that runs two virtual machines. For reasons unclear at this time, I can’t list domains, interfaces, volumes, or anything from virsh on the host. However, I can still start and stop virtual machines and flag them for autostart and all I need to do right now is start a vm.
kvm logo with tux juggling
Nothing there. I was expecting to see the same two virtual machines by name that I see from virt-manager. But i’m stuck on a console with no display available right now. virt-manager is not an option. I know there are other vm managers available that may be browser based or whatever, but I don’t have them installed.

Although virsh won’t let me query much information, I do know the names of the machines I wish to start and I have a simple ssh shell on the host machine. So lets give it a shot.

dontpanic@host:~# virsh autostart vm1
Domain vm1 marked as autostarted

dontpanic@host:~# virsh start vm1
Domain vm1 started

Everything went better than expected. So if you’re used to working in a virtual machine manager GUI but for whatever reason you can’t/won’t, give the virtual shell “virsh” a try.

If you’re on a debian based system, virsh is found in the libvirt-bin package. If you’re on an rpm based system, it’s in libvirt-client.

KVM live migration with sanlock using virsh

Sanlock helps you avoid screwing things up by starting the same virtual machine on multiple hosts, which will quickly mangle the guest file systems. But you may quickly find KVM live migration no longer works from the virt-manager GUI.

I was in shock when I first saw this migration error from virt-manager!

virt manager error dialog

Internal error cannot use migrate v2 protocol with lock manager sanlock? But alas, all is not lost.

Migration from the virsh command shell works just fine for me. And it’s quicker and more efficient than doing it pointy-clicky style from a gui anyway. If i’m migrating a guest from one host to another, it’s usually because I need to shut down the host hardware. I’m not migrating just one guest, i’m migrating one at a time.

With the shell method, I repeat the migration command in a loop and move them all, one after another automatically. Monday migrate, tuesday migrate, everybody. It’ll get in your bones!

root@host1~# for x in 1 2 3 4 5 6 7 8; do
virsh migrate –live guest$x qemu+ssh://host2/system
done

When I first set up sanlock for qemu locking, I saw errors during migration of several hosts that were already up and running with local locking before sanlock locking was implemented.

error: Failed to inquire lock: No such process

There is no impact to the guest or for users connected to the guest. The end result is the migration doesn’t complete and I found the guest still running on the original host. Other guests migrated without error.

A scheduled reboot of the guests allowed migration to proceed. No reboot or disruption to the existing host was required at all.

VM Guest as a NTPD Time Source

blue analog clockIf all I wanted to do was fix the clock on a VM guest, I would have stopped running ntp on the guest and left it running on the underlying physical host. Simple.

But what if you want to use a VM guest as a ntpd time source?! Reasons might be because you’re migrating from a physical server to a VM and don’t have access to the guests to redirect them to another host for whatever reason.

I know this problem is caused in different ways for each virtualization platform. And while my specific problem was using KVM, this should avoid the problem on all of them.

If your host system uses Time Stamp Counter (TSC)

# grep -m1 constant_tsc /proc/cpuinfo
flags  : fpu vme de pse tsc blah blah blah...
constant_tsc
# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

and Virtual Machine guests use kvm-clock

# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm-clock
# dmesg | grep clock
...
[    0.056001] kvm-clock: cpu 11, msr 0:11975701, secondary cpu clock
[    0.388036] Switching to clocksource kvm-clock
[    0.637349] rtc_cmos 00:01: setting system clock to 2012-01-21 19:18:21 UTC

Do not use ntp on the guest!

But what if you must use a guest as a time source? If the physical host synchronizes to a good Internet time source and the VM guest uses itself (127.0.0.1), what would happen? Well it would still be fighting the clock. So that’s not going to work. I was just hoping to avoid spike messages showing up in client ntp logs, but the clock can skew drastically in either direction.

iptables MASQUERADE to the rescue!

Stop running ntpd on the guest and forward all ntp requests that come in to a physical host serving NTP. You need two rules minimum:

iptables -t nat -A PREROUTING -i eth0 -p udp -m udp --dport 123 -j DNAT --to-destination $HOST
iptables -t nat -A POSTROUTING -o eth0 -p udp -m udp --dport 123 -j MASQUERADE

If you have two interfaces, you can forward the traffic from one network to the other this way too. Just change the -i eth0 to match the other network interface and then allow forwarding:

sysctl net.ipv4.conf.eth0.forwarding=1
sysctl net.ipv4.conf.eth1.forwarding=1

Even if you only have one interface and the ntp server is on the same network, the masquerade should still work.

You should really limit forwarding to ntp for your source and destination too. Default policies of ACCEPT for iptables are bad if you don’t have an explicit rule to drop everything not handled by a higher rule.

# forwarding for ntp requests to your ntp server
iptables -A FORWARD -p udp -d 10.9.8.7/32 --dport 123 -j ACCEPT
# forwarding for responses from your ntp server
iptables -A FORWARD -p udp -d 192.168.1.0/24 --dport 123 -j ACCEPT

Install VirtualBox Guest Additions on Linux Mint

I used to like Ubuntu, before they simultaneously dumbed down the user interface and complicated the administration. The current direction of Ubuntu and Fedora is just plain bad, but that’s a story for another day. Let’s look at our wonderful Ubuntu replacement, Linux Mint.
green lxde logo
When installing Mint, you get a choice of desktop environments. LXDE is fast, lightweight, and uses GTK, which I’m already very familiar with. It’s not bloated like Gnome. It’s everything Gnome wishes it was. If you use Gnome, stop right now and do yourself a favor, install LXDE. Even if you’re hooked on another distribution, there are usually alternative versions available such as Fedora’s LXDE spin. Or just go through package installation of .deb or .rpm files using your favorite package manager. You’ll never look back.

The only problem i’ve had with LXDE and Mint is that running as a guest in VirtualBox, the guest additions will not install properly. I tried mounting the additions from the host and installing them, no dice. I tried installing from the software repositories using the aptitude update manager, no dice.

After attempting an installation from either method, open a terminal and go to the source directory that you just installed. Run make and make install. Voila! Now restart the guest and enjoy the seamless mouse and keyboard integration.

So it didn’t work out of the box for me, but a quick recompile did the trick. No moving around files or manipulating configurations are necessary, just recompile for the running kernel and you’re in business.

VirtualBox can’t operate in VMX root mode

You might see this VirtualBox error when trying to start a virtual machine or create a new one.

VirtualBox can’t operate in VMX root mode. Please disable the KVM kernel extension, recompile your kernel and reboot.
VBox status code: -4011 (VERR_VMX_IN_VMX_ROOT_MODE).

Result Code:
0×80004005
Component:
Console
Interface:
IConsole {d5a1cbda-f5d7-4824-9afe-d640c94c7dcf}

But it’s an easy fix.

$ modprobe -l | grep kvm
kernel/arch/x86/kvm/kvm.ko
kernel/arch/x86/kvm/kvm-intel.ko
kernel/arch/x86/kvm/kvm-amd.ko

Remove the modules.

$ sudo modprobe-r kvm-amd
$ sudo modprobe-r kvm-intel
$ sudo modprobe-r kvm

Remove them in that same order or else you’ll get this error:
FATAL: Module kvm is in use.

VirtualBox Black Logo BoxThe modules will reload when you reboot and you’ll probably forget about this little problem… try to run a VM… Virtualbox will complain, and then you’ll have to remove the modules all over again.

You could blacklist the modules so they don’t load automatically, or stop using the generic distribution kernel and compile your own, or go back to a pre 2.6.20 kernel since KVM was first added to 2.6.20 in February 2007. But more of a pain than it’s worth for a generic desktop system running a Fedora, Ubuntu, Debian, Arch, etc. distribution.

$ cd /etc/modprobe.d/
$ echo “kvm-amd” >> ./blacklist.conf
$ echo “kvm-intel” >> ./blacklist.conf
$ echo “kvm” >> ./blacklist.conf