Virtually having fun
Several weeks ago, the people in charge of maintaining the Windows machines in my institute were desperate because of a series of virus outbreaks - Specially, as expected, in the public lab - but the whole network smell virulent. After seeing their desperation, I asked Rolman to help me come up with a solution. He suggested me to try replacing the Windows workstations by substituting local installations by a server having several virtual machines, all regenerated from a clean image every day, and exporting rdesktop sessions. He suggested using Xen for this, as it is the virtualization/paravirtualization solution until now best offered and supported by most Linux distributions (including, of course, RedHat, towards which he is biased, and Debian, towards I am… more than biased, even bent). So far, no hassle, right? Of course, I could just stay clear of this mess, as everything related to Windows is off my hands… But in October, we will be renewing ~150 antivirus licences. I want to save that money by giving a better solution, even if part of that money gets translated to a big server. Get the hardware But problems soon arose. The first issue was hardware. Xen can act in its paravirtualization mode on basically any x86 machine - but it requires a patched guest kernel. That means, I can paravitualize many several different free OSs on just any computer I lay my hands on here, but Windows requires full- or hardware-assisted- virtualization. And, of course, only one of the over 300 computers we have (around 100 of which are recent enough for me to expect to be usable as a proof-of-concept for this) has a CPU with VT extensions - And I’m not going to de-comission my firewall to become a test server! ;-) When software gets confused for hardware So, I requested a Intel® Core™2 Quad Q9300 CPU, which I could just drop in any box with a fitting motherboard. But, of course, I’m not the only person requiring computer-related stuff. So, after pestering the people in charge for buying stuff on a daily basis for three weeks, the head of acquisitions came smiling to my office with a little box in his hands. But no, it was not my Core 2 Quad CPU. It was a box containing… Microsoft Visio. Yes, they spent their effort looking for the wrong computer-related thingy :-/ And meanwhile, Debconf 8 is getting nearer and nearer. Why does that matter? Because I have a deadline: By October, I want the institute to decide not to buy 150 antivirus licenses! Debconf will take some time off that target from me. Anyway… The university vacations started on July 5. The first week of vacations I went to sweat my ass off at Monterrey, by Monday 14 I came back to my office, and that same day I finally got the box, together with two 2GB DIMMs. Experiences with a nice looking potential disaster Anyway, by Tuesday I got the CPU running, and a regular Debian install in place. A very nice workhorse: 5GB RAM, quad core CPU at 2.5GHz, 6MB cache (which seems to be split in two 3MB banks, each for two cores - but that’s pure speculation from me). I installed Lenny (Debian testing), which is very soon going to freeze and by the time this becomes a production server will be very close to being a stable release, and I wanted to take advantage of the newest Xen administration tools. Of course, the installation was for AMD64 - Because 64 bitness is a terrible thing to waste. But I started playing with Xen - And all kind of disasters stroke. First, although there is a Xen-enabled 2.6.25 Linux kernel, it is -686 only (i.e. no 64 bit support). Ok, install a second system on a second partition. Oh, but this kernel is only domU-able (this is, it will correctly run in a Xen paravirtualized host), but not dom0-able (it cannot act as a root domain). Grmbl. So, get Etch’s 2.6.18 AMD64 Xen-enabled kernel, and hope for the best. After all, up to this point, I was basically aware of many of the facts I mentioned (i.e. up to this point I did reinstall once, but not three times)… And I hoped the kernel team would have good news regarding a forward-port of the Xen dom0 patches to 2.6.25 - because losing dom0 support was IMO a big regression. But quite on time, this revealing thread came up on the debian-devel mailing list. In short: Xen is a disaster. The Xen developers have done their work quite far away from the kernel developers, and the last decent synchronization that was made was in 2.6.18, over two years ago. Not surprisingly, enterprise-editions of other Linux distributions also ship that kernel version. There are some forward-patches, but current support in Xen is… Lacking, to say the least. From my POV, Xen’s future in the Linux kernel looks bleakish. Now, on the lightweight side… Xen is also a bit too complicated - Of course, its role is also complicated as well, and it has a great deal of tunability. But I decided to keep a clean Lenny AMD64 install, and give KVM, the Kernel Virtual Machine a go. My first gripe? What a bad choice of name. Not only Google searches for KVM gives completely unrelated answers (to a name that’s already well known, even in the same context, even in the same community). KVM takes a much, much simpler approach to virtualization (both para- and full-): We don’t need no stinkin’ hypervisors. The kernel can just do that task. And then, kvm becomes just another almost-regular process. How nice! In fact, KVM borrows so very much from qemu that it even refers to qemu’s manpage for everything but two command-line switches. Qemu is a completely different project, which gets to a very similar place but from the other extreme - Qemu started off as Bochs, a very slow but very useful multi-architecture emulator. Qemu started adding all kinds of optimizations, and it is nearly useful (i.e. I use it in my desktop whenever I need a W2K machine). Instead of a heavyweight framework… KVM is just a modprobe away - Just ask Linux to modprobe kvm, and kvm -hda /path/to/your/hd/image gets you a working machine. Anyway - I was immediatly happy with KVM. It took me a week to get a whole “lab” of 15 virtual computers (256MB RAM works surprisingly well for a regular XP install!) configured to start at boot time off a single master image over qcow images. KVM’s shortcomings Xen has already been a long time in the enterprise, and has a nice suite of administrative tools. While Xen depends on having a configuration file for each host, KVM expects them to be passed at the command line. To get a bird-eye view of the system, xen has a load of utilities - KVM does not. And although RedHat’s virt-manager is said to support KVM and qemu virtualization (besides its native Xen, of course), it falls short of what I need (i.e. it relies on a configuration file… which lacks expresivity to specify a snapshot-based HD image). To my surprise, KVM has attained much of Xen’s most amazing capabilities, such as the live migration. And although it’s easier to just use fully virtualized devices (i.e. to use an emulation of the RTL8139 network card), as they require no drivers extraneous to the operating system, performance can be greatly enhanced by using the VirtIO devices. KVM is quickly evolving, and I predict it will largely overtake Xen’s (and of course, vmware and others) places. Where I am now So… Well, those of us that adopt KVM and want to get it into production now will have some work of building the tools to gracefully manage and report it, it seems. I won’t be touching much my setup until after Debconf, but so far I’ve done some work over Freddie Cash’s kvmctl script. I’m submitting him some patches to make his script (IMHO) more reliable and automatizable (if you are interested, you can get my current version of the script as well). And… Starting September, I expect to start working on a control interface able to cover my other needs (such as distributing configuration to the terminals-to-be, or centrally managing the configurations).
Comments
a. Anonymous 2008-07-23 17:23:52
hard work! congratulations
hard work! congratulations and…. keep on moving. and good luck! see u soon
gwolf 2008-07-24 09:58:08
Fixed
…Thanks :-/
Jeff Schroeder 2008-07-24 10:54:55
kvm management
The project is still too new to throw into production, but you should keep an eye on ovirt (http://www.ovirt.org).
jescudero 2008-08-25 10:14:48
Otra solución
Perdona pero en español me expreso mucho mejor y creo que me entenderás por lo que he visto en tu web:
Si las computadoras son parecidas (exactamente iguales) probaría a clonarlas, creo que es una solución más sencilla que la de las máquinas virtuales.
Yo he usado partimage con buenos resultados clonando windows y linux.
Más soluciones: AVG tiene un antivirus que se instala en linux y escanea las particiones windows, mano de santo.
Aaaaaaaadios.
Letic 2008-07-24 02:42:29
KVM Interface
Hey Gunnar,
If it can help you I found a very nice web GUI for openvz/kvm that is very intuitive and quite well developed. They provide all the tools, including customized kernel, in debian packages. We are currently testing it on our servers and it work very well.
Give it a go :o) Hope this helps LeTic
Mauro 2008-07-24 10:06:48
At the same spot
I’m currently at the same situation at work. We need to begin virtualizing some of our servers, preferrably with Debian, but after a while trying with Xen OSE, VMware, OpenVZ, Linux-VServer and KVM, we choose the last because of it’s simplicity and easy deployment, and because the current state and worrying future of Xen support in Debian and other distributions.
I’m still running some tests and benchmarks using paravirtualization and LVM, with quite reasonable results, which I expect to publish soon.
IMHO, simplicity is one of the best features of KVM, and as you say, pairing KVM with virtio will bring a good and needed performance improvement.
If everything goes well, I’ll attend DebConf this year. So maybe we can find some time to share experiences.
Saludos,
Roadmaster 2008-07-25 11:42:00
Fac. de Contaduria
Hey, I’ve also been playing with Xen at the fac. de contaduria, so far we have 4 xen hosts running real-world workloads, although all of them are paravirtualized linuxen; i’ve had limited success virtualizing Windows with HVM and VT-X support but who’d want to use that for production :)
I might add that, even though I was indeed forced to downgrade my lennies to 2.6.18 kernels, everything else is up-to-date and working nicely and at least two of our virtualized servers are mission-critical and have worked flawlessly since day one.
Also, keep an eye on what the Linode.com guys are doing, they run a VPS service on Xen and AFAIK they have integrated Xen with more recent kernels so there might be a way to bring all that to Debian.
So a) feel free to give me a holler if you need any kind of help with Xen , I’m still a bit green but two newbies think better than one, and b) I’ll be looking into KVM for our virtualization needs (next week since I’ll also be back to work by then).
Steve Kemp 2008-07-24 08:30:56
Missing file?
The link to your improved version of the script is leading to a 404 - which is a shame.
vicm3 2008-07-23 18:24:17
Well for a public lab
I will go more for the LITRO, well not exactly the litro but a solution like that to say reinstall to disk every day… in fact that’s the objective of a project that caught my attention http://www.clonezilla.org/
From the faq http://drbl.sourceforge.net/faq/index.php#path=./1_Common&entry=00_what_is_nchc.faq (a) Clonezilla, the opensource clone system. It’s a server version of imaging tool, similar to Ghost server edition, True image or Rembo. By using Clonezilla, you can clone a 5.6 GBytes system image to 40 computers within 10 minutes via multicasting.
I’m not in charge of the computer labs at my office but I wish to go for a solution like that, closed sourced software exist that create an abstraction layer valid for a session if not for the outrageus price maybe I will like this solution anyway I need to take a look to vserver and KVM where you have avanced a lot in little time.