Virtually having fun

Submitted by gwolf on Wed, 07/23/2008 - 18:50.

Several weeks ago, the people in charge of maintaining the Windows machines in my institute were desperate because of a series of virus outbreaks - Specially, as expected, in the public lab - but the whole network smell virulent. After seeing their desperation, I asked Rolman to help me come up with a solution. He suggested me to try replacing the Windows workstations by substituting local installations by a server having several virtual machines, all regenerated from a clean image every day, and exporting rdesktop sessions. He suggested using Xen for this, as it is the virtualization/paravirtualization solution until now best offered and supported by most Linux distributions (including, of course, RedHat, towards which he is biased, and Debian, towards I am... more than biased, even bent). So far, no hassle, right?
Of course, I could just stay clear of this mess, as everything related to Windows is off my hands... But in October, we will be renewing ~150 antivirus licences. I want to save that money by giving a better solution, even if part of that money gets translated to a big server.
Get the hardware
But problems soon arose. The first issue was hardware. Xen can act in its paravirtualization mode on basically any x86 machine - but it requires a patched guest kernel. That means, I can paravitualize many several different free OSs on just any computer I lay my hands on here, but Windows requires full- or hardware-assisted- virtualization. And, of course, only one of the over 300 computers we have (around 100 of which are recent enough for me to expect to be usable as a proof-of-concept for this) has a CPU with VT extensions - And I'm not going to de-comission my firewall to become a test server! ;-)
When software gets confused for hardware
So, I requested a Intel® Core™2 Quad Q9300 CPU, which I could just drop in any box with a fitting motherboard. But, of course, I'm not the only person requiring computer-related stuff. So, after pestering the people in charge for buying stuff on a daily basis for three weeks, the head of acquisitions came smiling to my office with a little box in his hands.
But no, it was not my Core 2 Quad CPU.
It was a box containing... Microsoft Visio. Yes, they spent their effort looking for the wrong computer-related thingy :-/ And meanwhile, Debconf 8 is getting nearer and nearer. Why does that matter? Because I have a deadline: By October, I want the institute to decide not to buy 150 antivirus licenses! Debconf will take some time off that target from me.
Anyway... The university vacations started on July 5. The first week of vacations I went to sweat my ass off at Monterrey, by Monday 14 I came back to my office, and that same day I finally got the box, together with two 2GB DIMMs.
Experiences with a nice looking potential disaster
Anyway, by Tuesday I got the CPU running, and a regular Debian install in place. A very nice workhorse: 5GB RAM, quad core CPU at 2.5GHz, 6MB cache (which seems to be split in two 3MB banks, each for two cores - but that's pure speculation from me). I installed Lenny (Debian testing), which is very soon going to freeze and by the time this becomes a production server will be very close to being a stable release, and I wanted to take advantage of the newest Xen administration tools. Of course, the installation was for AMD64 - Because 64 bitness is a terrible thing to waste.
But I started playing with Xen - And all kind of disasters stroke. First, although there is a Xen-enabled 2.6.25 Linux kernel, it is -686 only (i.e. no 64 bit support). Ok, install a second system on a second partition. Oh, but this kernel is only domU-able (this is, it will correctly run in a Xen paravirtualized host), but not dom0-able (it cannot act as a root domain). Grmbl.
So, get Etch's 2.6.18 AMD64 Xen-enabled kernel, and hope for the best. After all, up to this point, I was basically aware of many of the facts I mentioned (i.e. up to this point I did reinstall once, but not three times)... And I hoped the kernel team would have good news regarding a forward-port of the Xen dom0 patches to 2.6.25 - because losing dom0 support was IMO a big regression.
But quite on time, this revealing thread came up on the debian-devel mailing list. In short: Xen is a disaster. The Xen developers have done their work quite far away from the kernel developers, and the last decent synchronization that was made was in 2.6.18, over two years ago. Not surprisingly, enterprise-editions of other Linux distributions also ship that kernel version. There are some forward-patches, but current support in Xen is... Lacking, to say the least. From my POV, Xen's future in the Linux kernel looks bleakish.
Now, on the lightweight side...
Xen is also a bit too complicated - Of course, its role is also complicated as well, and it has a great deal of tunability. But I decided to keep a clean Lenny AMD64 install, and give KVM, the Kernel Virtual Machine a go. My first gripe? What a bad choice of name. Not only Google searches for KVM gives completely unrelated answers (to a name that's already well known, even in the same context, even in the same community).
KVM takes a much, much simpler approach to virtualization (both para- and full-): We don't need no stinkin' hypervisors. The kernel can just do that task. And then, kvm becomes just another almost-regular process. How nice!
In fact, KVM borrows so very much from qemu that it even refers to qemu's manpage for everything but two command-line switches.
Qemu is a completely different project, which gets to a very similar place but from the other extreme - Qemu started off as Bochs, a very slow but very useful multi-architecture emulator. Qemu started adding all kinds of optimizations, and it is nearly useful (i.e. I use it in my desktop whenever I need a W2K machine).
Instead of a heavyweight framework... KVM is just a modprobe away - Just ask Linux to modprobe kvm, and kvm -hda /path/to/your/hd/image gets you a working machine.
Anyway - I was immediatly happy with KVM. It took me a week to get a whole "lab" of 15 virtual computers (256MB RAM works surprisingly well for a regular XP install!) configured to start at boot time off a single master image over qcow images.
KVM's shortcomings
Xen has already been a long time in the enterprise, and has a nice suite of administrative tools. While Xen depends on having a configuration file for each host, KVM expects them to be passed at the command line. To get a bird-eye view of the system, xen has a load of utilities - KVM does not. And although RedHat's virt-manager is said to support KVM and qemu virtualization (besides its native Xen, of course), it falls short of what I need (i.e. it relies on a configuration file... which lacks expresivity to specify a snapshot-based HD image).
To my surprise, KVM has attained much of Xen's most amazing capabilities, such as the live migration. And although it's easier to just use fully virtualized devices (i.e. to use an emulation of the RTL8139 network card), as they require no drivers extraneous to the operating system, performance can be greatly enhanced by using the VirtIO devices. KVM is quickly evolving, and I predict it will largely overtake Xen's (and of course, vmware and others) places.
Where I am now
So... Well, those of us that adopt KVM and want to get it into production now will have some work of building the tools to gracefully manage and report it, it seems. I won't be touching much my setup until after Debconf, but so far I've done some work over Freddie Cash's kvmctl script. I'm submitting him some patches to make his script (IMHO) more reliable and automatizable (if you are interested, you can get my current version of the script as well). And... Starting September, I expect to start working on a control interface able to cover my other needs (such as distributing configuration to the terminals-to-be, or centrally managing the configurations).

Kept silent for a week...

Submitted by gwolf on Thu, 07/17/2008 - 20:42.

Last week (July 7-13) was basically hell on Earth, for me and for the group that somehow got the name Cabras locas, of which I am part since I joined the National Pedagogical University, where I worked full-time 2003-2005.
It was, yes, the first of my officially three weeks of Summer holiday at IIEc-UNAM, so no problems here. So, why hell on Earth? Because we were in charge basically of anything related with information flow, retrieval and manipulation at the 11th International Congress on Mathematical Education, in Monterrey.
What we thought would basically be one or two days of hard work followed by six days of relaxed vacations (we had even planned to have an internal seminar, showing off the shiny stuff each of us is working on) became... A mind-boggling eight day experience where we worked over 12 hours a day on being human replacements for Google, SQL engines, full-text parsers, report generators, printer watchdogs, and in general lines, just a bunch of unhappy firemen, ready to be called off for whatever task was necessary.
We did have, of course, several calm periods every now and then. We even had to learn how to look busy while doing something compeltely unrelated (that would explain, for example, a couple of low-hanging bugs I fixed for Debian, or some dozens of lines of code I could get off my head).
But my advice for whoever reads this: Don't trust people with long database-handling experience. Specially when they insist that hand-capturing a thousand registers is preferrable (i.e. less error-prone) than parsing three separate databases and discarding duplicates. And, of course, specially when this person is your boss, which is enough of an argument to have it his way.

If it happens, it's your fault

Submitted by gwolf on Wed, 07/16/2008 - 11:43.
If it happens, it's your fault

At the Cola de Caballo waterfall, near Monterrey, Nuevo León, Mexico

Wordle

Submitted by gwolf on Fri, 07/04/2008 - 21:56.

Via Planetalinux.mx, I read this post by César Espino refering to Wordle.
Quoting from Wordle's main page:

Wordle is a toy for generating “word clouds” from text that you provide. The clouds give greater prominence to words that appear more frequently in the source text. You can tweak your clouds with different fonts, layouts, and color schemes. The images you create with Wordle are yours to use however you like. You can print them out, or save them to the Wordle gallery to share with your friends.

I could not resist it. I even went to a computer with a Java runtime installed.

The application is very nice and usable, although its startup time is frankly irritating (specially as there is no feedback on why it's not loading). Anyway, the results are quite beautiful!

( categories: )

Firefox suckyness

Submitted by gwolf on Wed, 06/25/2008 - 14:32.

I have to agree with Wouter regarding Firefox Iceweasel 3's suckyness. It might be a superior product in many fronts (I prefer it overall to its predecesor), but were it not for the usefulness of its many available extensions (most notably Web Developer, which has become an integral part of my everyday life), I'd have jumped ship for basically any other browser.
I'm just... adding an <AOL>ME TOO!</AOL> on Wouter's comments... WTF, just go to about:config...

So... Are you telling me that Firefox (even if it were the original, Mozilla-blessed version) has a warranty? No? Didn't think so... Go to about:license. You will see the very familiar and expected:

7. Disclaimer of warranty
Covered code is provided under this license on an "as is" basis, without warranty of any kind, either expressed or implied, including, without limitation, warranties that the covered code is free of defects (...)

Other than this downright stupid issue (which by and large goes against the regular Free Software culture), my main gripe is the number of active regions in the location-entry boxes - Yes, I can jump straight to the search box with Ctrl-K and to the location bar with Ctrl-L, but if I happen to try to move between them with good ol' Tab, why must it be inconsistent?
What do I mean? Go to the search field, press Tab. As expected, you are in the Location entry. Now, press Shift-Tab. For 5 points, where are you now? Bzzzt, no, you are not back where you began - You are in a stupid button which looks like your favicon giving you the identity information about the page you are looking at.
Having this button is a great idea - but why does it have to sit in the way of my tabulation? Were it because, in an inspired moment, the Firefox interface designers decided that buttons should be keyboard-accessible, I'd be most happy (it is by far my most mouse-intensive application, and I hate that... But it's just a button embedded in what should be a clean text-entry box.
And for that matter, it is not even a consistent button - Shift-tabbing from the search field will not get you to the search engine selector. And no key combination will lead you to the noisy (and also, mutually inconsistent) iconic buttons at the far right of both fields... Which, again, have no reason to be inside a text entry field!
Oh, and I found another pretty little jewel: Go to any site which has a self-signed SSL certificate. Of course, Firefox will go to great lengths to make sure you understand how unsafe is it for you to trust anybody who didn't pay big bucks to Verisign... But this is enough reason for me to send a bug report:

I am connecting through a crypted HTTPS connection. The site is providing identity information - not certified, right, but it does provide something. And the connection is crypted. Firefox/Iceweasel 2 showed me the URL in a light-yellow background showing the connection was secure - Now it just denies penta.debconf.org the right to call itself secure.
(Yes, I am not an interface designer... But it seems neither are they)
[update]Bug filed. Any comments will be welcome in Bugzilla. The bug with the Debconf site (and I do regard it as a bug) is that Iceweasel displays that Your connection to this web site is not encrypted because some of the elements (i.e. the CSS, images..) are sent in the clear - Even though the real information is crypted. Ever heard of data/visualization decoupling?

( categories: )

eSATA, USB and friends

Submitted by gwolf on Wed, 06/18/2008 - 14:24.

Andrew asks about Linux-friendly eSATA controllers. A long time ago, I also looked for some - but basically no one seems to know about that, or will try to rip your eyes for it.
In January, however, I looked for (and bought) an external USB enclosure for SATA disks. As portability was more important than speed, I only asked for USB. Good, got the kit, paid for it.
The external enclosure had an external eSATA port. Not only that, it shipped with a standard PC expansion slot bracket and adaptor - Yes, it connects to your internal SATA port, and provides an external eSATA port. And, of course, it will work with your current kernel painlessly. And, of course, it is even cheaper than a SATA controller with an external port.

( categories: )

Freedom itches

Submitted by gwolf on Sat, 06/14/2008 - 01:43.

In this Free Software movement we have many mottos - One of which, describing what motivates us to work writing code, is scratch where it itches.
Of course, I could not keep it to myself - Almost a week ago, I took part of the World Naked Bike Ride. What I didn't tell you... Is that it became obvious I cannot reach most of by back - And it's because I'm mostly careless. When the WNBR started, it was still quite cloudy, even starting to rain... so I was mostly careless.
If you opened the newspaper PDF I attached to my previous post, you'll surely remember (not an easy sight to get out of your head, I guess) I had painted on my back "Vehículo libre de emisiones" - Emissions-free vehicle (and yes, it's strictly true: My bike is zero emissions. The animal riding it might not be... But that's a different story). Add incomplete sunblock to the equation, and...

Were it not for the poor lighting conditions under which I took the photo, you'd clearly appreciate the words "libre de" on my back.
And... Well, one week later, my freedom itches.
Badly.

( categories: )

No, it's not

Submitted by gwolf on Wed, 06/11/2008 - 15:30.

Several people have approached me (or I've stumbled upon their sites) asking me about something called Debian 5.0 Beta 2.
It. Is. Not. That.
Please read clearly the announcement for Debian Installer lenny beta 2 - Yes, I understand this reached many people who are not involved in Debian but are enthusiastic users nevertheless. In short: The only thing that reached the beta is the debian-installer program (usually called just d-i), the amazing piece of code that handles a Debian installation in your system. And yes, it is meant for wide testing and work.
But please, do not take this as a preview of the new Debian release - it is not. If you install a system using this version of d-i, you will be tracking the Testing branch of Debian, and your system will be in a continuous state of flux. Yes, we do expect a freeze of Lenny in the next couple of weeks, after what it will be quite close to a Beta release (i.e. almost no new versions, no fresh software, just bug fixes). But hey - A Beta is supposed to be close to release quality. And if you look at the release-critical bugs affecting the Testing branch (green line), you will clearly see we have over 400 bugs to fix before Lenny is allowed to be called stable. And that's only one of the criteria needed to reach Lenny - Glance over at the Debian Release Management page to quickly understand the nature of changes still to come.
Oh, and of course - Even if it is not necessarily up-to-date, I have found the Wiki page created by Peter Eisentraut as an excellent place to start working whenever I have some free time: Lenny Release Goals.
so... If you are not yet working towards making Debian the best distribution ever, and Lenny the best Debian release ever, you now know where we need your help ;-)
(side note: d-i team, maybe the next announcement could use some words pointing out we are not doing a Debian beta program, just a d-i beta release?)

( categories: )

Nekkid city - yet again!

Submitted by gwolf on Wed, 06/11/2008 - 04:01.

After thinking it over a couple of times, I did it. I told you here about the World Naked Bike Ride. Thousands of bikers, in over 130 cities around the world, voiced their concerns about the lack of caution drivers have towards us, about the abuse of fossil fuels for urban transportation, about the easy we are not to be seen. Many among us have been run over by careless drivers (in my case, no consequences except a broken helmet - And yes, MJ: although the impact was on the flat surface of the road and not on the kerb, the strength of the impact still amazes me). We feel naked against the motorized traffic. So, the WNBR decides to show it by taking the streets of our many cities - Naked.
It was a completely different experience of the massive naked Spencer Tunick photo, as we were there not just to show our freedom and enjoy, but to get the people to look at us. There were some of the same elements of comradeship and trust we had there (and, of course, that many of us learnt in Finland when we became GNUdists at DebConf 5's unforgettable saunas).
Anyway... I did not make the full route (I rode Chapultepec-Zócalo-Diana, ~15Km, but missed the Diana-Gandhi-Cibeles part, maybe some 5Km) as I had an appointment I was already late for. But it was a unique, great experience. If you are interested, we got a fair share of press coverage. Oh, and I must say: I am famous now. And in my favorite newspaper, nothing less :).

( categories: )

Running, biking, and the like

Submitted by gwolf on Tue, 06/10/2008 - 15:25.

Bubulle's improvements on running make me envy him. Of course, I'm nowhere close to Dirk's (or for that matter, I can only hope never to be as insane as Mauricio).
I must admit that I have heavily reduced my running. This is in no small part due to the fact that it's not easy to find decently-priced running shoes of the right size in Mexico - But well, that's only an excuse after all. But yes, if I run over 7Km with my current shoes, I get blisters, invariably. And I'm a bit short on money right now...
Partly, I've been moving my preferred excercise to biking - I don't do heavy biking, of course, but I do try to get at least 15Km every day (and as I bike to and from work, I only have to take small extra detours to reach that point - although splitting an excercise session in several sub-sessions is very close to cheating on myself) and at least a longer, continuous, 25-30Km ride every weekend. But yes, I've lowered my rythm.
Also, I'm running less because of where I am currently living - Very close to my usual area, but quite a bit more hilly, and it has made my knees ache every now and then.... And by far, it is not as nice and friendly area.
I do hope to work on this - And, of course, the Debconf Morning Jog is one of the events I most look forward to. Running in Edinburgh was, after all, my favorite and most productive way to get to know the city.

( categories: )

Hope...

Submitted by gwolf on Mon, 06/09/2008 - 14:31.

Exists. Feels nice. Makes me float. Is it real?
On the other hand... Fear exists as well...
And there even are nice sparks in my temporal reality...

( categories: )

I've fallen and I can't get up!

Submitted by gwolf on Thu, 06/05/2008 - 21:59.

I think I should follow up on Victor's lament. Yes, we have a Rails application which works fine most of the time... But quite often, throws out a segmentation fault I just have been unable to pin-point. It might be related to rmagick, the only non-pure-Ruby component I am using (and I'm tempted to try minimagick instead, even if I prefer in-memory operations than on-disk, piping an image and slurping it again).
Victor came up with an easy script to check the server - but to reduce the impact it has (I was running a single Mongrel instance, which meant, whenever it dies the whole system becomes inaccessible for everybody; I replaced it with a mongrel_cluster of five processes, plus pound as a easy-to-use balancer which looks quite nice), the very simplistic and to-the-point script did no longer work.
Anyway... Ruby rocks ;-) I'm sharing this with you mostly because I am sure some readers will find more than one useful construct, not because it is precisely beautiful code. And besides, we should work on fixing the cause, not the consequence, of the bug! :)

  1. #!/usr/bin/ruby
  2. require 'yaml'
  3. confdir = '/etc/mongrel-cluster/sites-enabled'
  4. restart_cmd = '/etc/init.d/mongrel-cluster restart'
  5. needs_restart = false
  6.  
  7. (Dir.open(confdir).entries - ['.', '..']).each do |site|
  8. conf = YAML.load_file "#{confdir}/#{site}"
  9. pid_location = [conf['cwd'], conf['pid_file']].join('/').gsub(/\.pid$/, '*.pid')
  10. pid_files = Dir.glob(pid_location)
  11.  
  12. pid_files.each do |pidf|
  13. pid = File.read(pidf)
  14. begin
  15. Process.getpgid(pid.to_i)
  16. rescue Errno::ESRCH
  17. warn "Process #{pid} (cluster #{site}) is dead!"
  18. File.unlink pidf
  19. needs_restart = true
  20. end
  21. end
  22. end
  23.  
  24. system(restart_cmd) if needs_restart

Works out of the box for any Debian-packaged mongrel-cluster. Sadly, mongrel-cluster does not provide a way to restart individual servers - Of course, I could (should, even) work it out to build the specific command-line... but at least, it works for now.
Uh-oh... Does that mean it's permanent?

( categories: )

A long-needed explanation on $self->status

Submitted by gwolf on Mon, 06/02/2008 - 04:01.

[Attention] Personal content follows. If you got to this post expecting technical or organizational content, go ahead and skip it.
Sometimes, when you don't want something to happen... You don't even talk about it. Sometimes you try hard not even to think about it - Well, it wasn't so in this case, as it has been a long time devoted to... Thinking, thinking a lot.
The important thing is that I feel there is a fact I should have shared with the people I consider close to me. And some people do know it, of course, but it has been hardest for me to share this with who I consider my closest friends - maybe out of hope that the result will be something different?
Anyway... the fact is I have been living on my own for the last two months. Nadezhda and I decided to take a three month period to... Decide what comes next. And most of the time since this strange period began has been quite decent... But the last week or so has been... A bit different. I have been much more... introspective. Maybe it could be seen as more depressive, more introverted.
So, what does this mean? Do I have my mind clear on what to do? No. Not by far. But I am working on finding my feelings and my reality, full-time. I might be under-achieving both in my Real-Life works and in Debian - please cope with me. There is a reason for it, as you now see.
And, no, this post is not aiming at getting sympathy and hugs... It is just... Because I need to finally share this thing I am going through with. Think of it as finally stepping out of the closet, more as a step for myself than for anything else... A significant share of who I consider dear to me, even though we share no contact besides mailing lists I have mostly abandoned and two cathartic and beautiful (and, of course, technically productive!) weeks a year, lives in different timezones and countries - I do hope to have stabilized on either of the two possible realities by the time I meet my Debian fellows. As for my Mexican friends following my life by this blog... Well, just excuse me for not speaking out face to face, as this should have been done.
Comments intentionally closed. Want to say something? Just think it hard enough, it will get to its destination ;-)

( categories: )

Proud as a father would be

Submitted by gwolf on Tue, 05/20/2008 - 01:54.

The recent OpenSSL incident can not be hidden. It was a very important blow to the Debian project's public face and reputation. A major hole slipped under the door in the form of a bugfix - and with all the good intention. This was not a deliberate attack, nor was it the result of a bad or sloppy maintainer - It was a honest, although painful, human mistake.
Several people started laughing at our processes and supposed strengths right away. I do, however, feel this shows how Debian is stronger security-wise than any other system. And it also shows how this saying, with enough eyeballs, all bugs are shallow, not only didn't lose validity, but was reaffirmed. Free Software development was also proved to be better than security through obscurity again.
Why?
Because were it not because of OpenSSL (and in this case in particular, Debian's packaging) being Free and subject to a code audit, this problem would have never been found. I have been asking to some friends who are part of different black-hat groups, and looking for this kind of information on the Web, it seems that -were it not for Luciano's work, we would still be running cryptographically weakened versions of OpenSSL for a long time. After all, 32768 possible keys is still quite a lot for a black-hat group to find as uneven noise, as a lead to showing the undeniable weakness.
It took two years to find the bug, yes. But it was found doing quality assurance work on publicly available source code. It was promptly fixed, mitigating (as far as possible) as much damage as could be caused. Tools for finding and fixing the defective keys were crafted and freed together with the announcement. Yes, there will be some compromises due to this, I'm sure, but an embarassing hole has been dealt with in the best way possible.
Anyway... I am very happy - I was going over Luciano's NM report, and found something I only suspected but was not sure about. I can now state clearly: I have never been so happy to advocate somebody to become a DD.
Luciano wrote a very good blog post (in Spanish) with his viewpoints on the Debian OpenSSL incident. If you happen to understand Spanish and are reading this blog, please drop over Luciano's.
Luciano: Once again, my hat goes off for you :-)

( categories: )

Ruby has a distribution problem

Submitted by gwolf on Tue, 05/06/2008 - 16:30.

I usually don't like me too comments... But this is something that really disappoints me of my otherwise-favorite development framework. I must echo Matt Palmer's comment on Luke Kanies' entry:
Ruby. Has. A. Distribution. Problem.
Nice, good read. Sadly, many Rails pushers see distributability as something very minor, something that should not worry Rails developers right now, as there is too much other serious work to be done - Better UTF8, a clearer language, better performance... And besides, any programmer can live well with gems. (yes, that's all taken from a rant I had with a very convinced person)
My gripe is that... Rails is no longer a small, fringe project. Rails is an enterprise-grade development framework, with thousands of deployed production systems. And if they don't start to act responsably, if the Rails developers keep pushing said problems as low-priority, the Rails developers' (that is, their users) culture will become rigid - and will constitute a serious harm to Rails' future.
Distributability and packageability is not only for OS distributors. Not only we Debian zealots care about software being easily packageable. By using Ruby Gems, you dramatically increase entropy and harm your systems' security.
Read Luke's text for more details. It is quite worth the time.

( categories: )