Gunnar Wolf

I like Drupal. It's a very, very flexible CMS that evolved into a full-fledged Web development framework. Mind you, it's written in PHP, and that makes it a nightmare to develop for (in ~6 years I have used it for all of my important websites I have only got around to develop a set of related modules for it once).

PHP programming sucks and makes my eyes and fingers bleed, but happily there are people who disagree with me — And they tend to write code. All the better!

Minor upgrades with Drupal are quite easy to handle. Not as easy as I'd like (i.e. whenever I upgrade the core system or a module, I have to log in as root^Wadmin^WUser #1 to ~15 different sites and run http://someurl/update.php — It very seldom causes any pain.

The updates that have to be run via this URL are usually on the database's structures, so I understand they have to be started (and watched) by a human. And yes, I know I could do that with Drush, the Drupal shell, but it is not very friendly to Debian-packaged Drupal... But easy enoguh.

But major updates are a royal pain, and they usually amount to quite a bit. First, disable all of the modules and revert to a known-safe theme. Ok, it makes sense. Second, check whether the modules exist for the newer version (as they won't work — Drupal changes enough between major versions that not only it's API-incompatible, I'd classify it as API-unrecognizable). Ok, all set? Now for the live migration itself... It has to be triggered from the browser.

So yes, I am now staring at a window making clever AJAX status updates. I am sitting at 46 of 199, but following the lovely ways of programmers, it's impossible to forsee whether update #47 will just be an UPDATE foo SET bar=0 WHERE bar IS NULL or a full-scale conversion between unspeakable serialized binary structures while rearranging the whole database structure.

And yes, while the meter progresses I stand in fear that update #n+1 will bomb giving me an ugly red error. I must keep the magic AJAX running, or the update might be botched.

And, of course, the update has sat at #69 all while I wrote the last two paragraphs. Sometimes the updates can progress after an interruption... And it seems I have no choice but to interrupt it.

/me crosses fingers...

[update] Wow... I am happy I got bored of looking at the meter and decided to write this blog post: After several minutes, and just as I was about to launch a second update session (130 updates to go), the meter advanced! I'm now sitting watching it at #75. Will it ever reach 199?

[update] And so it had to be... At around 115, I now got:

*sigh* The update process was aborted prematurely while running update #7000 in biblio.module...

Patrick tells about his experience moving from LVM to RAID.Now, why do this? I have two machines set up with LVM-based mirroring, and they work like a charm - I even think they work with better flexibility than setting it up in a RAID-controlled way, as each of the partitions in a volume group can be easily set to use (or stop using) the mirroring independently, and the requisite of having similar devices (regarding size) also disappears. Of course, this flexibility allows you to do very stupid things (such as setting up a mirror on two areas of the same rotational device - Good for toying around, but of course, never to be considered for production). And the ability to online grow and shrink partitions is just great.

So, Patrick, fellow readers, dear lazyweb, why would you prefer LVM-based mirroring to a RAID alternative? Or the other way around?

Lucas has written two long, insightful posts on the frustration about the social aspects of integrating Ruby stuff Debian – The first one, on September 12 and the second one, on September 29.

I cannot really blame (thoroughly) the Ruby guys for their position. After all, they have a vibrant community, and they are advancing great pieces of work. And they know who that code is meant for — Fellow programmers. And yes, although it is a pain to follow their API changes (and several of the Gems I regularly use do often get refactorings and functionality enhancements which break compatibility but introduce very nice new features), they say that's solved with one of Gems' main features being the simultaneous installability of different versions.

The key difference in Debian's worldview with Ruby's is they cater to Fellow programmers. Even leaving aside heaps of different positions and worldview/mindset, we have a fundamental difference: Debian cares about its users, whatever that means. So, our users should not even care what language a given application is implemented in – They should only care that it works. We, as packagers, should take care of all the infrastructural stuff.

And yes, that's where we find the conflicting spot: We don't want to ship many versions of a system library (that in this case would be a Gem). Specially if later versions fix known bugs in earlier versions and backports are not available or supported. Specially if upstream authors' only response to a bug in an older release will be "upgrade and rewrite whatever breaks in your application".

As an example of this, I am not currently updating the gems I maintain, as Debian is on a freeze to get out the next stable release. Or if at all, I am targetting those uploads to our Experimental branch, in order not to create a huge backlog for me when the freeze is over (just a series of rebuilds targetted at unstable). And yes, I will have to be responsible for any bugs that will most likely not be supported by most of my upstreams during the next ~2 years.

That's the role of a Linux distribution. And yes, as Lucas writes in the comments he got as responses to the first post – This dissonance comes in no small part because the Ruby developer community is mostly made from non-linuxers. People coming from a background where (mostly propietary) applications bundle up everything they need, where static linking is more popular than dynamic libraries, where there is no coordination between parts of the system are much less likely to understand our work.

And yes, the Perl community is a joy to work with in this regard. And that's the same I understand from the Python one. Because of their origins and where their main strength was grown and remains.

PS - And yes, I will join the flock of people saying that... The specific person that attacked your work is a great programmer, but well known as intolerant and obnoxious. Fortunately, even if our respective cultures fail to mix so much, most of our interactions just end with a "sigh" of lack of understanding, and not with the flames you got targetted with :-/

This morning, my laptop was stolen from my parked car while I was jogging. I do not want to make a big deal out of it.

Still, even though I am sure it was not targetted at my data (three other people at least were reporting similar facts in the same area), and the laptop's disk will probably just be reformatted, I am trying to limit the possible impact of my cryptographic identification being in somebody else's hands.

GPG makes it easy: I had on that machine just my old 1024D key, so it is just matter of generating a revocation certificate. I have done that, and uploaded it to the SKS keyservers - Anyway, here is my revocation certificate:

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: A revocation certificate should follow

iHIEIBEIADIFAkyaOZwrHQJBIGNvbXB1dGVyIGNvbnRhaW5pbmcgdGhpcyBrZXkg
d2FzIHN0b2xlbgAKCRDYDvNai7UnrzWAAKC34eF76JQjxrZqSjNwcC0dU/5VbACg
gMIMmYg91Sl3y8KsZXdGj/rV7UE=
=rdlT
-----END PGP PUBLIC KEY BLOCK-----

But… What worries me more is access to the computers my ssh key works for. Yes, the ssh key uses a nontrivial passphrase, but still — SSH keys cannot be revoked (and this makes sense, as SSH should not add the delay, or potential impossibility, to check with a remote infrastructure whenever you want to start a session).

So, I generated a new key (and stored it at ~/.ssh/id_rsa.new / ~/.ssh/id_rsa.new.pub) and came up with this snippet:

$ OLDKEY=xyHywJuHD3nsfLh03G1TqUEBKSj6NlzMfB1T759haoAQ
$ for host in $(cat .ssh/known_hosts | cut -f 1 -d \ |cut -f 1 -d , |
                sort | uniq); do 
    echo == $host
    ssh-copy-id -i .ssh/id_rsa.new.pub $host && 
    ssh $host "perl -n -i -e 'next if /$OLDKEY/;print' .ssh/authorized_keys"
done

Points about it you might scratch your head about:

• .ssh/known_hosts' lines start with the server's name (or names, if more than one, comma-separated), followed by the key algorithm and the key fingerprint (space-separated). That's the reason for the double cut – It could probably be better using a regex-enabled thingy understanding /[, ]/, but... I didn't think of that. Besides, the savings would be just for academic purposes ;-)
• I thought about not having the ssh line conditionally depend on ssh-copy-id. But OTOH, this makes sure I only try to remove the old key from the servers it is present on, and that I don't start sending my new key everywhere just for the sake of it.
• my $OLDKEY (declared in Shell, and only literally interpolated in the Perl one-liner below) contains the final bits of my old key. It is long enough for me not to think I'm risking collision with any other key. Why did I choose that particular length? Oh, it was a mouse motion.
• perl -n -i -e is one of my favorite ways to invoke perl. -i means in-line editing, it allows me to modify a file on the fly. This line just skips (removes) any keys containing $OLDKEY; -n tells it to loop all the lines over the provided program (and very similarly, -p would add a print at the end – Which in this particular ocassion, I prefer not to have). It is a sed lookalike, if you wish, but with a full Perl behind.

Caveats:

• This assumes you have set HashKnownHosts: no in your .ssh/config. It is a tradeoff, after all – I use a lot tab-expansion (via bash_completion) for hostnames, so I do have the fully parseable list of hosts I have used on each of my computers.
• I have always requested my account names to be gwolf. If you use more than one username... well, you will have to probably do more than one run of it connecting to foo@$host instead.
• Although most multiuser servers stick to the usual port 22, many people change the ports (me included) either because they perceive concealing them gives extra security, or (as in my case) because they are fed up with random connection attempts. Those hosts are stored as [hostname]:port (i.e. [foo.gwolf.org]:22000). Of course, a little refinement takes care of it all.,/li>
• Oh, I am not storing results... I should, so for successive runs I won't try to connect to a system I already did, or that already denied me access. Why would I want to? Because some computers are not currently turned on. So I'll run this script at least a couple of times.

Oh, by the way: If you noticed me knocking on your SSH ports... please disregard. Possibly at some point I connected to that machine to do something, or it landed in my .ssh/known_hosts for some reason. I currently have 144 hosts registered. I am sure I triggered at least one raised eyebrow.

And I will do it from a couple of different computers, to make it less probable that I miss some I have never connected from while at the particular computer I am sitting at right now.

So... Any ideas on how to make this better?

The release of OpenSSH 5.4 was announced today. Its announced features include many small improvements, in usability and in crypto strength.

One of my favorite tricks using ssh is what Ganneff named ssh jumphosts – Many (most?) of my machines are not directly accessible from across the firewall, so the ability to specify in the configuration files where to jump through is most welcome. Well, with this "netcat mode" it will be much clearer to read and less of a hack… Of course, it loses a bit of the hackish æsthetic value, but becomes easier!

(yes, this post is basically a marker so I remember about it — But others might find it interesting)

Nobody cares about me, I thought. Whatever I say is just like throwing a bottle to the infinite ocean.

No comments, no hopes of getting any, for several days. Weeks maybe? Not even the spammers cared about me.

Until I read this mail, by Thijs Kinkhorst commenting to my yesterday post:

(…)
(BTW, I was unable to comment on your blog - couldn't even read one letter of the CAPTCHA...)

And, yes, Drupal module «captcha» introduced in its 2.1 release (January 2) feature #571344: Mix multiple fonts.

Only... no fonts were selected. Grah.

New guidelines for periodic publications' websites at my University favor the different journals we have to use a standardized system — And it makes quite a bit of sense. It is quite hard to explain to the people I work with that the content is not only meant to be consumed by humans, but also by other systems; the reasons behind rich content tagging and deep hierarchies for what they would just see as a list of words (think list of authors for an article, list of keywords, and so on). After all, aggregator databases such as Latindex and SciELO have achieved getting this understanding through.

And I must be quite grateful, as the University's guidelines point to what appears to be a very well-thought and thorough system, the Open Journal Systems by the Public Knowledge Project, co-funded by several well-regarded universities. OJS is a GPL-2-covered PHP bundle.

Anyway… I am very glad at least one of my Institute's journal accepted the challenge and decided to go OJS. I know I will quite probably be administering this system long-term. And, being as snobbish as I am, I know I loathe anything installed in my machines that is not either developed by myself or comes in a Debian package. So, as it was not packaged, I made the package ☺

Note that I am still not filing an ITP (which means, I have not yet decided whether I will upload this to Debian) because I want first to make sure I do have the needed long-term commitment — Besides, I am by far not a PHP person, and being responsible for a package… Carries a nontrivial weight. Still, you might be interested in getting it. If you are interested, you can either download the .deb package or add it to your apt repositories (and stay updated with any new releases), by adding this to your /etc/apt/sources.list:

Note: My packaging has still a small bug: The installer fails to create the PostgreSQL database. The MySQL database works fine. I will look into it soon

So far, I am quite impressed with this program's functionality and the depth/quality of its (online) documentation. Besides, its usage statistics speak for themselves:

So, it is quite possible I will be uploading this into Debian in a couple of weeks (hopefully in time to be considered for Squeeze). The reasons I am making it available in my personal repository now is:

• I want to make it available for other Debian- and Ubuntu- users in my University, as I am sure several people will be installing it soon. And after apt-getting it, it is just ready to be used right away.
• As I said, I am no PHP guy. So if you want to criticize my packaging (and even my minor patch, fixing a silly detail that comes from upstream's bundling of several PHP and Javascript libraries, and those libraries' authors not sticking to a published API in a well-distributed version), please go ahead!

Every now and then, people ask me why Debian? Why, among so many projects to choose from, I first liked, then got into, and finally I got committed into Debian, and not anything else?

Of course, one of the main points —back in 2000-2001 when I started using it, and still to this very day— is a strong identification with the ideological side. Yes, I am a strong Free Software believer, and Debian is what best suites my ideology.

Still, I did not only get into Debian because of this — And I was reminded about this by an article in this month's Usenix ;login: magazine: An anecdotal piece by Thomas A. Limoncelli titled Hey! I have to install and maintain this crap too, ya know! (article requires ;login: subscription, but I'll be glad to share it with whoever requests it to me — I have of course no permission to openly put it here in whole online. Yes, I am expressly sending a copy of this text to the author, I will update this if/when I hear from him) [update] The author has kindly allowed me to redistribute his article's PDF — Download it here.

Before anything else… I'll go on a short digression: I am writing a bit regarding the Free Software participants' culture, and this is a trait I love about it: The lack of formality. Even though ;login: (and Usenix as a whole) is not exactly Free Software, it runs quite close to it), it is a well regarded magazine (and association) with an academic format and good (not deep or highly theoretical, but good) contents. Still, it is quite usual to see titles as informal and inviting as this one. And it happens not only here — I have been fearing having to explain at work, over and over, why I have requesting permissions to go to Yet Another Perl Conference, Festival de Software Libre or DebCamp, tagging them as academic settings. Or why I am wasting our library's resources on buying cookbooks, recipes and similar material on the most strange-sounding subjects.

Anyway, back on track… This article I found refers to the lack of value given to the system administrator's time when selling or purchasing (or more in general, as it happens also in Free Software, when offering or adopting) a product. Quoting Thomas:

A person purchasing a product is focused on the features and benefits and the salesperson is focused on closing the deal. If the topic of installation does come up, a user thinks, “Who cares! My sysadmin will install it for me!” as if such services are free. Ironically, it is the same non-technical executive who dismisses installation and upkeep as if they are “free” who might complain that IT costs are too high and go on a quest to kill IT spending. But I digress.

I can understand why a product might be difficult to install. It is hard enough to write software, and with the shortage of software developers it seems perfectly reasonable that the installation script becomes an afterthought, possibly given to a low-ranking developer. The person purchasing the product usually requires certain features, and ease of installation is not a consideration during the procurement process. However, my ability to install a product affects my willingness to purchase more of the product.

Thomas goes on to explain his experience with Silicon Graphics, how Irix was so great regarding install automation and how they blew it when switching to Windows NT; talks very briefly about IBM AIX's smit, a very nifty sysadmin aid which is basically a point-and-click interface to system administration with the very nice extra that allows you to view the commands smit executes to perform a given action (and then you can copy into a script and send over to your hundreds of AIX machines)… Incidentally, by the time I started digging out of what became the RedHat mess of the late 1990s and passed briefly through OpenBSD on my way to Debian enlightenment, I was temporarily the sysadmin for an AIX machine — And I too loved this Smit approach, having it as the ultimate pedagogical tool you could ever find.

Anyway, I won't comment and paraphrase the full article. I'll just point out to the fact that… this was what ultimately sold me into Debian. The fact that I could just install anything and (by far) most of the times it will be configured and ready to use. Debian made my life so much easier! As a sysadmin, I didn't have to download, browse documentation, scratch head, redo from start until I got a package working — Just apt-get into it, and I'd be set. Of course, one of the bits I learnt back then was that Debian was for lazy people — Everything works in a certain way. Policy is enforced throughout.

So as a sysadmin, I should better get well acquinted with the Debian policy and know it by heart. In order to be able to enjoy my laziness, I should read it and study it. And so I did, and fell in love. And that is where my journey into becoming a Debian Developer started.

Why am I talking so nostalgic here? Because I got this magazine on the mail just last weekend… And coincidentally, I also got bug report #551258 — I packaged and uploaded the Haml Ruby library (Gem, as the Rubyists would call it). Haml is a great, succint markup language which makes HTML generation less of a mess. It is even fun and amazing to write Haml, and the result is always nicely formatted, valid HTML! And well, one of Haml's components is haml-elisp, the Emacs Lisp major mode to do proper syntax highlighting in Haml files.

Of course, I am an Emacs guy (and have been for over 25 years), so I had to package it. But I don't do Emacs Lisp! So I just stuffed the file in its (supposed) place, copying some stuff over from other Emacs packages. During DebConf, I got the very valuable help of Axel Beckert to fix a simple bug which prevented my package from properly being installed, and thought I was basically done with it. I was happy just to add this to my ~/.emacs and get over with it:

(require 'haml-mode)
(add-to-list 'auto-mode-alist '("\\.haml$" . haml-mode))
(require 'sass-mode)
(add-to-list 'auto-mode-alist '("\\.sass$" . sass-mode))

However… As Mike Castleman points out: This requires manual intervention. So it is not the Debian Way!

Reading Mike's bug report, and reading Thomas' article, made me realize I was dilluting something I held so dearly as to commit myself to the best Free Software-based distribution out there. And the solution, of course, was very simple: Debian allows us to be very lazy, not only as sysadmins, but as Debian packagers. Just drop this (simplified version) as $pkgroot/debian/haml-elisp.emacsen.startup and you are set!

(let ((package-dir (concat "/usr/share/"
                           (symbol-name flavor)
                           "/site-lisp/haml-elisp")))
;; If package-dir does not exist, the haml-mode package must have
;; removed but not purged, and we should skip the setup.
  (when (file-directory-p package-dir)
    ;; Use debian-pkg-add-load-path-item per §9 of debian emacs subpolicy
    (debian-pkg-add-load-path-item package-dir )
    (autoload 'haml-mode "haml-mode"
      "Major mode for editing haml-mode files." t)
    (add-to-list 'auto-mode-alist '("\\.haml\\'" . haml-mode))
    ;; The same package provides HAML and SASS modes in the same
    ;; directory - So repeat only the last two instructions for sass
    (autoload 'sass-mode "sass-mode"
      "Major mode for editing sass-mode files." t)
    (add-to-list 'auto-mode-alist '("\\.sass\\'" . sass-mode))
    ))

This will make the package just work as soon as it is installed, with no manual intervention required from the user. And it does not, contrary to what I feared, bloat up Emacs — Adding it to the auto-mode-alist leaves it as known to Emacs, but is not loaded or compiled unless it is required.

Deepest thanks to both of you! (and of course, thanks also to Manoj, for pointing out at the right spells in emacs-land)

Humm... Has anybody else seen a pattern like this?

I am getting a flurry of root login attempts at my main server at the University since yesterday 7:30AM (GMT-5). Now, from the machines I run in the 132.248.0.0/16 network (UNAM), only two listen to the world with ssh at port 22 — And yes, it is a very large network, but I am only getting this pattern on one of them (they are on different subnets, quite far apart). They are all attempting to log in as root, with a frequency that varies wildly, but is consistently over three times a minute right now. This is a sample of what I get in my logs:

[update] Logs omitted from blog post, as it is too wide and breaks displays for most users. You can download the log file instead.

Anyway… This comes from all over the world, and all the attempts are made as root (no attempts from unprivileged users). Of course, I have PermitRootLogin to no in /etc/ssh/sshd_config, but… I want to understand this as much as possible.

Initially it struck me that most of the attempts appeared to come from Europe (quite atypical for the usual botnet distribution), so I passed my logs through:

#!/usr/bin/perl
use Geo::IP;
use IO::File;
use strict;
my ($geoip, $fh, %by_ip, %by_ctry);
 
$fh = IO::File->new('/tmp/sshd_log');
$geoip=Geo::IP->new(GEOIP_STANDARD); 
while (my $lin = <$fh>) { next unless $lin =~ /rhost=(\S+)/; $by_ip{$1}++}; 
 
print "        Incidence by IP:\n", "Num  Ctry   IP\n", ('='x60),"\n";
 
for my $ip ( sort {$by_ip{$a} <=> $by_ip{$b}} keys %by_ip) {
    my $ctry = ($ip =~ /^[\d\.]+$/) ? 
	$geoip->country_code_by_addr($ip) : 
	$geoip->country_code_by_name($ip);
 
    $by_ctry{$ctry}++;
    printf "%3d  %3s  %s\n", $by_ip{$ip}, $ctry, $ip;
} 
 
print "        Incidence by country:\n", "Num  Country\n", "============\n";
map {printf "%3d  %s\n", $by_ctry{$_}, $_}
    sort {$by_ctry{$b} <=> $by_ctry{$a}} 
    keys(%by_ctry);

The top countries (where the number of attempts ≥ 5) are:

104  CN
 78  US
 58  BR
 49  DE
 43  PL
 20  ES
 20  IN
 19  RU
 17  CO
 17  UA
 16  IT
 13  AR
 12  ZA
 10  CA
 10  CH
  8  GB
  8  AT
  8  JP
  8  FR
  7  KR
  7  HK
  7  PE
  7  ID
  6  PT
  5  CZ
  5  AU
  5  BE
  5  SE
  5  RO
  5  MX

I am attaching to this post the relevant log (filtering out all the information I could regarding legitimate users) as well as the full output. In case somebody has seen this kind of wormish botnetish behaviour lately… please comment.

[Update] I have tried getting some data regarding the attacking machines, running a simple nmap -O -vv against a random sample (five machines, I hope I am not being too agressive in anybody's eyes). They all seem to be running some flavor of Linux (according to the OS fingerprinting), but the list of open ports varies wildly — I have seen the following:

Not shown: 979 closed ports
PORT      STATE    SERVICE
21/tcp    open     ftp
22/tcp    open     ssh
23/tcp    open     telnet
111/tcp   open     rpcbind
135/tcp   filtered msrpc
139/tcp   filtered netbios-ssn
445/tcp   filtered microsoft-ds
593/tcp   filtered http-rpc-epmap
992/tcp   open     telnets
1025/tcp  filtered NFS-or-IIS
1080/tcp  filtered socks
1433/tcp  filtered ms-sql-s
1434/tcp  filtered ms-sql-m
2049/tcp  open     nfs
4242/tcp  filtered unknown
4444/tcp  filtered krb524
6346/tcp  filtered gnutella
6881/tcp  filtered bittorrent-tracker
8888/tcp  filtered sun-answerbook
10000/tcp open     snet-sensor-mgmt
45100/tcp filtered unknown
Device type: general purpose|WAP|PBX
Running (JUST GUESSING) : Linux 2.6.X|2.4.X (96%), (…)
 
 
Not shown: 993 filtered ports
PORT     STATE  SERVICE
22/tcp   open   ssh
25/tcp   open   smtp
80/tcp   open   http
443/tcp  open   https
444/tcp  open   snpp
3389/tcp open   ms-term-serv
4125/tcp closed rww
Device type: general purpose|phone|WAP|router
Running (JUST GUESSING) : Linux 2.6.X (91%), (…)
 
Not shown: 994 filtered ports
PORT    STATE  SERVICE
22/tcp  open   ssh
25/tcp  closed smtp
53/tcp  closed domain
80/tcp  open   http
113/tcp closed auth
443/tcp closed https
Device type: general purpose
Running (JUST GUESSING) : Linux 2.6.X (90%)
OS fingerprint not ideal because: Didn't receive UDP response. Please try again with -sSU
Aggressive OS guesses: Linux 2.6.15 - 2.6.26 (90%), Linux 2.6.23 (89%), (…)
 
Not shown: 982 closed ports
PORT      STATE    SERVICE
21/tcp    open     ftp
22/tcp    open     ssh
37/tcp    open     time
80/tcp    open     http
113/tcp   open     auth
135/tcp   filtered msrpc
139/tcp   filtered netbios-ssn
445/tcp   filtered microsoft-ds
1025/tcp  filtered NFS-or-IIS
1080/tcp  filtered socks
1433/tcp  filtered ms-sql-s
1434/tcp  filtered ms-sql-m
4242/tcp  filtered unknown
4444/tcp  filtered krb524
6346/tcp  filtered gnutella
6881/tcp  filtered bittorrent-tracker
8888/tcp  filtered sun-answerbook
45100/tcp filtered unknown
Device type: general purpose|WAP|broadband router
Running (JUST GUESSING) : Linux 2.6.X|2.4.X (95%), (…)
 
Not shown: 994 filtered ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
53/tcp   open  domain
80/tcp   open  http
110/tcp  open  pop3
3389/tcp open  ms-term-serv
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: firewall|general purpose
Running: Linux 2.6.X
OS details: Smoothwall firewall (Linux 2.6.16.53), Linux 2.6.13 - 2.6.24, Linux 2.6.16

Of course, it strikes me that several among said machines seem to be Linuxes, but (appear to) run Microsoft services. Oh, and they also have P2P clients.

I met my friend Josef Daberning, who did his Austrian Social Service working with Drupal at the Casa de los Tres Mundos NGO, in Granada, Nicaragua, at the Central American Free Software Encounter, last May. He told me that, when going back to Austria, he would spend some days in Mexico, and wanted to give a workshop on Drupal.

The course has just started, and will take place today and tomorrow — You can follow the live stream at http://www.iiec.unam.mx:18000/drupal.ogg — The videos will be uploaded soon as well, I will post them on this same node.

This node will be used for whatever is needed to make public for people following the talk. As of right now, you can download his presentation — http://gwolf.org/files/gira-drupal.odp and http://gwolf.org/files/gira-drupal.pdf

[update](s):

• If you are following the stream and want to say something, connect via IRC to OFTC (irc.debian.org) and join the #edusol channel.
• The videos for the talks will be made available soon, licensed under CC-BY 3.0 (generic)
• If you want to come, you are more than welcome. There is still space! We are in Ciudad Universitaria, South of Mexico City. Instituto de Investigaciones Económicas - UNAM. We are starting the second session (16:00-10:00), tomorrow we have a third and final session (10:00-14:00).
• Of course,it is all over by now. I will be processing the videos and uploading them to Archive.org's Open Source Movies archive. The three videos (for the three sessions) are available!

Oops.

Yesterday (Sunday, 31/08/09) I far from any computer-like object for most of the day. When I got back home, of course, I promptly opened my laptop to check my mail — who knows what destiny might have for me in a 24 hour period? Maybe I won yet another fortune I have to cash in Nigeria? Maybe there is (GASP!) a new RC bug on one of my packages?

But no, my mail server didn't feel like answering to my ssh queries. The connection was established, but shut down before even sending the protocolary SSH-2.0-OpenSSH_5.1p1 string. Fearing an overload (after all, the little bugger is just a Mac Mini running in another room in my house), I tried to check (via Web) its Munin status — Apache didn't want to listen either. It answered, but got only access denied. Things started worrying me… But (silly me) not enough — The machine runs headless1, so I just danced the boring raising elephants song2.

Allowed for a couple of minutes for everything to settle, and tried to connect. Horror, now even pings didn't work!

So I ran to fetch my old, bulky and trusty monitor. Went back to the machine, plugged it in, switched it off and back on. Everything worked fine this time — At least appearingly. I opened up mutt and started happily reading mails, while trying to understand on another console what happened at 07:06 that didn't get logged anywhere and had the machine dead for basically all the day. And then, BRRRT-BRRRT-BRRRT, I started hearing the HDD seeking.

I was able to send a couple of mails, but decided to let the machine rest and... Will reduce its disk usage to an absolute minimum. Fortunately, I have already the machine meant to replace it — A much nicer, beefier iMac G5, waiting to be vacated from its data, task which has suddenly become prioritary.

So, in short: If you need to get in touch with me in the next day or two, don't count on my usual @gwolf.org mail, as it is down. I hope to be able to get the data out of the poor little bugger painlessly after it rests a bit. And I hope not to drown in a sea of mails after I get the replacement back online :-/

• 1. for those not used to computerspeak: without a connected monitor
• 2. For those following at home who don't understand how Raising Elephants Is So Utterly Boring, this is a (silly, stupid but useful) mnemotecnic for as-properly-as-possible restarting hung Linux systems: Hold AltGr (right-Alt) + SysRq (in reduced/notebook keyboards, Fn+SysRq), and type R E I S U B (leaving a couple of seconds between key, ideally waiting for all disk activity to settle between each). That means:

  unRaw

  Take control of keyboard back from X,

  tErminate

  Send SIGTERM to all processes, allowing them to terminate gracefully,

  kIll

  send SIGKILL to all processes, forcing them to terminate immediately,

  Sync

  Flush data to disk),

  Unmount

  Remount all filesystems read-only,

  reBoot.

  Well... Reboot the system

Ufff...

I have spent a couple of hours connected from Norman García's house, in Managua. Norman is most kindly hosting me at home for a couple of days before we leave (tomorrow) for Estelí, where the Central American Free Software Encounter will be held.

Now, the network feels really slow. However, it can sustain download rates of around 512Kbps, quite acceptable. Latency is what kills. But... I was stunned with mtr's results to my home server:

 Host                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. speedtouch.lan                                           0.0%    16  101.1  51.0   3.2 101.1  32.2
 2. 1-0-212-190.enitel.net.ni                               12.5%    16  210.6 156.3  21.4 376.4  91.7
 3. 17-83-62-200.enitel.net.ni                              86.7%    16   23.0  22.8  22.6  23.0   0.2
 4. 10.192.100.49                                           66.7%    16  219.7 166.1  30.9 283.0  94.4
 5. 10.192.4.89                                             86.7%    16  136.7 106.7  76.7 136.7  42.5
 6. 10.192.4.82                                             66.7%    16  125.2 130.2  42.2 233.5  72.0
 7. 10.192.4.46                                             66.7%    16  249.9 130.3  55.2 249.9  75.3
 8. 10.192.4.50                                             92.9%    15   41.3  41.3  41.3  41.3   0.0
 9. 10.192.4.54                                             66.7%    15   85.6 138.8  85.6 169.7  33.4
10. 10.192.4.86                                             64.3%    15  141.1 154.9  82.4 264.0  66.8
11. 10.192.5.17                                             78.6%    15   86.4 150.3  74.1 290.5 121.5
12. 10.192.5.21                                             86.7%    15  112.6  78.0  43.4 112.6  49.0
13. 10.192.3.17                                             57.1%    15  225.1 152.6  52.5 246.1  77.7
14. 10.192.3.9                                              53.8%    14   69.5 151.4  69.5 235.6  52.8
15. 10.192.3.5                                              84.6%    14  236.0 141.1  46.1 236.0 134.3
16. 10.192.3.30                                             71.4%    14  116.3 161.2  39.4 258.0 101.9
17. 10.192.3.34                                             84.6%    14   67.9  52.5  37.1  67.9  21.8
18. 10.192.2.129                                            76.9%    14  258.2 172.0 112.6 258.2  76.4
19. 10.192.2.241                                            61.5%    14  159.8 117.3  42.3 159.8  51.8
20. 10.192.6.13                                             78.6%    14  128.1 192.4 128.1 240.4  57.9
21. 10.192.2.141                                            92.3%    14   63.8  63.8  63.8  63.8   0.0
22. 10.192.2.45                                             58.3%    13  249.3 196.3 136.3 249.3  46.2
23. 10.192.2.41                                             91.7%    13  260.5 260.5 260.5 260.5   0.0
24. 10.192.2.34                                             69.2%    13  231.1 208.7  82.5 260.7  85.3
25. ???
26. inet-pue-fuertes-24-pos6-2.uninet.net.mx                63.6%    12  158.0 172.0 111.3 261.2  63.4
27. dsl-mex-quevedo-2-pos2-0.uninet.net.mx                  90.9%    12  161.0 161.0 161.0 161.0   0.0
28. ???

Please, somebody explain the basics of routing to Claro/Enitel. This just does not make any sense.

You might have noticed that during last week the Mexican Debian mirror, nisamox.fciencias.unam.mx (a.k.a. ftp.mx.debian.org, a.k.a. debian.unam.mx), went offline. The motherboard died on us, and Facultad de Ciencias was kind enough to give us a brand new one. So, excuse us for the blackout, but we are back – Meaner and badder than ever before!

Now, Sergio (nisamox's main admin) prefered to rebuild the whole mirror, as there was a shadow of doubt regarding the data integrity. So, rsync was pulling as fast as he could for the whole weekend (leading to some people scratching their heads regarding the 404 for the missing files; sorry, we should have left Apache shut down until the mirror was complete!). After three days of sustaining a 10-20Mbps download from the main mirrors, all 364GB of Debian are finally installed and –as you can clearly see– we are back to normality, with small, regular mirror pulses and a nice sustained 5-10Mbps (with some up to 40Mbps peaks — We have seen up to 100Mbps peaks in the past, and I doubt with the current network infrastructure we won't top that).

You can see we have currently plenty of disk space still to fill up. Among our plans is to host the most popular ISOs, which are a common request, and... What else? Well, ask us and we shall do so (quite probably).

So, if you switched away from ftp.mx.debian.org due to our downtime, readjust your mirror settings. Nisamox is back!

The phrase on this title is often attributed to Pancho Villa (1878-1923), Mexican Revolution leader. He had a fame of cruelty, killing suspects before even questioning them.
Today, it started as a very nice day. I had even time in the morning to find, fix, upload and send upstream a trivial bug in libgruff-ruby... At 11:00, I left the Institute as my father came to the city to do some paperwork... We sat having a cup of coffee in a restaurant near the office we had went to at around 12:00, and my phone rang.
And it was from work. That's never a good sign. My boss told me he was facing a massive virus infection, and decided to disconnect the firewall. I corrected him - that will do no good once the virus is in our system, if you want to disconnect anything, disconnect all of our switches.
Came back, and found him and my coworker stunned and not knowing what to do. He says, the antivirus alarm went off almost simultaneously on the two computers he had on his desk, and in few minutes over 15 computers all over the Institute were ill. The symptoms? Programs not showing up in the taskbar, copy/paste functionality b0rken, many programs misbehaved or just didn't open... They were grimly facing a complete recovery operation they have grown used to: The whole OS has become corrupted or destroyed, we will have to open the computer, extract the HD, install it elsewhere, back it up, reinstall OS and applications, restore the backup. Yes, I know too many extra steps are included here, but I have come to accept their ways of dealing with Windows. Nobody says dealing with Windows is fun. I like my work to be fun, so I stay clear of theirs.
I insisted on turning back one one of the switches, the one for the servers and my machine (and some more in the same physical area). OK'd. But they didn't want to switch on any other switch, so a traffic capture (tcpdump / wireshark) led nowhere - but at least it gave my my Google back.
They have configured the antivirus software we deploy to all of the Windows machines in such a way that it deletes upon sight any malware - And when they manually scan, they blindly hit Delete whenever anything is found as well. Of course, no infected binary was left alive for me to inspect, and the machines were dead. But I was able to glimpse at the name of the deleted file: rpcss.dll.
After googling a bit - Bliss! Joy! I found the answer. So here is the set of interactions, and how they led to this killing spree. Please remember I am a Windows newbie and speak just out of guesswork.

1. This is a fast-spreading virus. My friend Rubén at DGSCA suggest it might be related to this report submitted today; at Barrapunto there is a thread about another virus that appeared four days ago, infected 1.1 million Windows machines on its first day, and so far is around the ninth million. Update: Equivalent thread at Slashdot, for the Spanish-impaired.
2. The virus infects at least two copies of a system binary: %system32%\rpcss.dll and \Windows\ServicePackFiles\i386\rpcss.dll. Windows uses the second one to restore the first one in case it is damaged, if I understood correctly.
3. The antivirus does not detect the infection when the library files are written, but when they are linked, so it only spots it the next time %system32%\rpcss.dll is brought into memory.
4. This is a very common library - It takes care of, well, RPC. So, quite probably, this file will be linked again on the next program launch - or accessed when a running program requires anything not currently in RAM? Dunno. The thing is, the library gets linked.
5. The antivirus will happily tell you it has killed a threat! Your nice RPC library is now defunct. ¡Mátalo, luego virigüas!
6. So, of course, notifying the taskbar of a new window appearing, or clipboard actions, or whatnot will refuse to work.
7. Machine restart, full system scan requested. The antivirus finds de second copy of this library in the master directory (\Windows\ServicePackFiles\i386). The virus used this location so that Windows won't restore a clean version over it. But yes, it will fall again under the claws of the antivirus... I guess. Anyway, the antivirus offers to delete this file as well, and does so.
8. User is desperate. My coworkers are desperate. I am... mildly annoyed?

Once I found this line of thought... I went to a working machine, inserted my flash memory, and copied %system32%\rpcss.dll to it. Went back to a sick machine, and ran cmd. Then, it was just matter of copy f:\rpcss.dll c:\windows\system32, a simple reboot (it never hurts to reboot in Windows!), and problem solved!
Oh, as a side rant: I find it extremely annoying and sad that many people I know, sometimes with more experience as a computer operator/supporter than what I have of experience as a living human being, are so scared of using a command-line interface. They were dismayed at seeing no drag-and-drop and no copy/paste functionality were available! copy is not an option.
Anyway... Today was an experience on how a simple, mostly-harmless and quite-fertile virus is able to be terribly magnified by the presence of a trigger-happy antivirus.
Why won't they give themselves a chance to try something else? Say, GNU/Linux? :-/

Wouter insists that Ruby Gems are enough of an argument to keep Rails at a distance. Even though I agree with the basic claim and think that Gems are basically insane and sick, this should be taken a bit more under perspective.
We are blessed. We are blessed to have Debian, such a rich OS with such a great package management system, and with superb integration between so many packages. Blessed are also the users of most Free Software based distributions, as they share the advantages of systems growing with full consciousness of their interaction's benefits. However, integration with the rest of the world is not seamless.
Most scripting languages have their own infrastructure for managing the modules/libraries/pacakges/whatevers dependencies. Perl has CPAN, PHP has PEAR, Ruby has Gems... I do see Gems as the most obnoxious of them all, but the basics are the same.
The Rails crowd started being Unix-centric, but the Windows (and MacOS - they are no better, believe me, at least in this regard) world has exerted its pressure. Gems caters very well to their needs, but we do suffer the integration at the distro side.
The only sane way the propietary-minded people have managed to stay clear of the well known "DLL hell" is to ship everything a given program requires bundled together - that's the main reason for the bloat of lots of applications, and for the sloppiness of security support. Every application packager is responsible for shipping updated versions to any library it bundles in, except for the very basic core that the OS itself provides. That seems so annoyingly backwards to us that... it is unbelievable.
So, yes, Rails application trees often include Rails itself. For $DEITY sake, even I have grown used to working that way, as things tend to break under your nose otherwise. My proposal (which we talked over at DebConf, but have not pushed so far) is to support simultaneous versions of Rails installed in a Debian system (of course, via different packages), more or less in the way that simultaneous versions of Ruby, PHP or Python (and, in some limited fashion, Perl - Although Perl does not suffer from this incompatible bumps. Yay for Perl!) can be installed.
...And, yes, together with the pkg-ruby-extras team, I have been trying to -slowly, yes- package whatever modules we often use so they don't have to be included in Rails applications.
So far, the best way (although by far not optimal) I have found to limit this explosion of trees is to include most libraries in my Rails application trees as git submodule trees - If they are not explicitly downloaded, the systemwide libraries will be used.
Yes, the "ship the whole thing as a bundle" approach is quite annoying. However, at least I must acknowledge that it works better than the approach I took with previous (mod_perl-based) webapps I wrote... As relatively few people grok mod_perl, I ended up with quite some apps which were not installed by anybody but me. Rails is obnoxious... but seems to be parsable by the humanity at large.