Stuff I have written/presented
Submitted by gwolf on Fri, 02/06/2015 - 12:51
I would expect brute-force login attacks to be more common. And yes, at some point I got tired of ssh scans, and added rate-limiting firewall rules, even switched the daemon to a nonstandard port... But I have very seldom received an IMAP brute-force attack. I have received countless phishing scams on my users, and I know some of them have bitten because the scammers then use their passwords on my servers to send tons of spam. Activity is clearly atypical.
Anyway, yesterday we got a brute-force attack on IMAP. A very childish atack, attempted from an IP in the largest ISP in Mexico, but using only usernames that would not belong in our culture (mosty English firstnames and some usual service account names).
What I find interesting to see is that each login was attempted a limited (and different) amount of times: Four account names were attempted only once, eight were attempted twice, and so on — following this pattern:
1 • 2 •• 3 •• 4 ••••• 5 ••••••• 6 •••••• 7 ••••• 8 •••••••• 9 ••••••••• 10 •••••••• 11 •••••••• 12 •••••••••• 13 ••••••• 14 •••••••••• 15 ••••••••• 16 •••••••••••• 17 ••••••••••• 18 •••••••••••••• 19 ••••••••••••••• 20 •••••••••••• 21 •••••••••••• 22 ••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
(each dot represents four attempts)
So... What's significant in all this? Very little, if anything at all. But for such a naïve login attack, it's interesting to see the number of attempted passwords per login varies so much. Yes, 273 (over ¼ of the total) did 22 requests, and another 200 were 18 and more. The rest... Fell quite shorter.
In case you want to play with the data, you can grab the list of attempts with the number of requests. I filtered out all other data, as i was basically meaningless. This file is the result of:
Submitted by gwolf on Tue, 04/29/2014 - 13:15
I have heard many good things about Docker, and decided to give it a spin on my systems. I think application-level virtualization has a lot to offer to my workflow...
But the process to understand and later adopt it has left me somewhat heart-torn.
Docker is clearly great technology, but its documentation is... Condescending and completely out of line with what I have grown used to in my years using Linux. First, there is so much simplistic self-praise sprinkled throughout it. There is almost no page I landed on that does not mention how user-friendly and user-centric Docker's commandline arguments are — They let you talk in almost plain1 English. What they don't mention is that... Well, that's the way of basically every command-line tool. Of course, as soon as you start specifying details to it, the plain-Englishness starts dilluting into a more realistic English-inspiredness...
Then... Things that go against our historical culture. It is often said that Windows documentation tends to be repetitive because users don't have the patience to read a full document. And our man pages are succint and to the point, because in our culture it is expected that users know how to search for the bit of information they are after. But reading documentation that's so excited with itself and praises again and again the same values and virtues, but never gets to the point I am interested in getting at (be it deployment, interoperation, description of the in-disk images+overlays layout, or anything moderately technical) never gets there... makes me quite unhappy.
Last (for now)... Such a continuous sales pitch, an insistence on the good virtues, makes me wary of something they might be hiding.
Anyway, at least for now, I just wanted to play a bit with it; I will wait at least until there is a backport to the stable Debian version before I consider moving my LXC VMs setup over to Docker (and a backport does not seem trivial to achieve, as Docker has several updated low-level dependencies we are unlikely to see in Wheezy).
But I had to vent this. OK, now go back to your regular work ;-)
Submitted by gwolf on Fri, 05/31/2013 - 13:23
I like Drupal. It's a very, very flexible CMS that evolved into a full-fledged Web development framework. Mind you, it's written in PHP, and that makes it a nightmare to develop for (in ~6 years I have used it for all of my important websites I have only got around to develop a set of related modules for it once).
PHP programming sucks and makes my eyes and fingers bleed, but happily there are people who disagree with me — And they tend to write code. All the better!
Minor upgrades with Drupal are quite easy to handle. Not as easy as I'd like (i.e. whenever I upgrade the core system or a module, I have to log in as
The updates that have to be run via this URL are usually on the database's structures, so I understand they have to be started (and watched) by a human. And yes, I know I could do that with Drush, the Drupal shell, but it is not very friendly to Debian-packaged Drupal... But easy enoguh.
But major updates are a royal pain, and they usually amount to quite a bit. First, disable all of the modules and revert to a known-safe theme. Ok, it makes sense. Second, check whether the modules exist for the newer version (as they won't work — Drupal changes enough between major versions that not only it's API-incompatible, I'd classify it as API-unrecognizable). Ok, all set? Now for the live migration itself... It has to be triggered from the browser.
So yes, I am now staring at a window making clever AJAX status updates. I am sitting at 46 of 199, but following the lovely ways of programmers, it's impossible to forsee whether update #47 will just be an UPDATE foo SET bar=0 WHERE bar IS NULL or a full-scale conversion between unspeakable serialized binary structures while rearranging the whole database structure.
And yes, while the meter progresses I stand in fear that update #n+1 will bomb giving me an ugly red error. I must keep the magic AJAX running, or the update might be botched.
And, of course, the update has sat at #69 all while I wrote the last two paragraphs. Sometimes the updates can progress after an interruption... And it seems I have no choice but to interrupt it.
/me crosses fingers...
[update] Wow... I am happy I got bored of looking at the meter and decided to write this blog post: After several minutes, and just as I was about to launch a second update session (130 updates to go), the meter advanced! I'm now sitting watching it at #75. Will it ever reach 199?
[update] And so it had to be... At around 115, I now got:
*sigh* The update process was aborted prematurely while running update #7000 in biblio.module...
Submitted by gwolf on Sat, 09/17/2011 - 13:06
Patrick tells about his experience moving from LVM to RAID.Now, why do this? I have two machines set up with LVM-based mirroring, and they work like a charm - I even think they work with better flexibility than setting it up in a RAID-controlled way, as each of the partitions in a volume group can be easily set to use (or stop using) the mirroring independently, and the requisite of having similar devices (regarding size) also disappears. Of course, this flexibility allows you to do very stupid things (such as setting up a mirror on two areas of the same rotational device - Good for toying around, but of course, never to be considered for production). And the ability to online grow and shrink partitions is just great.
So, Patrick, fellow readers, dear lazyweb, why would you prefer LVM-based mirroring to a RAID alternative? Or the other way around?
Submitted by gwolf on Wed, 09/29/2010 - 11:48
I cannot really blame (thoroughly) the Ruby guys for their position. After all, they have a vibrant community, and they are advancing great pieces of work. And they know who that code is meant for — Fellow programmers. And yes, although it is a pain to follow their API changes (and several of the Gems I regularly use do often get refactorings and functionality enhancements which break compatibility but introduce very nice new features), they say that's solved with one of Gems' main features being the simultaneous installability of different versions.
The key difference in Debian's worldview with Ruby's is they cater to Fellow programmers. Even leaving aside heaps of different positions and worldview/mindset, we have a fundamental difference: Debian cares about its users, whatever that means. So, our users should not even care what language a given application is implemented in – They should only care that it works. We, as packagers, should take care of all the infrastructural stuff.
And yes, that's where we find the conflicting spot: We don't want to ship many versions of a system library (that in this case would be a Gem). Specially if later versions fix known bugs in earlier versions and backports are not available or supported. Specially if upstream authors' only response to a bug in an older release will be "upgrade and rewrite whatever breaks in your application".
As an example of this, I am not currently updating the gems I maintain, as Debian is on a freeze to get out the next stable release. Or if at all, I am targetting those uploads to our Experimental branch, in order not to create a huge backlog for me when the freeze is over (just a series of rebuilds targetted at unstable). And yes, I will have to be responsible for any bugs that will most likely not be supported by most of my upstreams during the next ~2 years.
That's the role of a Linux distribution. And yes, as Lucas writes in the comments he got as responses to the first post – This dissonance comes in no small part because the Ruby developer community is mostly made from non-linuxers. People coming from a background where (mostly propietary) applications bundle up everything they need, where static linking is more popular than dynamic libraries, where there is no coordination between parts of the system are much less likely to understand our work.
And yes, the Perl community is a joy to work with in this regard. And that's the same I understand from the Python one. Because of their origins and where their main strength was grown and remains.
PS - And yes, I will join the flock of people saying that... The specific person that attacked your work is a great programmer, but well known as intolerant and obnoxious. Fortunately, even if our respective cultures fail to mix so much, most of our interactions just end with a "sigh" of lack of understanding, and not with the flames you got targetted with :-/
Submitted by gwolf on Wed, 09/22/2010 - 13:36
This morning, my laptop was stolen from my parked car while I was jogging. I do not want to make a big deal out of it.
Still, even though I am sure it was not targetted at my data (three other people at least were reporting similar facts in the same area), and the laptop's disk will probably just be reformatted, I am trying to limit the possible impact of my cryptographic identification being in somebody else's hands.
GPG makes it easy: I had on that machine just my old 1024D key, so it is just matter of generating a revocation certificate. I have done that, and uploaded it to the SKS keyservers - Anyway, here is my revocation certificate:
-----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: A revocation certificate should follow iHIEIBEIADIFAkyaOZwrHQJBIGNvbXB1dGVyIGNvbnRhaW5pbmcgdGhpcyBrZXkg d2FzIHN0b2xlbgAKCRDYDvNai7UnrzWAAKC34eF76JQjxrZqSjNwcC0dU/5VbACg gMIMmYg91Sl3y8KsZXdGj/rV7UE= =rdlT -----END PGP PUBLIC KEY BLOCK-----
But… What worries me more is access to the computers my ssh key works for. Yes, the ssh key uses a nontrivial passphrase, but still — SSH keys cannot be revoked (and this makes sense, as SSH should not add the delay, or potential impossibility, to check with a remote infrastructure whenever you want to start a session).
So, I generated a new key (and stored it at ~/.ssh/id_rsa.new / ~/.ssh/id_rsa.new.pub) and came up with this snippet:
Points about it you might scratch your head about:
Oh, by the way: If you noticed me knocking on your SSH ports... please disregard. Possibly at some point I connected to that machine to do something, or it landed in my .ssh/known_hosts for some reason. I currently have 144 hosts registered. I am sure I triggered at least one raised eyebrow.
And I will do it from a couple of different computers, to make it less probable that I miss some I have never connected from while at the particular computer I am sitting at right now.
So... Any ideas on how to make this better?
Submitted by gwolf on Mon, 03/08/2010 - 12:32
One of my favorite tricks using ssh is what Ganneff named ssh jumphosts – Many (most?) of my machines are not directly accessible from across the firewall, so the ability to specify in the configuration files where to jump through is most welcome. Well, with this "netcat mode" it will be much clearer to read and less of a hack… Of course, it loses a bit of the hackish æsthetic value, but becomes easier!
(yes, this post is basically a marker so I remember about it — But others might find it interesting)
Submitted by gwolf on Thu, 01/28/2010 - 08:35
Nobody cares about me, I thought. Whatever I say is just like throwing a bottle to the infinite ocean.
No comments, no hopes of getting any, for several days. Weeks maybe? Not even the spammers cared about me.
Until I read this mail, by Thijs Kinkhorst commenting to my yesterday post:
And, yes, Drupal module «captcha» introduced in its 2.1 release (January 2) feature #571344: Mix multiple fonts.
Only... no fonts were selected. Grah.
Submitted by gwolf on Wed, 01/27/2010 - 15:23
New guidelines for periodic publications' websites at my University favor the different journals we have to use a standardized system — And it makes quite a bit of sense. It is quite hard to explain to the people I work with that the content is not only meant to be consumed by humans, but also by other systems; the reasons behind rich content tagging and deep hierarchies for what they would just see as a list of words (think list of authors for an article, list of keywords, and so on). After all, aggregator databases such as Latindex and SciELO have achieved getting this understanding through.
And I must be quite grateful, as the University's guidelines point to what appears to be a very well-thought and thorough system, the Open Journal Systems by the Public Knowledge Project, co-funded by several well-regarded universities. OJS is a GPL-2-covered PHP bundle.
Anyway… I am very glad at least one of my Institute's journal accepted the challenge and decided to go OJS. I know I will quite probably be administering this system long-term. And, being as snobbish as I am, I know I loathe anything installed in my machines that is not either developed by myself or comes in a Debian package. So, as it was not packaged, I made the package ☺
Note that I am still not filing an ITP (which means, I have not yet decided whether I will upload this to Debian) because I want first to make sure I do have the needed long-term commitment — Besides, I am by far not a PHP person, and being responsible for a package… Carries a nontrivial weight. Still, you might be interested in getting it. If you are interested, you can either download the .deb package or add it to your apt repositories (and stay updated with any new releases), by adding this to your /etc/apt/sources.list:
Note: My packaging has still a small bug: The installer fails to create the PostgreSQL database. The MySQL database works fine. I will look into it soon
So far, I am quite impressed with this program's functionality and the depth/quality of its (online) documentation. Besides, its usage statistics speak for themselves:
So, it is quite possible I will be uploading this into Debian in a couple of weeks (hopefully in time to be considered for Squeeze). The reasons I am making it available in my personal repository now is:
Submitted by gwolf on Mon, 10/19/2009 - 23:42
Every now and then, people ask me why Debian? Why, among so many projects to choose from, I first liked, then got into, and finally I got committed into Debian, and not anything else?
Of course, one of the main points —back in 2000-2001 when I started using it, and still to this very day— is a strong identification with the ideological side. Yes, I am a strong Free Software believer, and Debian is what best suites my ideology.
Still, I did not only get into Debian because of this — And I was reminded about this by an article in this month's Usenix ;login: magazine: An anecdotal piece by Thomas A. Limoncelli titled Hey! I have to install and maintain this crap too, ya know! (article requires ;login: subscription, but I'll be glad to share it with whoever requests it to me — I have of course no permission to openly put it here in whole online. Yes, I am expressly sending a copy of this text to the author, I will update this if/when I hear from him) [update] The author has kindly allowed me to redistribute his article's PDF — Download it here.
Before anything else… I'll go on a short digression: I am writing a bit regarding the Free Software participants' culture, and this is a trait I love about it: The lack of formality. Even though ;login: (and Usenix as a whole) is not exactly Free Software, it runs quite close to it), it is a well regarded magazine (and association) with an academic format and good (not deep or highly theoretical, but good) contents. Still, it is quite usual to see titles as informal and inviting as this one. And it happens not only here — I have been fearing having to explain at work, over and over, why I have requesting permissions to go to Yet Another Perl Conference, Festival de Software Libre or DebCamp, tagging them as academic settings. Or why I am wasting our library's resources on buying cookbooks, recipes and similar material on the most strange-sounding subjects.
Anyway, back on track… This article I found refers to the lack of value given to the system administrator's time when selling or purchasing (or more in general, as it happens also in Free Software, when offering or adopting) a product. Quoting Thomas:
Thomas goes on to explain his experience with Silicon Graphics, how Irix was so great regarding install automation and how they blew it when switching to Windows NT; talks very briefly about IBM AIX's smit, a very nifty sysadmin aid which is basically a point-and-click interface to system administration with the very nice extra that allows you to view the commands smit executes to perform a given action (and then you can copy into a script and send over to your hundreds of AIX machines)… Incidentally, by the time I started digging out of what became the RedHat mess of the late 1990s and passed briefly through OpenBSD on my way to Debian enlightenment, I was temporarily the sysadmin for an AIX machine — And I too loved this Smit approach, having it as the ultimate pedagogical tool you could ever find.
Anyway, I won't comment and paraphrase the full article. I'll just point out to the fact that… this was what ultimately sold me into Debian. The fact that I could just install anything and (by far) most of the times it will be configured and ready to use. Debian made my life so much easier! As a sysadmin, I didn't have to download, browse documentation, scratch head, redo from start until I got a package working — Just apt-get into it, and I'd be set. Of course, one of the bits I learnt back then was that Debian was for lazy people — Everything works in a certain way. Policy is enforced throughout.
So as a sysadmin, I should better get well acquinted with the Debian policy and know it by heart. In order to be able to enjoy my laziness, I should read it and study it. And so I did, and fell in love. And that is where my journey into becoming a Debian Developer started.
Why am I talking so nostalgic here? Because I got this magazine on the mail just last weekend… And coincidentally, I also got bug report #551258 — I packaged and uploaded the Haml Ruby library (Gem, as the Rubyists would call it). Haml is a great, succint markup language which makes HTML generation less of a mess. It is even fun and amazing to write Haml, and the result is always nicely formatted, valid HTML! And well, one of Haml's components is haml-elisp, the Emacs Lisp major mode to do proper syntax highlighting in Haml files.
Of course, I am an Emacs guy (and have been for over 25 years), so I had to package it. But I don't do Emacs Lisp! So I just stuffed the file in its (supposed) place, copying some stuff over from other Emacs packages. During DebConf, I got the very valuable help of Axel Beckert to fix a simple bug which prevented my package from properly being installed, and thought I was basically done with it. I was happy just to add this to my ~/.emacs and get over with it:
However… As Mike Castleman points out: This requires manual intervention. So it is not the Debian Way!
Reading Mike's bug report, and reading Thomas' article, made me realize I was dilluting something I held so dearly as to commit myself to the best Free Software-based distribution out there. And the solution, of course, was very simple: Debian allows us to be very lazy, not only as sysadmins, but as Debian packagers. Just drop this (simplified version) as $pkgroot/debian/haml-elisp.emacsen.startup and you are set!
This will make the package just work as soon as it is installed, with no manual intervention required from the user. And it does not, contrary to what I feared, bloat up Emacs — Adding it to the auto-mode-alist leaves it as known to Emacs, but is not loaded or compiled unless it is required.
Deepest thanks to both of you! (and of course, thanks also to Manoj, for pointing out at the right spells in emacs-land)
Submitted by gwolf on Thu, 10/01/2009 - 18:04
Humm... Has anybody else seen a pattern like this?
I am getting a flurry of root login attempts at my main server at the University since yesterday 7:30AM (GMT-5). Now, from the machines I run in the 220.127.116.11/16 network (UNAM), only two listen to the world with ssh at port 22 — And yes, it is a very large network, but I am only getting this pattern on one of them (they are on different subnets, quite far apart). They are all attempting to log in as root, with a frequency that varies wildly, but is consistently over three times a minute right now. This is a sample of what I get in my logs:
[update] Logs omitted from blog post, as it is too wide and breaks displays for most users. You can download the log file instead.
Anyway… This comes from all over the world, and all the attempts are made as root (no attempts from unprivileged users). Of course, I have PermitRootLogin to no in /etc/ssh/sshd_config, but… I want to understand this as much as possible.
Initially it struck me that most of the attempts appeared to come from Europe (quite atypical for the usual botnet distribution), so I passed my logs through:
The top countries (where the number of attempts ≥ 5) are:
I am attaching to this post the relevant log (filtering out all the information I could regarding legitimate users) as well as the full output. In case somebody has seen this kind of wormish botnetish behaviour lately… please comment.
[Update] I have tried getting some data regarding the attacking machines, running a simple nmap -O -vv against a random sample (five machines, I hope I am not being too agressive in anybody's eyes). They all seem to be running some flavor of Linux (according to the OS fingerprinting), but the list of open ports varies wildly — I have seen the following:
Of course, it strikes me that several among said machines seem to be Linuxes, but (appear to) run Microsoft services. Oh, and they also have P2P clients.
Submitted by gwolf on Wed, 09/02/2009 - 11:05
I met my friend Josef Daberning, who did his Austrian Social Service working with Drupal at the Casa de los Tres Mundos NGO, in Granada, Nicaragua, at the Central American Free Software Encounter, last May. He told me that, when going back to Austria, he would spend some days in Mexico, and wanted to give a workshop on Drupal.
The course has just started, and will take place today and tomorrow — You can follow the live stream at http://www.iiec.unam.mx:18000/drupal.ogg — The videos will be uploaded soon as well, I will post them on this same node.
This node will be used for whatever is needed to make public for people following the talk. As of right now, you can download his presentation — http://gwolf.org/files/gira-drupal.odp and http://gwolf.org/files/gira-drupal.pdf
Submitted by gwolf on Mon, 08/31/2009 - 10:56
Yesterday (Sunday, 31/08/09) I far from any computer-like object for most of the day. When I got back home, of course, I promptly opened my laptop to check my mail — who knows what destiny might have for me in a 24 hour period? Maybe I won yet another fortune I have to cash in Nigeria? Maybe there is (GASP!) a new RC bug on one of my packages?
But no, my mail server didn't feel like answering to my ssh queries. The connection was established, but shut down before even sending the protocolary SSH-2.0-OpenSSH_5.1p1 string. Fearing an overload (after all, the little bugger is just a Mac Mini running in another room in my house), I tried to check (via Web) its Munin status — Apache didn't want to listen either. It answered, but got only access denied. Things started worrying me… But (silly me) not enough — The machine runs headless1, so I just danced the boring raising elephants song2.
Allowed for a couple of minutes for everything to settle, and tried to connect. Horror, now even pings didn't work!
So I ran to fetch my old, bulky and trusty monitor. Went back to the machine, plugged it in, switched it off and back on. Everything worked fine this time — At least appearingly. I opened up mutt and started happily reading mails, while trying to understand on another console what happened at 07:06 that didn't get logged anywhere and had the machine dead for basically all the day. And then, BRRRT-BRRRT-BRRRT, I started hearing the HDD seeking.
I was able to send a couple of mails, but decided to let the machine rest and... Will reduce its disk usage to an absolute minimum. Fortunately, I have already the machine meant to replace it — A much nicer, beefier iMac G5, waiting to be vacated from its data, task which has suddenly become prioritary.
So, in short: If you need to get in touch with me in the next day or two, don't count on my usual @gwolf.org mail, as it is down. I hope to be able to get the data out of the poor little bugger painlessly after it rests a bit. And I hope not to drown in a sea of mails after I get the replacement back online :-/
Submitted by gwolf on Mon, 06/15/2009 - 22:37
I have spent a couple of hours connected from Norman García's house, in Managua. Norman is most kindly hosting me at home for a couple of days before we leave (tomorrow) for Estelí, where the Central American Free Software Encounter will be held.
Now, the network feels really slow. However, it can sustain download rates of around 512Kbps, quite acceptable. Latency is what kills. But... I was stunned with mtr's results to my home server:
Please, somebody explain the basics of routing to Claro/Enitel. This just does not make any sense.
Submitted by gwolf on Tue, 05/19/2009 - 14:39
You might have noticed that during last week the Mexican Debian mirror, nisamox.fciencias.unam.mx (a.k.a. ftp.mx.debian.org, a.k.a. debian.unam.mx), went offline. The motherboard died on us, and Facultad de Ciencias was kind enough to give us a brand new one. So, excuse us for the blackout, but we are back – Meaner and badder than ever before!
Now, Sergio (nisamox's main admin) prefered to rebuild the whole mirror, as there was a shadow of doubt regarding the data integrity. So, rsync was pulling as fast as he could for the whole weekend (leading to some people scratching their heads regarding the 404 for the missing files; sorry, we should have left Apache shut down until the mirror was complete!). After three days of sustaining a 10-20Mbps download from the main mirrors, all 364GB of Debian are finally installed and –as you can clearly see– we are back to normality, with small, regular mirror pulses and a nice sustained 5-10Mbps (with some up to 40Mbps peaks — We have seen up to 100Mbps peaks in the past, and I doubt with the current network infrastructure we won't top that).
You can see we have currently plenty of disk space still to fill up. Among our plans is to host the most popular ISOs, which are a common request, and... What else? Well, ask us and we shall do so (quite probably).
So, if you switched away from ftp.mx.debian.org due to our downtime, readjust your mirror settings. Nisamox is back!
Talks, papers and documents by category
Blog posts by category