Tepatche - Automatic OpenBSD system patcher OpenBSD is my operating system of choice for highly exposed or security oriented servers. OpenBSD is, after all, the only operating system designed specifically with security as its first goal, and -quoting from www.openbsd.org- with "Only one remote hole in the default install, in more than 7 years" it becomes the obvious choice for any exposed server. No code written by humans, however, can be claimed to be perfect. OpenBSD is, of course, no exception. A new OpenBSD release comes out every six months, and during that period some flaws are always found. Being OpenBSD a system targeted at security, their core team releases a source code patch every time a bug is found and corrected. During the one time life period of any given release (see sidebar 1) an average of 30 patches are issued. PATCHING AN OpenBSD SYSTEM - THE STANDARD WAY OpenBSD was designed with simplicity as a base - As well as simple code tends to be more secure, a simple implementation tends to be and stay more secure than a complex one. An OpenBSD system administrator is expected to be a real Unix user, to learn from the tools he has access to. And, as part of this philosophy (and, of course, in order to reduce overhead for the OpenBSD developers, given the number of architectures supported) updates are distributed as source code patches - In order to install them, the system administrator must patch the OpenBSD sources and recompile and install the relevant portions of the system. This process, by the way, is very easy to carry out - Patches have full instructions on how to do this at their very beginning. Being a responsible OpenBSD system administrator and keeping the systems updated is quite easy - Subscribe to security-announce@openbsd.org, promptly download the patches, apply and compile them, and you are set. However, there are a couple of problems with this approach: - If you manage many OpenBSD machines, the process can be tedious. Applying the patches to a couple of machines is quite easy, but patching a whole data center is a process that can take up hours - specially if you check the status on each machine (as you should always do). - We humans don't always have time for doing all of our work. Patches can be relegated for some days, weeks... And even be completely forgotten despite our best intentions. This is specially important today, that automatic worms are scanning the whole network finding and compromising vulnerable systems. In order to address those two problems, I wrote Tepatche. I wrote the first version in June 2002, shortly after the -to this day- only known remote vulnerability in the default configuration of OpenBSD surfaced. Some blackhat groups promptly released exploits for this vunlerability - and many system administrators were left vulnerable. I decided to write Tepatche because, after all, periodically checking and applying simple patches is a task which a computer can do better than a a human. PROGRAM LOGIC Tepatche should be run automatically - I suggest running it daily as a cron job. Its operating logic is quite simple. Roughly: - Reads from the configuration file the address of the FTP site for updates (almost always ftp.openbsd.org/pub/OpenBSD/patches//) and starts up a connection to the site - Get the list of files available in both the common and architecture-specific directories of the FTP site. Compare it to our local copy, and if there are any new files, retrieve them and schedule them for processing - Process the patch instructions. Execute each patch instruction, checking if it is in the allowed set of instructions (explained in greater detail in the next section). - Notify the administrator of any changes made by sending him two mails: One with the full output of the execution, one with a short summary. The FTP connection is handled using Perl module Net::FTP, which can be either found in the CPAN or as p5-libnet in OpenBSD's ports tree. IMPORTANT DESIGN CONSIDERATIONS The program logic, as I said above, is plainly simple. The whole program is slightly over 400 lines of code, being roughly 25% comments. However, this is a program intended to be run as root, having a lot of interaction with the operating system, and intended to modify vital system binaries. This means that no amount of checking can be excessive here. First of all, I did my best in order to ensure that no error condition would go unnoticed, checking basically every statement that uses the FTP module, invokes an external command or deals with the filesystem. This resulted in almost 60 possible causes for early program termination - And, of course, early termination is always prominently reported to the system administrator. Secondly, as OpenBSD patches are distributed by FTP and they are not cryptographically signed, we can never really be sure if they were issued by a real OpenBSD developer or they are trojans - We must trust the file is legitimate in order for this process to be automated. As the patches' headers are basically recipes on how they are to be applied, I decided to deal with this trust problem being very selective on which commands will Tepatche automatically execute. The commands Tepatche will agree to run are: - cd: You must always change to the affected directory before patching - patch: The command which applies the patch - make: Compile and install the affected portion of the code This is enough for the vast majority of the patches - Not for all of them. One third of the patches in 3.2 involves killing a running process or removing some older files. Although it is currently not implemented, by the time you get this magazine I should have implemented a configuration option (of course, disabled by default) allowing Tepatche to execute any command listed on the patches' headers. Lastly, although most patches are applied to userland (i.e., the binaries, libraries and other files available in a Unix environment), some patches must be applied to the kernel. I decided Tepatche would only patch the kernel source tree, but would not rebuild the kernel. The reason for this is not only that many people custom-configure their kernels and it is not easy to come up with a sane standard version of compiling it, but because the only way to really apply it would be to reboot the system - and no mission-critical system should automatically reboot itself, even more when faced with the danger of having an incorrectly built, unbootable kernel. The kernel must always be rebuilt by the administrator. Tepatche will only alert him that he must do so. CONCLUSION I have been running Tepatche for over a year now, and although I do not administer many OpenBSD boxes, it has clearly improved my life as a SysAdmin. Knowing I am up to date with the patches, knowing that most patches will be automatically applied (or at least, I will get a kind notice from my computer telling me what to do) has made my life much easier. I have worked with Tepatche on OpenBSD versions 3.1 and 3.2 (as of today version 3.3 has not yet released a security patch) and have never had problems with it. Tepatche is free software, under the BSD license. You can download it from http://www.gwolf.cx/soft/tepatche/ ---------- Sidebar 1: OpenBSD releases' life period OpenBSD is a noncommercial project, led completely by volunteers. They follow a strict schedule of releasing two versions every year, in May and in November. Many commercial companies continue to support older versions of their products. In OpenBSD, this is simply not possible - it takes up too much of the developer's time, which can be much better used at hacking at new features or the ongoing security audit. The OpenBSD developers will always support two releases of OpenBSD - the latest stable version and the immediately previous version. After one year, old versions become deprecated and unmaintained. As OpenBSD is free software, this should not me much of a problem for administrators - And it guarantees the users that OpenBSD will continue being a high-quality, innovative, functional, secure and free opeating system. ---------- Sidebar 2: Why Tepatche? Tepache is a popular, slightly alcoholic drink in Mexico, where I live and where this program was devised. Tepache is the result of fermenting pineapple in water. Quoting from http://www.fao.org/docrep/x0560e/x0560e09.htm : Tepache is a light, refreshing beverage prepared and consumed throughout Mexico. In the past, tepache was prepared from maize, but nowadays various fruits such as pineapple, apple and orange are used. The pulp and juice of the fruit are allowed to ferment for one or two days in water with some added brown sugar. The mixture is contained in a lidless wooden barrel called a "tepachera", which is covered with cheese cloth. After a day or two, the tepache is a sweet and refreshing beverage. If fermentation is allowed to proceed longer, it turns into an alcoholic beverage and later into vinegar. The microorganisms associated with the product include Bacillus subtilis, B. graveolus and the yeasts, Torulopsis insconspicna, Saccharomyces cerevisiae and Candida queretana (Aidoo, 1986). One of the best ways for a sysadmin to get enough free time to go and enjoy some glasses of tepache is, precisely, by having their patching work automatized. If you are curious, you can find recipes to prepare tepache (in Spanish) at: http://www.chi.itesm.mx/chihuahua/arte_cultura/cocina/bebidas/tepache.html http://mexico.udg.mx/cocina/bebidas/tepache.html ---------- Sidebar 3: Other similar projects I know of two projects with goals similar to Tepatche's - although with a very different way of attaining them. One of them is Gerardo Santana Gómez's binary patches project (http://www.openbsd.org.mx/~santana/binpatch.html). Gerardo's main motivation is that it is not always feasible or desirable to compile from source in order to keep a running system up to date - If a server has too much processor load or too little hard disk space, or if we have tens of identical machines, distributing precompiled patches makes more sense than recompiling everywhere. Leonardo Costa wrote OpenBechede (http://openbechede.sourceforge.net/en/). OpenBechede aims at managing OpenBSD's ports infrastructure, handling dependency chains and updating them. The ports are not maintained together with the core OpenBSD system - for a better explanation on the ports, refer to http://www.openbsd.org/ports.html ---------- About the author: I have been a system administrator -either as a hobby or professionaly- since 1993, and security has always interested me. I have worked with and promoted Free Software since 1997, being actually one of Mexico's Free Software community's most active members.