Damage control: Cleaning up compromised SSH keys
This morning, my laptop was stolen from my parked car while I was jogging. I do not want to make a big deal out of it.
Still, even though I am sure it was not targetted at my data (three other people at least were reporting similar facts in the same area), and the laptop’s disk will probably just be reformatted, I am trying to limit the possible impact of my cryptographic identification being in somebody else’s hands.
GPG makes it easy: I had on that machine just my old 1024D key, so it is just matter of generating a revocation certificate. I have done that, and uploaded it to the SKS keyservers - Anyway, here is my revocation certificate:
-----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: A revocation certificate should follow iHIEIBEIADIFAkyaOZwrHQJBIGNvbXB1dGVyIGNvbnRhaW5pbmcgdGhpcyBrZXkg d2FzIHN0b2xlbgAKCRDYDvNai7UnrzWAAKC34eF76JQjxrZqSjNwcC0dU/5VbACg gMIMmYg91Sl3y8KsZXdGj/rV7UE= =rdlT -----END PGP PUBLIC KEY BLOCK-----
But… What worries me more is access to the computers my ssh key works for. Yes, the ssh key uses a nontrivial passphrase, but still — SSH keys cannot be revoked (and this makes sense, as SSH should not add the delay, or potential impossibility, to check with a remote infrastructure whenever you want to start a session).
So, I generated a new key (and stored it at ~/.ssh/id_rsa.new / ~/.ssh/id_rsa.new.pub) and came up with this snippet:
$ OLDKEY=xyHywJuHD3nsfLh03G1TqUEBKSj6NlzMfB1T759haoAQ
$ for host in $(cat .ssh/known_hosts | cut -f 1 -d \ |cut -f 1 -d , |
sort | uniq); do
echo == $host
ssh-copy-id -i .ssh/id_rsa.new.pub $host &&
ssh $host "perl -n -i -e 'next if /$OLDKEY/;print' .ssh/authorized_keys"
done
Points about it you might scratch your head about:
- .ssh/known_hosts' lines start with the server's name (or names, if more than one, comma-separated), followed by the key algorithm and the key fingerprint (space-separated). That's the reason for the double cut – It could probably be better using a regex-enabled thingy understanding
/[, ]/
, but... I didn't think of that. Besides, the savings would be just for academic purposes ;-) - I thought about not having the ssh line conditionally depend on ssh-copy-id. But OTOH, this makes sure I only try to remove the old key from the servers it is present on, and that I don't start sending my new key everywhere just for the sake of it.
- my $OLDKEY (declared in Shell, and only literally interpolated in the Perl one-liner below) contains the final bits of my old key. It is long enough for me not to think I'm risking collision with any other key. Why did I choose that particular length? Oh, it was a mouse motion.
- perl -n -i -e is one of my favorite ways to invoke perl. -i means in-line editing, it allows me to modify a file on the fly. This line just skips (removes) any keys containing $OLDKEY; -n tells it to loop all the lines over the provided program (and very similarly, -p would add a print at the end – Which in this particular ocassion, I prefer not to have). It is a sed lookalike, if you wish, but with a full Perl behind.
Caveats:
- This assumes you have set HashKnownHosts: no in your .ssh/config. It is a tradeoff, after all – I use a lot tab-expansion (via bash_completion) for hostnames, so I do have the fully parseable list of hosts I have used on each of my computers.
- I have always requested my account names to be gwolf. If you use more than one username... well, you will have to probably do more than one run of it connecting to foo@$host instead.
- Although most multiuser servers stick to the usual port 22, many people change the ports (me included) either because they perceive concealing them gives extra security, or (as in my case) because they are fed up with random connection attempts. Those hosts are stored as [hostname]:port (i.e. [foo.gwolf.org]:22000). Of course, a little refinement takes care of it all.,/li>
- Oh, I am not storing results... I should, so for successive runs I won't try to connect to a system I already did, or that already denied me access. Why would I want to? Because some computers are not currently turned on. So I'll run this script at least a couple of times. </ul> Oh, by the way: If you noticed me knocking on your SSH ports... please disregard. Possibly at some point I connected to that machine to do something, or it landed in my .ssh/known_hosts for some reason. I currently have 144 hosts registered. I am sure I triggered at least one raised eyebrow. And I will do it from a couple of different computers, to make it less probable that I miss some I have never connected from while at the particular computer I am sitting at right now. So... Any ideas on how to make this better? ### Comments [Anonymous]() 2010-09-22 15:37:57 **Maybe you should adopt an** Maybe you should adopt an Ubuntu-like setup with a eCrypts-encrypted $HOME? http://blog.dustinkirkland.com/2009/02/how-encrypted-home-ecryptfs-works.html http://blog.dustinkirkland.com/2009/02/jaunty-encrypted-home-directories.html ----- [Anonymous]() 2010-09-22 17:54:00 **SSH config Host option** Maybe this doesn't scale well to the number of hosts you connect to as I have access to about an order of magnitude fewer hosts than you do, but every host I connect to has a Host entry in my .ssh/config file like Host shortname HostName shortname.example.com User myname Port 48247 Any of those can be left out for the defaults, of course (If HostName is left out there then it would try to find "shortname" with DNS.) Also you can set other per-host options if applicable (ex. I have certain hosts set to always forward agent or always forward X11). Bash tab-completes on the Hosts from that file, and they would be pretty easy to parse out in a script like yours. That said, I also have a different SSH key on each of my computers, so in a similar situation, I would just connect to each host and delete the line in authorized_keys with myname@stolen_computers_name at the end. Once again, this may only work well with few computers involved. ----- [Anonymous]() 2010-09-22 17:58:04 **SSH config Host option** Maybe this doesn't scale well to the number of hosts you connect to as I have access to about an order of magnitude fewer hosts than you do, but every host I connect to has a Host entry in my .ssh/config file like Host shortname HostName shortname.example.com User myname Port 48247 Any of those can be left out for the defaults, of course (If HostName is left out there then it would try to find "shortname" with DNS.) Also you can set other per-host options if applicable (ex. I have certain hosts set to always forward agent or always forward X11). Bash tab-completes on the Hosts from that file, and they would be pretty easy to parse out in a script like yours. That said, I also have a different SSH key on each of my computers, so in a similar situation, I would just connect to each host and delete the line in authorized_keys with myname@stolen_computers_name at the end. Once again, this may only work well with few computers involved. ----- [Anonymous]() 2010-09-23 22:36:02 **Encrypt your system, full** Encrypt your system, full disk encryption is not that bad, and I dont understand the reason for stopping doing it just because you suspend to RAM. Also, i found the monkeysphere to be a perfect solution to dealing with my ssh keys as easily as my pgp keys. try it out! ----- [Christian]() 2010-09-22 19:40:47 **Hard-drive encryption** To avoid such scenarios, I have my $HOME on a LUKS-encrypted partition which gets mounted on log-on via libpam-mount. I use a two-factor authentication: a keyfile on a USB stick, encrypted with my logon password. I can highly recommend this approach. This is fully transparent, and quite easy to implement. I could have gone for an encrypted root, too, but I didn't consider it worth the trouble. (Even the two-factor authentication might be pushing it). ----- [Christian]() 2010-09-23 05:39:35 **Encrypted homes and limitations**
I almost always store and carry my laptop suspended to RAM. Of course, there is still one layer of password-protection to do anything useful (i.e. the session is locked).This is indeed one of the downsides. However, newer cryptsetup versions support a luksSuspend and luksResume option. I haven't tested this feature because there are no drop-in scripts for pm-utils yet, unfortunately. ----- [Daniel Kahn Gillmor]() 2010-09-22 14:46:00 **monkeysphere offers ssh key revocation** That sucks, Gunnar! Looks like you're moving forward in a responsible fashion, but as you say, you won't necessarily get all the hosts taken care of (if they're offline or unreachable to you, for example). Note also that if you work in an institutional context (e.g. as a systems administrator), your key might have been deployed on other systems or in other accounts that you have never logged into, and so it wouldn't be listed in your known_hosts file. I just wanted to mention that the monkeysphere project offers the possibility of revocation for OpenSSH. If you had used monkeysphere to bind this ssh key to your OpenPGP identity (as an authentication-capable subkey), and the hosts in question were authenticating you by your OpenPGP User ID instead of by raw key material, it would be enough to revoke the key itself via the WoT, and all monkeysphere-authenticating services would refuse to recognize this key as valid for authenticating any account associated with you. This is a potential benefit for both the keyholder (less worry about someone compromising your accounts) and the service administrator (automated revocation of responsible user's keys means less worry about someone compromising your service). good luck with your revocations! i'm glad your new OpenPGP key is still secured. ----- [fcestrada@fcestrada.com](http://fcestrada.com) 2010-09-24 19:30:02 **¡Que poca madre!** ¡Que poca madre de quien te dió baje con la lap! :-( ----- [gwolf]() 2010-09-22 20:18:45 **Good point, thanks!** I changed my key on the Debian servers (which are managed by a mail-to-LDAP gateway), but didn't think of those. Ok, should do early tomorrow! ----- [gwolf]() 2010-09-22 20:19:06 **Encrypted homes and limitations** For a long time, I used a whole encrypted filesystem. LUKS (long integrated in the Debian installer) makes it very easy to set up, and to be honest, I never felt a speed difference. Still, I decided to give up on it – I almost always store and carry my laptop suspended to RAM. Of course, there is still one layer of password-protection to do anything useful (i.e. the session is locked). So, encrypting the whole filesystem gave me very little, to be honest. Of course, the difference would be made in wetware... and my wetware is too rigid on "do what's easier to you". ----- [jmw]() 2010-09-22 12:05:00 **It sucks that your machine** It sucks that your machine was stolen, I'm sorry to hear it :( I hope somebody honest comes across it and sends it back to you... (maybe you can console yourself with the thought that the criminal might boot the machine, wonder what this 'Debian' thing is and come to know freedom through it? or maybe I am wishing too much.) ----- [Kalman]() 2010-09-22 16:46:04 **HDD encryption** Encrypt you disk, or just your $HOME if you're not paranoid enough, your data will be safe. ----- [Matt Ford]() 2010-09-23 03:03:49 **I use git to manage my SSH** I use git to manage my SSH directory - a hook fixes permissions. Replacing your key is a
ssh $SERVER "cd .ssh && git pull"away. Plenty of flexibility on how to manage: something in .bashrc, a daily cron over a list of servers (star topology), as-and-when. The only issue is that your bound to find machines without git on. And then it fails, fails in a big useless way. WARNING: this script will nuke your .ssh directory ensure you check in the files from ssh you want first....
#!/usr/bin/perl # Matt Ford # Setup git repo use strict; use warnings; use File::Path; my $ssh_dir="$ENV{HOME}/.ssh"; my $git_dir="$ssh_dir/.git"; my $remote_repo='ssh://zzalsmf3@storage008.vidar.ngs.manchester.ac.uk/~zzalsmf3/git/ssh'; my $FH; # Remove existing $ssh_dir if ( -d $ssh_dir ) { rmtree($ssh_dir) or die "Cannot remove $ssh_dir: $!"; } # Create new bare bones repo and add remote repo to sync with mkpath($ssh_dir,{mode => 0700}) or die "Cannot make $ssh_dir: $!"; chdir($ssh_dir) or die "Cannot change to $ssh_dir: $!"; ! system(qw/git init/) or die "Cannot create git repo: $!"; # Add gitignore settings to prevent the wrong files getting into origin open ($FH,'>',".gitignore"); print $FH <<EOF; identity id_dsa id_rsa known_hosts EOF close $FH; # ! system(qw/git commit --allow-empty -m/, "Create empty Master branch") # or die "Cannot create empty master branch: $!"; ! system(qw/git add .gitignore/) or die "Cannot add .gitignore to repo:$!"; ! system(qw/git commit -m/,"Initial import") or die "Cannot commit to repo:$!"; ! system(qw/git remote add origin/, $remote_repo) or die "Cannot add remote repo: $!"; # Add remote branch open ($FH,">>","$git_dir/config") or die "Cannot open $git_dir: $!"; print $FH <<'EOF'; [branch "master"] remote = origin merge = refs/heads/master EOF close $FH; # Add git hooks to ensure ssh client files have correct permissions chdir("$git_dir/hooks") or die "Cannot change to $git_dir/hooks: $!"; foreach ("pre-commit","post-merge","post-checkout") { open ($FH,'>',$_); print $FH <<'EOF'; #!/bin/sh echo "Checking permissions..." $HOME/sysadm/ssh-fix-permissions EOF close $FH; chmod 0755, $_; }; # Pull down repo chdir($ssh_dir); ! system(qw/git pull/) or die "Cannot pull from git repo: $!";----- [Matthew W.S. Bell]() 2010-09-22 17:54:51 **SSH keys do rather seem to** SSH keys do rather seem to lack an authorisation layer/management system ----- [Tobias]() 2010-09-23 04:21:35 **Encrypted homes** This is the reason why I only use Suspend-to-Disk - My system is crypted any time the laptop is off. ----- [vicm3](http://blografia.net/vicm3) 2010-09-22 22:51:02 **It sucks** Badly... but knowing this country the robber probably don't even try to access the file system, go for formatting install W7 and sells... On the good side you now had the excuse to go for a i3|5|7 notebook... One extra thing if other machines where stolen and where on the same zone, without breaking on the ones that not have machines, the thing looks like or the robber had very good studied the zone or is someone working|studying|visiting|staying in your institute... ----- [Yarik]() 2010-09-22 19:39:27 **don't forget about web forms for the keys** I believe even on alioth you should enter your pub key in webinterface and then it would propagate into a real authorized_keys... so deleting it from there automagically would not suffice... other services with similar approach: github ports/usernames -- I just store those in .ssh/config ;) so I never have to use them directly in ssh/scp cmdlines