Damage control: Cleaning up compromised SSH keys

Submitted by gwolf on Wed, 09/22/2010 - 13:36

This morning, my laptop was stolen from my parked car while I was jogging. I do not want to make a big deal out of it.

Still, even though I am sure it was not targetted at my data (three other people at least were reporting similar facts in the same area), and the laptop's disk will probably just be reformatted, I am trying to limit the possible impact of my cryptographic identification being in somebody else's hands.

GPG makes it easy: I had on that machine just my old 1024D key, so it is just matter of generating a revocation certificate. I have done that, and uploaded it to the SKS keyservers - Anyway, here is my revocation certificate:

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: A revocation certificate should follow

iHIEIBEIADIFAkyaOZwrHQJBIGNvbXB1dGVyIGNvbnRhaW5pbmcgdGhpcyBrZXkg
d2FzIHN0b2xlbgAKCRDYDvNai7UnrzWAAKC34eF76JQjxrZqSjNwcC0dU/5VbACg
gMIMmYg91Sl3y8KsZXdGj/rV7UE=
=rdlT
-----END PGP PUBLIC KEY BLOCK-----

But… What worries me more is access to the computers my ssh key works for. Yes, the ssh key uses a nontrivial passphrase, but still — SSH keys cannot be revoked (and this makes sense, as SSH should not add the delay, or potential impossibility, to check with a remote infrastructure whenever you want to start a session).

So, I generated a new key (and stored it at ~/.ssh/id_rsa.new / ~/.ssh/id_rsa.new.pub) and came up with this snippet:

  1. $ OLDKEY=xyHywJuHD3nsfLh03G1TqUEBKSj6NlzMfB1T759haoAQ
  2. $ for host in $(cat .ssh/known_hosts | cut -f 1 -d \ |cut -f 1 -d , |
  3. sort | uniq); do
  4. echo == $host
  5. ssh-copy-id -i .ssh/id_rsa.new.pub $host &&
  6. ssh $host "perl -n -i -e 'next if /$OLDKEY/;print' .ssh/authorized_keys"
  7. done

Points about it you might scratch your head about:

  • .ssh/known_hosts' lines start with the server's name (or names, if more than one, comma-separated), followed by the key algorithm and the key fingerprint (space-separated). That's the reason for the double cut – It could probably be better using a regex-enabled thingy understanding /[, ]/, but... I didn't think of that. Besides, the savings would be just for academic purposes ;-)
  • I thought about not having the ssh line conditionally depend on ssh-copy-id. But OTOH, this makes sure I only try to remove the old key from the servers it is present on, and that I don't start sending my new key everywhere just for the sake of it.
  • my $OLDKEY (declared in Shell, and only literally interpolated in the Perl one-liner below) contains the final bits of my old key. It is long enough for me not to think I'm risking collision with any other key. Why did I choose that particular length? Oh, it was a mouse motion.
  • perl -n -i -e is one of my favorite ways to invoke perl. -i means in-line editing, it allows me to modify a file on the fly. This line just skips (removes) any keys containing $OLDKEY; -n tells it to loop all the lines over the provided program (and very similarly, -p would add a print at the end – Which in this particular ocassion, I prefer not to have). It is a sed lookalike, if you wish, but with a full Perl behind.

Caveats:

  • This assumes you have set HashKnownHosts: no in your .ssh/config. It is a tradeoff, after all – I use a lot tab-expansion (via bash_completion) for hostnames, so I do have the fully parseable list of hosts I have used on each of my computers.
  • I have always requested my account names to be gwolf. If you use more than one username... well, you will have to probably do more than one run of it connecting to foo@$host instead.
  • Although most multiuser servers stick to the usual port 22, many people change the ports (me included) either because they perceive concealing them gives extra security, or (as in my case) because they are fed up with random connection attempts. Those hosts are stored as [hostname]:port (i.e. [foo.gwolf.org]:22000). Of course, a little refinement takes care of it all.,/li>
  • Oh, I am not storing results... I should, so for successive runs I won't try to connect to a system I already did, or that already denied me access. Why would I want to? Because some computers are not currently turned on. So I'll run this script at least a couple of times.

Oh, by the way: If you noticed me knocking on your SSH ports... please disregard. Possibly at some point I connected to that machine to do something, or it landed in my .ssh/known_hosts for some reason. I currently have 144 hosts registered. I am sure I triggered at least one raised eyebrow.

And I will do it from a couple of different computers, to make it less probable that I miss some I have never connected from while at the particular computer I am sitting at right now.

So... Any ideas on how to make this better?

( categories: )
jmw's picture

It sucks that your machine

It sucks that your machine was stolen, I'm sorry to hear it :( I hope somebody honest comes across it and sends it back to you...

(maybe you can console yourself with the thought that the criminal might boot the machine, wonder what this 'Debian' thing is and come to know freedom through it? or maybe I am wishing too much.)

Daniel Kahn Gillmor's picture

monkeysphere offers ssh key revocation

That sucks, Gunnar! Looks like you're moving forward in a responsible fashion, but as you say, you won't necessarily get all the hosts taken care of (if they're offline or unreachable to you, for example). Note also that if you work in an institutional context (e.g. as a systems administrator), your key might have been deployed on other systems or in other accounts that you have never logged into, and so it wouldn't be listed in your known_hosts file.

I just wanted to mention that the monkeysphere project offers the possibility of revocation for OpenSSH.

If you had used monkeysphere to bind this ssh key to your OpenPGP identity (as an authentication-capable subkey), and the hosts in question were authenticating you by your OpenPGP User ID instead of by raw key material, it would be enough to revoke the key itself via the WoT, and all monkeysphere-authenticating services would refuse to recognize this key as valid for authenticating any account associated with you.

This is a potential benefit for both the keyholder (less worry about someone compromising your accounts) and the service administrator (automated revocation of responsible user's keys means less worry about someone compromising your service).

good luck with your revocations! i'm glad your new OpenPGP key is still secured.

Anonymous's picture

Maybe you should adopt an

gwolf's picture

Encrypted homes and limitations

For a long time, I used a whole encrypted filesystem. LUKS (long integrated in the Debian installer) makes it very easy to set up, and to be honest, I never felt a speed difference. Still, I decided to give up on it – I almost always store and carry my laptop suspended to RAM. Of course, there is still one layer of password-protection to do anything useful (i.e. the session is locked).

So, encrypting the whole filesystem gave me very little, to be honest. Of course, the difference would be made in wetware... and my wetware is too rigid on "do what's easier to you".

Tobias's picture

Encrypted homes

This is the reason why I only use Suspend-to-Disk - My system is crypted any time the laptop is off.

Kalman's picture

HDD encryption

Encrypt you disk, or just your $HOME if you're not paranoid enough, your data will be safe.

Anonymous's picture

SSH config Host option

Maybe this doesn't scale well to the number of hosts you connect to as I have access to about an order of magnitude fewer hosts than you do, but every host I connect to has a Host entry in my .ssh/config file like

Host shortname
HostName shortname.example.com
User myname
Port 48247

Any of those can be left out for the defaults, of course (If HostName is left out there then it would try to find "shortname" with DNS.) Also you can set other per-host options if applicable (ex. I have certain hosts set to always forward agent or always forward X11).

Bash tab-completes on the Hosts from that file, and they would be pretty easy to parse out in a script like yours.

That said, I also have a different SSH key on each of my computers, so in a similar situation, I would just connect to each host and delete the line in authorized_keys with myname@stolen_computers_name at the end. Once again, this may only work well with few computers involved.

Matthew W.S. Bell's picture

SSH keys do rather seem to

SSH keys do rather seem to lack an authorisation layer/management system

Anonymous's picture

SSH config Host option

Maybe this doesn't scale well to the number of hosts you connect to as I have access to about an order of magnitude fewer hosts than you do, but every host I connect to has a Host entry in my .ssh/config file like

Host shortname
HostName shortname.example.com
User myname
Port 48247

Any of those can be left out for the defaults, of course (If HostName is left out there then it would try to find "shortname" with DNS.) Also you can set other per-host options if applicable (ex. I have certain hosts set to always forward agent or always forward X11).

Bash tab-completes on the Hosts from that file, and they would be pretty easy to parse out in a script like yours.

That said, I also have a different SSH key on each of my computers, so in a similar situation, I would just connect to each host and delete the line in authorized_keys with myname@stolen_computers_name at the end. Once again, this may only work well with few computers involved.

Yarik's picture

don't forget about web forms for the keys

I believe even on alioth you should enter your pub key in webinterface and then it would propagate into a real authorized_keys... so deleting it from there automagically would not suffice... other services with similar approach: github

ports/usernames -- I just store those in .ssh/config ;) so I never have to use them directly in ssh/scp cmdlines

gwolf's picture

Good point, thanks!

I changed my key on the Debian servers (which are managed by a mail-to-LDAP gateway), but didn't think of those. Ok, should do early tomorrow!

Christian's picture

Hard-drive encryption

To avoid such scenarios, I have my $HOME on a LUKS-encrypted partition which gets mounted on log-on via libpam-mount. I use a two-factor authentication: a keyfile on a USB stick, encrypted with my logon password.

I can highly recommend this approach. This is fully transparent, and quite easy to implement. I could have gone for an encrypted root, too, but I didn't consider it worth the trouble. (Even the two-factor authentication might be pushing it).

vicm3's picture

It sucks

Badly... but knowing this country the robber probably don't even try to access the file system, go for formatting install W7 and sells...

On the good side you now had the excuse to go for a i3|5|7 notebook...

One extra thing if other machines where stolen and where on the same zone, without breaking on the ones that not have machines, the thing looks like or the robber had very good studied the zone or is someone working|studying|visiting|staying in your institute...

Matt Ford's picture

I use git to manage my SSH

I use git to manage my SSH directory - a hook fixes permissions.

Replacing your key is a

  ssh $SERVER "cd .ssh && git pull" 

away.

Plenty of flexibility on how to manage: something in .bashrc, a daily cron over a list of servers (star topology), as-and-when.

The only issue is that your bound to find machines without git on. And then it fails, fails in a big useless way.

WARNING: this script will nuke your .ssh directory ensure you check in the files from ssh you want first....

#!/usr/bin/perl
# Matt Ford
# Setup git repo

use strict;
use warnings;
use File::Path;

my $ssh_dir="$ENV{HOME}/.ssh";
my $git_dir="$ssh_dir/.git";
my $remote_repo='ssh://zzalsmf3@storage008.vidar.ngs.manchester.ac.uk/~zzalsmf3/git/ssh';
my $FH;

# Remove existing $ssh_dir
if ( -d $ssh_dir ) {
    rmtree($ssh_dir) or die "Cannot remove $ssh_dir: $!";
}

# Create new bare bones repo and add remote repo to sync with
mkpath($ssh_dir,{mode => 0700}) or die "Cannot make $ssh_dir: $!";
chdir($ssh_dir)
    or die "Cannot change to $ssh_dir: $!";
! system(qw/git init/) 
    or die "Cannot create git repo: $!";
# Add gitignore settings to prevent the wrong files getting into origin
open ($FH,'>',".gitignore");
print $FH <>","$git_dir/config") 
    or die "Cannot open $git_dir: $!";
print $FH <<'EOF';
[branch "master"]
    remote = origin
    merge = refs/heads/master
EOF
close $FH;

# Add git hooks to ensure ssh client files have correct permissions
chdir("$git_dir/hooks") 
    or die "Cannot change to $git_dir/hooks: $!";
foreach ("pre-commit","post-merge","post-checkout") {
    open ($FH,'>',$_);
    print $FH <<'EOF';
#!/bin/sh
echo "Checking permissions..."
$HOME/sysadm/ssh-fix-permissions
EOF
    close $FH;
    chmod 0755, $_;
};


# Pull down repo
chdir($ssh_dir);
! system(qw/git pull/) 
    or die "Cannot pull from git repo: $!";
Christian's picture

Encrypted homes and limitations

I almost always store and carry my laptop suspended to RAM. Of course, there is still one layer of password-protection to do anything useful (i.e. the session is locked).

This is indeed one of the downsides. However, newer cryptsetup versions support a luksSuspend and luksResume option. I haven't tested this feature because there are no drop-in scripts for pm-utils yet, unfortunately.

Anonymous's picture

Encrypt your system, full

Encrypt your system, full disk encryption is not that bad, and I dont understand the reason for stopping doing it just because you suspend to RAM.

Also, i found the monkeysphere to be a perfect solution to dealing with my ssh keys as easily as my pgp keys. try it out!

fcestrada@fcestrada.com's picture

¡Que poca madre!

¡Que poca madre de quien te dió baje con la lap! :-(