Strange scanning on my server?
Humm… Has anybody else seen a pattern like this?
I am getting a flurry of root login attempts at my main server at the University since yesterday 7:30AM (GMT-5). Now, from the machines I run in the 132.248.0.0/16 network (UNAM), only two listen to the world with ssh at port 22 — And yes, it is a very large network, but I am only getting this pattern on one of them (they are on different subnets, quite far apart). They are all attempting to log in as root, with a frequency that varies wildly, but is consistently over three times a minute right now. This is a sample of what I get in my logs:
[update] Logs omitted from blog post, as it is too wide and breaks displays for most users. You can download the log file instead.
Anyway… This comes from all over the world, and all the attempts are made as root (no attempts from unprivileged users). Of course, I have PermitRootLogin to no in /etc/ssh/sshd_config, but… I want to understand this as much as possible.
Initially it struck me that most of the attempts appeared to come from Europe (quite atypical for the usual botnet distribution), so I passed my logs through:
#!/usr/bin/perl
use Geo::IP;
use IO::File;
use strict;
my ($geoip, $fh, %by_ip, %by_ctry);
$fh = IO::File->new(‘/tmp/sshd_log’); $geoip=Geo::IP->new(GEOIP_STANDARD); while (my $lin = <$fh>) { next unless $lin =~ /rhost=(\S+)/; $by_ip{$1}++};
print “ Incidence by IP:\n”, “Num Ctry IP\n”, (‘=’x60),”\n”;
for my $ip ( sort {$by_ip{$a} <=> $by_ip{$b}} keys %by_ip) { my $ctry = ($ip =~ /^[\d.]+$/) ? $geoip->country_code_by_addr($ip) : $geoip->country_code_by_name($ip);
$by_ctry{$ctry}++;
printf "%3d %3s %s\n", $by_ip{$ip}, $ctry, $ip; }
print “ Incidence by country:\n”, “Num Country\n”, “============\n”; map {printf “%3d %s\n”, $by_ctry{$}, $} sort {$by_ctry{$b} <=> $by_ctry{$a}} keys(%by_ctry); </code>
The top countries (where the number of attempts ≥ 5) are:
104 CN
78 US
58 BR
49 DE
43 PL
20 ES
20 IN
19 RU
17 CO
17 UA
16 IT
13 AR
12 ZA
10 CA
10 CH
8 GB
8 AT
8 JP
8 FR
7 KR
7 HK
7 PE
7 ID
6 PT
5 CZ
5 AU
5 BE
5 SE
5 RO
5 MX
I am attaching to this post the relevant log (filtering out all the information I could regarding legitimate users) as well as the full output. In case somebody has seen this kind of wormish botnetish behaviour lately… please comment.
[Update] I have tried getting some data regarding the attacking machines, running a simple nmap -O -vv against a random sample (five machines, I hope I am not being too agressive in anybody’s eyes). They all seem to be running some flavor of Linux (according to the OS fingerprinting), but the list of open ports varies wildly — I have seen the following:
Not shown: 979 closed ports
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
23/tcp open telnet
111/tcp open rpcbind
135/tcp filtered msrpc
139/tcp filtered netbios-ssn
445/tcp filtered microsoft-ds
593/tcp filtered http-rpc-epmap
992/tcp open telnets
1025/tcp filtered NFS-or-IIS
1080/tcp filtered socks
1433/tcp filtered ms-sql-s
1434/tcp filtered ms-sql-m
2049/tcp open nfs
4242/tcp filtered unknown
4444/tcp filtered krb524
6346/tcp filtered gnutella
6881/tcp filtered bittorrent-tracker
8888/tcp filtered sun-answerbook
10000/tcp open snet-sensor-mgmt
45100/tcp filtered unknown
Device type: general purpose|WAP|PBX
Running (JUST GUESSING) : Linux 2.6.X|2.4.X (96%), (…)
Not shown: 993 filtered ports PORT STATE SERVICE 22/tcp open ssh 25/tcp open smtp 80/tcp open http 443/tcp open https 444/tcp open snpp 3389/tcp open ms-term-serv 4125/tcp closed rww Device type: general purpose|phone|WAP|router Running (JUST GUESSING) : Linux 2.6.X (91%), (…)
Not shown: 994 filtered ports PORT STATE SERVICE 22/tcp open ssh 25/tcp closed smtp 53/tcp closed domain 80/tcp open http 113/tcp closed auth 443/tcp closed https Device type: general purpose Running (JUST GUESSING) : Linux 2.6.X (90%) OS fingerprint not ideal because: Didn’t receive UDP response. Please try again with -sSU Aggressive OS guesses: Linux 2.6.15 - 2.6.26 (90%), Linux 2.6.23 (89%), (…)
Not shown: 982 closed ports PORT STATE SERVICE 21/tcp open ftp 22/tcp open ssh 37/tcp open time 80/tcp open http 113/tcp open auth 135/tcp filtered msrpc 139/tcp filtered netbios-ssn 445/tcp filtered microsoft-ds 1025/tcp filtered NFS-or-IIS 1080/tcp filtered socks 1433/tcp filtered ms-sql-s 1434/tcp filtered ms-sql-m 4242/tcp filtered unknown 4444/tcp filtered krb524 6346/tcp filtered gnutella 6881/tcp filtered bittorrent-tracker 8888/tcp filtered sun-answerbook 45100/tcp filtered unknown Device type: general purpose|WAP|broadband router Running (JUST GUESSING) : Linux 2.6.X|2.4.X (95%), (…)
Not shown: 994 filtered ports PORT STATE SERVICE 22/tcp open ssh 25/tcp open smtp 53/tcp open domain 80/tcp open http 110/tcp open pop3 3389/tcp open ms-term-serv Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port Device type: firewall|general purpose Running: Linux 2.6.X OS details: Smoothwall firewall (Linux 2.6.16.53), Linux 2.6.13 - 2.6.24, Linux 2.6.16 </code>
Of course, it strikes me that several among said machines seem to be Linuxes, but (appear to) run Microsoft services. Oh, and they also have P2P clients.
Attachments
Relevant portion of the logs (1073 KB)
Results of parsing the logs (22 KB)
Comments
Anonymous 2009-10-01 22:37:00
Increase in ssh root access attempts
http://isc.sans.org/diary.html?storyid=7213
Anonymous 2009-10-01 23:29:00
I strongly suspect the
I strongly suspect the traffic you’ve observed represents the usual SSH worms: look for weak passwords, log into host as root, rootkit, recurse.
Anonymous 2009-10-02 06:19:00
Has anyone tried denyhosts in
Has anyone tried denyhosts in its “synchronization mode”?
Anonymous 2009-10-03 02:32:23
ISC report
I was going to refer to the ISC port report.
However at only another 500 sources, it is pretty small beer, so probably not fully automated, just another school kid trying his luck with SSH password guessing probably.
I don’t think the port scan is terribly revealing. All have port 22 listening, so an SSH based attack could explain their compromise.
I use a non-standard port for SSH on one virtual server, bit of a pain to remember to type it, but I get almost no abusive attempts.
Anonymous 2010-03-28 10:11:00
it like so much work its not even worth trying
granted im no code genius. in fact i purport it as magic the way a 1 and a 0 make color or movie. so when i paid way too much for a macbookpro with the caveat “no viruses” to later discover the root and have to invest in books because apples geniuses hush up if you mention root. at first i thought i pissed off some brilliant scientist like you mr wolf. nothing personal but it appears more focused than usual bot stuff. but maybe i retrofit that into my paranoia. certainly cutting edge engineers wouldnt have the time or care to read my email.
sorry for the off context entry and keep up the good work
aurel32 2009-10-02 00:27:00
Same pattern
I have the same problem on a machine from a university network (134.214.0.0/16 subnet), and it has started at about the same time.
Moreover some of the IP adresses from your logs match the ones that I have in my logs. It looks like we are hit by the same botnet.
bd_ 2009-10-01 21:36:00
Typically these are other
Typically these are other compromised servers acting in a botnet (and presumably trying to expand the botnet further). The reason you’re seeing attacks from so many hosts is to avoid ip-based rate limiters like fail2ban - by the time you can identify one as being malicious, it’s too late.
daily log 2009-10-06 08:18:00
In my servers with ssh
In my servers with ssh service enabled. It’s a common thing see this brute force. As long as you restrict the user who can connect through ssh. It’s safe.
garaged 2009-10-02 08:40:00
SMB exploit
There is an unpatched exploit against windows’s smb, I’m pretty sure its related to it.
It’s surprising that the attack is so well coordinated, looks like someone is trying to accomplish something special
gwolf 2009-10-01 20:37:05
Not blockable via that recipe…
Of course, I have the maximum rate limit set to 4 ssh connections per minute, as the article suggests… But here, by far most hosts have not even tried connecting four times in total. It’s more a huge number of hosts than anything else :-/
gwolf 2009-10-12 08:54:29
Don’t think so…
Why would this be related to SMB when it tries to connect as root to the ssh port? Even more: Where does ssh run with PermitRootLogin? On systems running openssh (the vast majority of ssh-enabled servers AFAICT), I would suppose it is blocked by default everywhere, and only few administrators open it. Maybe it is an attack against some other, vulnerable ssh server? Or it assumes that any administrator stupid enough to open root ssh connections to a publically accessable machine is dumb enough to use ‘root’ as the password?
gwolf 2009-10-12 08:59:06
fail2ban won’t work here
(sorry, I’m trimming your rather long list of hosts here)
In my case, servers are being scanned few times (usually one only) by each of the IPs, so fail2ban would not gain me much. Of course, there is a real reason to use fail2ban, but in this scenario it would help very little.
What I ended up doing is just to push my server to a non-standard port after notifying my users. This particular attack seems to be brute-forcing across the internet, and does not worry me too much, but it makes my logs longer to parse. I do not need ssh to be on port 22, so I’ll very happilly have it elsewhere.
gwolf 2009-10-12 08:59:56
It is safe, but…
It clutters up the logs. So I’d rather not have them. Even if the only way out is to stop using port 22.
Helmut Grohne 2009-10-02 02:24:00
Observed before
I observed a similar scanning on my server a bit ago. It looks like a distributed brute force attack designed to work around solutions like denyhosts or fail2ban. Solutions that come to my mind are changing the ssh port and port knocking. Both are disruptive to users. Indeed I did switch to a port knocking approach from denyhosts. You can mail me if you want more details about my solution.
Good luck with this
Helmut
imcsk8 2009-10-01 16:19:00
block them!!
Here’s a iptables recipe to limit te incoming ssh connections
http://www.debian-administration.org/articles/187
jorge 2012-03-01 09:09:41
GeoIP
No he tocado mucho Perl pero está interesante el módulo de GeoIP. Actualmente obtengo el país con consultas whois y filtrando el país o dirección: PAIS=$(whois “$IP” | grep ‘country’| awk ‘{print $2}’); DIRECCION=$(whois “$IP” | grep ‘address’);
PD: Normalmente el puerto 22 lo permito a IPs de donde me conecto y la política por defecto de mi fw es denegar.
kad 2009-10-01 17:41:54
There’s a simple explanation
They could be appliances, i.e. routers, which wouldn’t be surprisingly built on top of linux, although the traffic coming from inside their subnets could be very well be from windows machines.
Kai 2009-10-02 11:49:44
Same here
Found the same in my auth-log. However, noticed that one Host was actually a hosted server (reliablehosting.com).
Michael Goetze 2009-10-02 00:42:00
Home Router Botnet?
Maybe this is some kind of home router botnet? I would expect more of those to be deployed in Europe than in Asia.
sourchier 2009-10-02 03:44:00
ssh problems
Unfortunatly it is a worldwide problem. It it so bad that even SANS has notticed a problem. Please see here: http://isc.sans.org/diary.html?storyid=7213
I hope these attacks end soon.
Stefano Rivera 2009-10-02 15:52:22
Yup
Seeing identical scanning on the machine running www.za.debian.org. Also on a university network. No other machines I administer are getting this.
toxickore 2009-10-02 06:10:00
the theory from kad makes
the theory from kad makes sense. Could be machines inside a network running windows infected by worms or some malicious stuff.
vicm3 2009-10-12 18:43:32
logwatch, logcheck, systraq, etc…
And also logrotate, I had huge log files, but purge it every four months, and you had bigger and probably faster hdd’s ; ), I manage to take a look on my log with nested greps, also the mentioned script on the title helps with the logs.
vicm3 2009-10-01 19:24:00
Mhhh…
Mhh… I Fail2ban on physical machines (for some unknown reason the program seems to use to much CPU on a VM), and Denyhost for the VMs… yes it’s heavier than a rule on iptables, but is neat to have it summarized via logwatch on both cases… also both programs are python and where written before iptables include that option (predates iptables as fail2ban where written originally for ipchains) ; )
vicm3 2009-10-02 21:59:00
Fail2ban & Denyhost
Well as I have the PermitRootLogin = no on my machines, fail2ban & denyhost automatically ban all root login attempts, in my fail2ban settings for 1 hour if memory serves right, on my denyhost one month.
Anyway, except for backup purposes I don’t see why one must allow root login ssh to normal user then use su - to do admin work, or like has popularized ubuntu use sudo (when in ubuntu I still do sudo su - : )
For example from august to today.
DenyHosts: Sat Aug 8 02:19:23 2009 | sshd: 80.191.63.110
sshd: 80.191.63.110
DenyHosts: Sat Aug 8 03:28:27 2009 | sshd: 119.147.105.180
sshd: 119.147.105.180
(…)
DenyHosts: Fri Oct 2 19:59:56 2009 | sshd: 190.146.56.112
sshd: 190.146.56.112
Of course had a longer list, but these are the most recent, by the way I kept banned hosts for 8weeks… kind of ideas from me, as the VM’s are far away and the physical machines are 4 floors downstairs…