Strange scanning on my server?

Submitted by gwolf on Thu, 10/01/2009 - 18:04

Humm... Has anybody else seen a pattern like this?

I am getting a flurry of root login attempts at my main server at the University since yesterday 7:30AM (GMT-5). Now, from the machines I run in the 132.248.0.0/16 network (UNAM), only two listen to the world with ssh at port 22 — And yes, it is a very large network, but I am only getting this pattern on one of them (they are on different subnets, quite far apart). They are all attempting to log in as root, with a frequency that varies wildly, but is consistently over three times a minute right now. This is a sample of what I get in my logs:

[update] Logs omitted from blog post, as it is too wide and breaks displays for most users. You can download the log file instead.

Anyway… This comes from all over the world, and all the attempts are made as root (no attempts from unprivileged users). Of course, I have PermitRootLogin to no in /etc/ssh/sshd_config, but… I want to understand this as much as possible.

Initially it struck me that most of the attempts appeared to come from Europe (quite atypical for the usual botnet distribution), so I passed my logs through:

  1. #!/usr/bin/perl
  2. use Geo::IP;
  3. use IO::File;
  4. use strict;
  5. my ($geoip, $fh, %by_ip, %by_ctry);
  6.  
  7. $fh = IO::File->new('/tmp/sshd_log');
  8. $geoip=Geo::IP->new(GEOIP_STANDARD);
  9. while (my $lin = <$fh>) { next unless $lin =~ /rhost=(\S+)/; $by_ip{$1}++};
  10.  
  11. print " Incidence by IP:\n", "Num Ctry IP\n", ('='x60),"\n";
  12.  
  13. for my $ip ( sort {$by_ip{$a} <=> $by_ip{$b}} keys %by_ip) {
  14. my $ctry = ($ip =~ /^[\d\.]+$/) ?
  15. $geoip->country_code_by_addr($ip) :
  16. $geoip->country_code_by_name($ip);
  17.  
  18. $by_ctry{$ctry}++;
  19. printf "%3d %3s %s\n", $by_ip{$ip}, $ctry, $ip;
  20. }
  21.  
  22. print " Incidence by country:\n", "Num Country\n", "============\n";
  23. map {printf "%3d %s\n", $by_ctry{$_}, $_}
  24. sort {$by_ctry{$b} <=> $by_ctry{$a}}
  25. keys(%by_ctry);

The top countries (where the number of attempts ≥ 5) are:

  1. 104 CN
  2. 78 US
  3. 58 BR
  4. 49 DE
  5. 43 PL
  6. 20 ES
  7. 20 IN
  8. 19 RU
  9. 17 CO
  10. 17 UA
  11. 16 IT
  12. 13 AR
  13. 12 ZA
  14. 10 CA
  15. 10 CH
  16. 8 GB
  17. 8 AT
  18. 8 JP
  19. 8 FR
  20. 7 KR
  21. 7 HK
  22. 7 PE
  23. 7 ID
  24. 6 PT
  25. 5 CZ
  26. 5 AU
  27. 5 BE
  28. 5 SE
  29. 5 RO
  30. 5 MX

I am attaching to this post the relevant log (filtering out all the information I could regarding legitimate users) as well as the full output. In case somebody has seen this kind of wormish botnetish behaviour lately… please comment.

[Update] I have tried getting some data regarding the attacking machines, running a simple nmap -O -vv against a random sample (five machines, I hope I am not being too agressive in anybody's eyes). They all seem to be running some flavor of Linux (according to the OS fingerprinting), but the list of open ports varies wildly — I have seen the following:

  1. Not shown: 979 closed ports
  2. PORT STATE SERVICE
  3. 21/tcp open ftp
  4. 22/tcp open ssh
  5. 23/tcp open telnet
  6. 111/tcp open rpcbind
  7. 135/tcp filtered msrpc
  8. 139/tcp filtered netbios-ssn
  9. 445/tcp filtered microsoft-ds
  10. 593/tcp filtered http-rpc-epmap
  11. 992/tcp open telnets
  12. 1025/tcp filtered NFS-or-IIS
  13. 1080/tcp filtered socks
  14. 1433/tcp filtered ms-sql-s
  15. 1434/tcp filtered ms-sql-m
  16. 2049/tcp open nfs
  17. 4242/tcp filtered unknown
  18. 4444/tcp filtered krb524
  19. 6346/tcp filtered gnutella
  20. 6881/tcp filtered bittorrent-tracker
  21. 8888/tcp filtered sun-answerbook
  22. 10000/tcp open snet-sensor-mgmt
  23. 45100/tcp filtered unknown
  24. Device type: general purpose|WAP|PBX
  25. Running (JUST GUESSING) : Linux 2.6.X|2.4.X (96%), ()
  26.  
  27.  
  28. Not shown: 993 filtered ports
  29. PORT STATE SERVICE
  30. 22/tcp open ssh
  31. 25/tcp open smtp
  32. 80/tcp open http
  33. 443/tcp open https
  34. 444/tcp open snpp
  35. 3389/tcp open ms-term-serv
  36. 4125/tcp closed rww
  37. Device type: general purpose|phone|WAP|router
  38. Running (JUST GUESSING) : Linux 2.6.X (91%), ()
  39.  
  40. Not shown: 994 filtered ports
  41. PORT STATE SERVICE
  42. 22/tcp open ssh
  43. 25/tcp closed smtp
  44. 53/tcp closed domain
  45. 80/tcp open http
  46. 113/tcp closed auth
  47. 443/tcp closed https
  48. Device type: general purpose
  49. Running (JUST GUESSING) : Linux 2.6.X (90%)
  50. OS fingerprint not ideal because: Didn't receive UDP response. Please try again with -sSU
  51. Aggressive OS guesses: Linux 2.6.15 - 2.6.26 (90%), Linux 2.6.23 (89%), (…)
  52.  
  53. Not shown: 982 closed ports
  54. PORT STATE SERVICE
  55. 21/tcp open ftp
  56. 22/tcp open ssh
  57. 37/tcp open time
  58. 80/tcp open http
  59. 113/tcp open auth
  60. 135/tcp filtered msrpc
  61. 139/tcp filtered netbios-ssn
  62. 445/tcp filtered microsoft-ds
  63. 1025/tcp filtered NFS-or-IIS
  64. 1080/tcp filtered socks
  65. 1433/tcp filtered ms-sql-s
  66. 1434/tcp filtered ms-sql-m
  67. 4242/tcp filtered unknown
  68. 4444/tcp filtered krb524
  69. 6346/tcp filtered gnutella
  70. 6881/tcp filtered bittorrent-tracker
  71. 8888/tcp filtered sun-answerbook
  72. 45100/tcp filtered unknown
  73. Device type: general purpose|WAP|broadband router
  74. Running (JUST GUESSING) : Linux 2.6.X|2.4.X (95%), (…)
  75.  
  76. Not shown: 994 filtered ports
  77. PORT STATE SERVICE
  78. 22/tcp open ssh
  79. 25/tcp open smtp
  80. 53/tcp open domain
  81. 80/tcp open http
  82. 110/tcp open pop3
  83. 3389/tcp open ms-term-serv
  84. Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
  85. Device type: firewall|general purpose
  86. Running: Linux 2.6.X
  87. OS details: Smoothwall firewall (Linux 2.6.16.53), Linux 2.6.13 - 2.6.24, Linux 2.6.16

Of course, it strikes me that several among said machines seem to be Linuxes, but (appear to) run Microsoft services. Oh, and they also have P2P clients.

AttachmentSize
Results of parsing the logs21.58 KB
Relevant portion of the logs1.05 MB
( categories: )
imcsk8's picture

block them!!

Here's a iptables recipe to limit te incoming ssh connections

http://www.debian-administration.org/articles/187

vicm3's picture

Mhhh...

Mhh... I Fail2ban on physical machines (for some unknown reason the program seems to use to much CPU on a VM), and Denyhost for the VMs... yes it's heavier than a rule on iptables, but is neat to have it summarized via logwatch on both cases... also both programs are python and where written before iptables include that option (predates iptables as fail2ban where written originally for ipchains) ; )

gwolf's picture

Not blockable via that recipe…

Of course, I have the maximum rate limit set to 4 ssh connections per minute, as the article suggests... But here, by far most hosts have not even tried connecting four times in total. It's more a huge number of hosts than anything else :-/

kad's picture

There's a simple explanation

They could be appliances, i.e. routers, which wouldn't be surprisingly built on top of linux, although the traffic coming from inside their subnets could be very well be from windows machines.

bd_'s picture

Typically these are other

Typically these are other compromised servers acting in a botnet (and presumably trying to expand the botnet further). The reason you're seeing attacks from so many hosts is to avoid ip-based rate limiters like fail2ban - by the time you can identify one as being malicious, it's too late.

Anonymous's picture

Increase in ssh root access attempts

Anonymous's picture

I strongly suspect the

I strongly suspect the traffic you've observed represents the usual SSH worms: look for weak passwords, log into host as root, rootkit, recurse.

aurel32's picture

Same pattern

I have the same problem on a machine from a university network (134.214.0.0/16 subnet), and it has started at about the same time.

Moreover some of the IP adresses from your logs match the ones that I have in my logs. It looks like we are hit by the same botnet.

Michael Goetze's picture

Home Router Botnet?

Maybe this is some kind of home router botnet? I would expect more of those to be deployed in Europe than in Asia.

Helmut Grohne's picture

Observed before

I observed a similar scanning on my server a bit ago. It looks like a distributed brute force attack designed to work around solutions like denyhosts or fail2ban. Solutions that come to my mind are changing the ssh port and port knocking. Both are disruptive to users. Indeed I did switch to a port knocking approach from denyhosts. You can mail me if you want more details about my solution.

Good luck with this

Helmut

sourchier's picture

ssh problems

Unfortunatly it is a worldwide problem. It it so bad that even SANS has notticed a problem. Please see here:
http://isc.sans.org/diary.html?storyid=7213

I hope these attacks end soon.

Anonymous's picture

ISC report

I was going to refer to the ISC port report.

However at only another 500 sources, it is pretty small beer, so probably not fully automated, just another school kid trying his luck with SSH password guessing probably.

I don't think the port scan is terribly revealing. All have port 22 listening, so an SSH based attack could explain their compromise.

I use a non-standard port for SSH on one virtual server, bit of a pain to remember to type it, but I get almost no abusive attempts.

toxickore's picture

the theory from kad makes

the theory from kad makes sense. Could be machines inside a network running windows infected by worms or some malicious stuff.

Anonymous's picture

Has anyone tried denyhosts in

Has anyone tried denyhosts in its "synchronization mode"?

garaged's picture

SMB exploit

There is an unpatched exploit against windows's smb, I'm pretty sure its related to it.

It's surprising that the attack is so well coordinated, looks like someone is trying to accomplish something special

gwolf's picture

Don't think so...

Why would this be related to SMB when it tries to connect as root to the ssh port? Even more: Where does ssh run with PermitRootLogin? On systems running openssh (the vast majority of ssh-enabled servers AFAICT), I would suppose it is blocked by default everywhere, and only few administrators open it. Maybe it is an attack against some other, vulnerable ssh server? Or it assumes that any administrator stupid enough to open root ssh connections to a publically accessable machine is dumb enough to use 'root' as the password?

Kai's picture

Same here

Found the same in my auth-log. However, noticed that one Host was actually a hosted server (reliablehosting.com).

Stefano Rivera's picture

Yup

Seeing identical scanning on the machine running www.za.debian.org. Also on a university network. No other machines I administer are getting this.

vicm3's picture

Fail2ban & Denyhost

Well as I have the PermitRootLogin = no on my machines, fail2ban & denyhost automatically ban all root login attempts, in my fail2ban settings for 1 hour if memory serves right, on my denyhost one month.

Anyway, except for backup purposes I don't see why one must allow root login ssh to normal user then use su - to do admin work, or like has popularized ubuntu use sudo (when in ubuntu I still do sudo su - : )

For example from august to today.

# DenyHosts: Sat Aug 8 02:19:23 2009 | sshd: 80.191.63.110
sshd: 80.191.63.110
# DenyHosts: Sat Aug 8 03:28:27 2009 | sshd: 119.147.105.180
sshd: 119.147.105.180
# (…)
# DenyHosts: Fri Oct 2 19:59:56 2009 | sshd: 190.146.56.112
sshd: 190.146.56.112

Of course had a longer list, but these are the most recent, by the way I kept banned hosts for 8weeks... kind of ideas from me, as the VM's are far away and the physical machines are 4 floors downstairs...

gwolf's picture

fail2ban won't work here

(sorry, I'm trimming your rather long list of hosts here)

In my case, servers are being scanned few times (usually one only) by each of the IPs, so fail2ban would not gain me much. Of course, there is a real reason to use fail2ban, but in this scenario it would help very little.

What I ended up doing is just to push my server to a non-standard port after notifying my users. This particular attack seems to be brute-forcing across the internet, and does not worry me too much, but it makes my logs longer to parse. I do not need ssh to be on port 22, so I'll very happilly have it elsewhere.

daily log's picture

In my servers with ssh

In my servers with ssh service enabled. It's a common thing see this brute force. As long as you restrict the user who can connect through ssh. It's safe.

gwolf's picture

It is safe, but...

It clutters up the logs. So I'd rather not have them. Even if the only way out is to stop using port 22.

vicm3's picture

logwatch, logcheck, systraq, etc...

And also logrotate, I had huge log files, but purge it every four months, and you had bigger and probably faster hdd's ; ), I manage to take a look on my log with nested greps, also the mentioned script on the title helps with the logs.

Anonymous's picture

it like so much work its not even worth trying

granted im no code genius. in fact i purport it as magic the way a 1 and a 0 make color or movie. so when i paid way too much for a macbookpro with the caveat "no viruses" to later discover the root and have to invest in books because apples geniuses hush up if you mention root. at first i thought i pissed off some brilliant scientist like you mr wolf. nothing personal but it appears more focused than usual bot stuff. but maybe i retrofit that into my paranoia. certainly cutting edge engineers wouldnt have the time or care to read my email.

sorry for the off context entry and keep up the good work

jorge's picture

GeoIP

No he tocado mucho Perl pero está interesante el módulo de GeoIP. Actualmente obtengo el país con consultas whois y filtrando el país o dirección:
PAIS=$(whois "$IP" | grep 'country'| awk '{print $2}');
DIRECCION=$(whois "$IP" | grep 'address');

PD: Normalmente el puerto 22 lo permito a IPs de donde me conecto y la política por defecto de mi fw es denegar.