Strange scanning on my server?

Submitted by gwolf on Thu, 10/01/2009 - 18:04

Humm... Has anybody else seen a pattern like this?

I am getting a flurry of root login attempts at my main server at the University since yesterday 7:30AM (GMT-5). Now, from the machines I run in the 132.248.0.0/16 network (UNAM), only two listen to the world with ssh at port 22 — And yes, it is a very large network, but I am only getting this pattern on one of them (they are on different subnets, quite far apart). They are all attempting to log in as root, with a frequency that varies wildly, but is consistently over three times a minute right now. This is a sample of what I get in my logs:

[update] Logs omitted from blog post, as it is too wide and breaks displays for most users. You can download the log file instead.

Anyway… This comes from all over the world, and all the attempts are made as root (no attempts from unprivileged users). Of course, I have PermitRootLogin to no in /etc/ssh/sshd_config, but… I want to understand this as much as possible.

Initially it struck me that most of the attempts appeared to come from Europe (quite atypical for the usual botnet distribution), so I passed my logs through:

  1. #!/usr/bin/perl
  2. use Geo::IP;
  3. use IO::File;
  4. use strict;
  5. my ($geoip, $fh, %by_ip, %by_ctry);
  6.  
  7. $fh = IO::File->new('/tmp/sshd_log');
  8. $geoip=Geo::IP->new(GEOIP_STANDARD); 
  9. while (my $lin = <$fh>) { next unless $lin =~ /rhost=(\S+)/; $by_ip{$1}++}; 
  10.  
  11. print "        Incidence by IP:\n", "Num  Ctry   IP\n", ('='x60),"\n";
  12.  
  13. for my $ip ( sort {$by_ip{$a} <=> $by_ip{$b}} keys %by_ip) {
  14.     my $ctry = ($ip =~ /^[\d\.]+$/) ? 
  15. 	$geoip->country_code_by_addr($ip) : 
  16. 	$geoip->country_code_by_name($ip);
  17.  
  18.     $by_ctry{$ctry}++;
  19.     printf "%3d  %3s  %s\n", $by_ip{$ip}, $ctry, $ip;
  20. } 
  21.  
  22. print "        Incidence by country:\n", "Num  Country\n", "============\n";
  23. map {printf "%3d  %s\n", $by_ctry{$_}, $_}
  24.     sort {$by_ctry{$b} <=> $by_ctry{$a}} 
  25.     keys(%by_ctry);

The top countries (where the number of attempts ≥ 5) are:

  1. 104  CN
  2.  78  US
  3.  58  BR
  4.  49  DE
  5.  43  PL
  6.  20  ES
  7.  20  IN
  8.  19  RU
  9.  17  CO
  10.  17  UA
  11.  16  IT
  12.  13  AR
  13.  12  ZA
  14.  10  CA
  15.  10  CH
  16.   8  GB
  17.   8  AT
  18.   8  JP
  19.   8  FR
  20.   7  KR
  21.   7  HK
  22.   7  PE
  23.   7  ID
  24.   6  PT
  25.   5  CZ
  26.   5  AU
  27.   5  BE
  28.   5  SE
  29.   5  RO
  30.   5  MX

I am attaching to this post the relevant log (filtering out all the information I could regarding legitimate users) as well as the full output. In case somebody has seen this kind of wormish botnetish behaviour lately… please comment.

[Update] I have tried getting some data regarding the attacking machines, running a simple nmap -O -vv against a random sample (five machines, I hope I am not being too agressive in anybody's eyes). They all seem to be running some flavor of Linux (according to the OS fingerprinting), but the list of open ports varies wildly — I have seen the following:

  1. Not shown: 979 closed ports
  2. PORT      STATE    SERVICE
  3. 21/tcp    open     ftp
  4. 22/tcp    open     ssh
  5. 23/tcp    open     telnet
  6. 111/tcp   open     rpcbind
  7. 135/tcp   filtered msrpc
  8. 139/tcp   filtered netbios-ssn
  9. 445/tcp   filtered microsoft-ds
  10. 593/tcp   filtered http-rpc-epmap
  11. 992/tcp   open     telnets
  12. 1025/tcp  filtered NFS-or-IIS
  13. 1080/tcp  filtered socks
  14. 1433/tcp  filtered ms-sql-s
  15. 1434/tcp  filtered ms-sql-m
  16. 2049/tcp  open     nfs
  17. 4242/tcp  filtered unknown
  18. 4444/tcp  filtered krb524
  19. 6346/tcp  filtered gnutella
  20. 6881/tcp  filtered bittorrent-tracker
  21. 8888/tcp  filtered sun-answerbook
  22. 10000/tcp open     snet-sensor-mgmt
  23. 45100/tcp filtered unknown
  24. Device type: general purpose|WAP|PBX
  25. Running (JUST GUESSING) : Linux 2.6.X|2.4.X (96%), (…)
  26.  
  27.  
  28. Not shown: 993 filtered ports
  29. PORT     STATE  SERVICE
  30. 22/tcp   open   ssh
  31. 25/tcp   open   smtp
  32. 80/tcp   open   http
  33. 443/tcp  open   https
  34. 444/tcp  open   snpp
  35. 3389/tcp open   ms-term-serv
  36. 4125/tcp closed rww
  37. Device type: general purpose|phone|WAP|router
  38. Running (JUST GUESSING) : Linux 2.6.X (91%), (…)
  39.  
  40. Not shown: 994 filtered ports
  41. PORT    STATE  SERVICE
  42. 22/tcp  open   ssh
  43. 25/tcp  closed smtp
  44. 53/tcp  closed domain
  45. 80/tcp  open   http
  46. 113/tcp closed auth
  47. 443/tcp closed https
  48. Device type: general purpose
  49. Running (JUST GUESSING) : Linux 2.6.X (90%)
  50. OS fingerprint not ideal because: Didn't receive UDP response. Please try again with -sSU
  51. Aggressive OS guesses: Linux 2.6.15 - 2.6.26 (90%), Linux 2.6.23 (89%), (…)
  52.  
  53. Not shown: 982 closed ports
  54. PORT      STATE    SERVICE
  55. 21/tcp    open     ftp
  56. 22/tcp    open     ssh
  57. 37/tcp    open     time
  58. 80/tcp    open     http
  59. 113/tcp   open     auth
  60. 135/tcp   filtered msrpc
  61. 139/tcp   filtered netbios-ssn
  62. 445/tcp   filtered microsoft-ds
  63. 1025/tcp  filtered NFS-or-IIS
  64. 1080/tcp  filtered socks
  65. 1433/tcp  filtered ms-sql-s
  66. 1434/tcp  filtered ms-sql-m
  67. 4242/tcp  filtered unknown
  68. 4444/tcp  filtered krb524
  69. 6346/tcp  filtered gnutella
  70. 6881/tcp  filtered bittorrent-tracker
  71. 8888/tcp  filtered sun-answerbook
  72. 45100/tcp filtered unknown
  73. Device type: general purpose|WAP|broadband router
  74. Running (JUST GUESSING) : Linux 2.6.X|2.4.X (95%), (…)
  75.  
  76. Not shown: 994 filtered ports
  77. PORT     STATE SERVICE
  78. 22/tcp   open  ssh
  79. 25/tcp   open  smtp
  80. 53/tcp   open  domain
  81. 80/tcp   open  http
  82. 110/tcp  open  pop3
  83. 3389/tcp open  ms-term-serv
  84. Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
  85. Device type: firewall|general purpose
  86. Running: Linux 2.6.X
  87. OS details: Smoothwall firewall (Linux 2.6.16.53), Linux 2.6.13 - 2.6.24, Linux 2.6.16

Of course, it strikes me that several among said machines seem to be Linuxes, but (appear to) run Microsoft services. Oh, and they also have P2P clients.

AttachmentSize
Results of parsing the logs21.58 KB
Relevant portion of the logs1.05 MB
( categories: )
imcsk8's picture

block them!!

Submitted by imcsk8 (not verified) on Thu, 10/01/2009 - 18:19.

Here's a iptables recipe to limit te incoming ssh connections

http://www.debian-administration.org/articles/187

vicm3's picture

Mhhh...

Submitted by vicm3 (not verified) on Thu, 10/01/2009 - 21:24.

Mhh... I Fail2ban on physical machines (for some unknown reason the program seems to use to much CPU on a VM), and Denyhost for the VMs... yes it's heavier than a rule on iptables, but is neat to have it summarized via logwatch on both cases... also both programs are python and where written before iptables include that option (predates iptables as fail2ban where written originally for ipchains) ; )

gwolf's picture

Not blockable via that recipe…

Submitted by gwolf on Thu, 10/01/2009 - 22:37.

Of course, I have the maximum rate limit set to 4 ssh connections per minute, as the article suggests... But here, by far most hosts have not even tried connecting four times in total. It's more a huge number of hosts than anything else :-/

kad's picture

There's a simple explanation

Submitted by kad (not verified) on Thu, 10/01/2009 - 19:41.

They could be appliances, i.e. routers, which wouldn't be surprisingly built on top of linux, although the traffic coming from inside their subnets could be very well be from windows machines.

bd_'s picture

Typically these are other

Submitted by bd_ (not verified) on Thu, 10/01/2009 - 23:36.

Typically these are other compromised servers acting in a botnet (and presumably trying to expand the botnet further). The reason you're seeing attacks from so many hosts is to avoid ip-based rate limiters like fail2ban - by the time you can identify one as being malicious, it's too late.

Anonymous's picture

Increase in ssh root access attempts

Submitted by Anonymous (not verified) on Fri, 10/02/2009 - 00:37.

http://isc.sans.org/diary.html?storyid=7213

Anonymous's picture

I strongly suspect the

Submitted by Anonymous (not verified) on Fri, 10/02/2009 - 01:29.

I strongly suspect the traffic you've observed represents the usual SSH worms: look for weak passwords, log into host as root, rootkit, recurse.

aurel32's picture

Same pattern

Submitted by aurel32 (not verified) on Fri, 10/02/2009 - 02:27.

I have the same problem on a machine from a university network (134.214.0.0/16 subnet), and it has started at about the same time.

Moreover some of the IP adresses from your logs match the ones that I have in my logs. It looks like we are hit by the same botnet.

Michael Goetze's picture

Home Router Botnet?

Submitted by Michael Goetze (not verified) on Fri, 10/02/2009 - 02:42.

Maybe this is some kind of home router botnet? I would expect more of those to be deployed in Europe than in Asia.

Helmut Grohne's picture

Observed before

Submitted by Helmut Grohne (not verified) on Fri, 10/02/2009 - 04:24.

I observed a similar scanning on my server a bit ago. It looks like a distributed brute force attack designed to work around solutions like denyhosts or fail2ban. Solutions that come to my mind are changing the ssh port and port knocking. Both are disruptive to users. Indeed I did switch to a port knocking approach from denyhosts. You can mail me if you want more details about my solution.

Good luck with this

Helmut

sourchier's picture

ssh problems

Submitted by sourchier (not verified) on Fri, 10/02/2009 - 05:44.

Unfortunatly it is a worldwide problem. It it so bad that even SANS has notticed a problem. Please see here:
http://isc.sans.org/diary.html?storyid=7213

I hope these attacks end soon.

Anonymous's picture

ISC report

Submitted by Anonymous (not verified) on Sat, 10/03/2009 - 04:32.

I was going to refer to the ISC port report.

However at only another 500 sources, it is pretty small beer, so probably not fully automated, just another school kid trying his luck with SSH password guessing probably.

I don't think the port scan is terribly revealing. All have port 22 listening, so an SSH based attack could explain their compromise.

I use a non-standard port for SSH on one virtual server, bit of a pain to remember to type it, but I get almost no abusive attempts.

toxickore's picture

the theory from kad makes

Submitted by toxickore (not verified) on Fri, 10/02/2009 - 08:10.

the theory from kad makes sense. Could be machines inside a network running windows infected by worms or some malicious stuff.

Anonymous's picture

Has anyone tried denyhosts in

Submitted by Anonymous (not verified) on Fri, 10/02/2009 - 08:19.

Has anyone tried denyhosts in its "synchronization mode"?

garaged's picture

SMB exploit

Submitted by garaged (not verified) on Fri, 10/02/2009 - 10:40.

There is an unpatched exploit against windows's smb, I'm pretty sure its related to it.

It's surprising that the attack is so well coordinated, looks like someone is trying to accomplish something special

gwolf's picture

Don't think so...

Submitted by gwolf on Mon, 10/12/2009 - 10:54.

Why would this be related to SMB when it tries to connect as root to the ssh port? Even more: Where does ssh run with PermitRootLogin? On systems running openssh (the vast majority of ssh-enabled servers AFAICT), I would suppose it is blocked by default everywhere, and only few administrators open it. Maybe it is an attack against some other, vulnerable ssh server? Or it assumes that any administrator stupid enough to open root ssh connections to a publically accessable machine is dumb enough to use 'root' as the password?

Kai's picture

Same here

Submitted by Kai (not verified) on Fri, 10/02/2009 - 13:49.

Found the same in my auth-log. However, noticed that one Host was actually a hosted server (reliablehosting.com).

Stefano Rivera's picture

Yup

Submitted by Stefano Rivera (not verified) on Fri, 10/02/2009 - 17:52.

Seeing identical scanning on the machine running www.za.debian.org. Also on a university network. No other machines I administer are getting this.

vicm3's picture

Fail2ban & Denyhost

Submitted by vicm3 (not verified) on Fri, 10/02/2009 - 23:59.

Well as I have the PermitRootLogin = no on my machines, fail2ban & denyhost automatically ban all root login attempts, in my fail2ban settings for 1 hour if memory serves right, on my denyhost one month.

Anyway, except for backup purposes I don't see why one must allow root login ssh to normal user then use su - to do admin work, or like has popularized ubuntu use sudo (when in ubuntu I still do sudo su - : )

For example from august to today.

# DenyHosts: Sat Aug 8 02:19:23 2009 | sshd: 80.191.63.110
sshd: 80.191.63.110
# DenyHosts: Sat Aug 8 03:28:27 2009 | sshd: 119.147.105.180
sshd: 119.147.105.180
# (…)
# DenyHosts: Fri Oct 2 19:59:56 2009 | sshd: 190.146.56.112
sshd: 190.146.56.112

Of course had a longer list, but these are the most recent, by the way I kept banned hosts for 8weeks... kind of ideas from me, as the VM's are far away and the physical machines are 4 floors downstairs...

gwolf's picture

fail2ban won't work here

Submitted by gwolf on Mon, 10/12/2009 - 10:59.

(sorry, I'm trimming your rather long list of hosts here)

In my case, servers are being scanned few times (usually one only) by each of the IPs, so fail2ban would not gain me much. Of course, there is a real reason to use fail2ban, but in this scenario it would help very little.

What I ended up doing is just to push my server to a non-standard port after notifying my users. This particular attack seems to be brute-forcing across the internet, and does not worry me too much, but it makes my logs longer to parse. I do not need ssh to be on port 22, so I'll very happilly have it elsewhere.

daily log's picture

In my servers with ssh

Submitted by daily log (not verified) on Tue, 10/06/2009 - 10:18.

In my servers with ssh service enabled. It's a common thing see this brute force. As long as you restrict the user who can connect through ssh. It's safe.

gwolf's picture

It is safe, but...

Submitted by gwolf on Mon, 10/12/2009 - 10:59.

It clutters up the logs. So I'd rather not have them. Even if the only way out is to stop using port 22.

vicm3's picture

logwatch, logcheck, systraq, etc...

Submitted by vicm3 (not verified) on Mon, 10/12/2009 - 20:43.

And also logrotate, I had huge log files, but purge it every four months, and you had bigger and probably faster hdd's ; ), I manage to take a look on my log with nested greps, also the mentioned script on the title helps with the logs.

Anonymous's picture

it like so much work its not even worth trying

Submitted by Anonymous (not verified) on Sun, 03/28/2010 - 12:11.

granted im no code genius. in fact i purport it as magic the way a 1 and a 0 make color or movie. so when i paid way too much for a macbookpro with the caveat "no viruses" to later discover the root and have to invest in books because apples geniuses hush up if you mention root. at first i thought i pissed off some brilliant scientist like you mr wolf. nothing personal but it appears more focused than usual bot stuff. but maybe i retrofit that into my paranoia. certainly cutting edge engineers wouldnt have the time or care to read my email.

sorry for the off context entry and keep up the good work

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account associated with the e-mail address you provide, it will be used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <br> <b> <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote> <img> <h1> <h2> <h3> <tt> <pre> <strike> <table> <tr> <th> <td>
  • Lines and paragraphs break automatically.
  • Use <bib>citekey</bib> or [bib]citekey[/bib] to insert automatically numbered references.
  • Use [fn]...[/fn] (or <fn>...</fn>) to insert automatically numbered footnotes.
  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>. The supported tag styles are: <foo>, [foo].

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Keep in mind that all comments will also have to be administrator-moderated. Don't waste your time writing a spam that no one will read.
Image CAPTCHA
Enter the characters shown in the image.