Mátalo, luego virigüas (roughly: Kill him, ask questions later)
The phrase on this title is often attributed to Pancho Villa (1878-1923), Mexican Revolution leader. He had a fame of cruelty, killing suspects before even questioning them.
Today, it started as a very nice day. I had even time in the morning to find, fix, upload and send upstream a trivial bug in libgruff-ruby… At 11:00, I left the Institute as my father came to the city to do some paperwork… We sat having a cup of coffee in a restaurant near the office we had went to at around 12:00, and my phone rang.
And it was from work. That’s never a good sign. My boss told me he was facing a massive virus infection, and decided to disconnect the firewall. I corrected him - that will do no good once the virus is in our system, if you want to disconnect anything, disconnect all of our switches.
Came back, and found him and my coworker stunned and not knowing what to do. He says, the antivirus alarm went off almost simultaneously on the two computers he had on his desk, and in few minutes over 15 computers all over the Institute were ill. The symptoms? Programs not showing up in the taskbar, copy/paste functionality b0rken, many programs misbehaved or just didn’t open… They were grimly facing a complete recovery operation they have grown used to: The whole OS has become corrupted or destroyed, we will have to open the computer, extract the HD, install it elsewhere, back it up, reinstall OS and applications, restore the backup. Yes, I know too many extra steps are included here, but I have come to accept their ways of dealing with Windows. Nobody says dealing with Windows is fun. I like my work to be fun, so I stay clear of theirs.
I insisted on turning back one one of the switches, the one for the servers and my machine (and some more in the same physical area). OK’d. But they didn’t want to switch on any other switch, so a traffic capture (tcpdump / wireshark) led nowhere - but at least it gave my my Google back.
They have configured the antivirus software we deploy to all of the Windows machines in such a way that it deletes upon sight any malware - And when they manually scan, they blindly hit Delete whenever anything is found as well. Of course, no infected binary was left alive for me to inspect, and the machines were dead. But I was able to glimpse at the name of the deleted file: rpcss.dll.
After googling a bit - Bliss! Joy! I found the answer. So here is the set of interactions, and how they led to this killing spree. Please remember I am a Windows newbie and speak just out of guesswork.
- This is a fast-spreading virus. My friend Rubén at DGSCA suggest it might be related to this report submitted today; at Barrapunto there is a thread about another virus that appeared four days ago, infected 1.1 million Windows machines on its first day, and so far is around the ninth million. Update: Equivalent thread at Slashdot, for the Spanish-impaired.
- The virus infects at least two copies of a system binary:
%system32%\rpcss.dlland\Windows\ServicePackFiles\i386\rpcss.dll. Windows uses the second one to restore the first one in case it is damaged, if I understood correctly. - The antivirus does not detect the infection when the library files are written, but when they are linked, so it only spots it the next time
%system32%\rpcss.dllis brought into memory. - This is a very common library - It takes care of, well, RPC. So, quite probably, this file will be linked again on the next program launch - or accessed when a running program requires anything not currently in RAM? Dunno. The thing is, the library gets linked.
- The antivirus will happily tell you it has killed a threat! Your nice RPC library is now defunct. ¡Mátalo, luego virigüas!
- So, of course, notifying the taskbar of a new window appearing, or clipboard actions, or whatnot will refuse to work.
- Machine restart, full system scan requested. The antivirus finds de second copy of this library in the master directory (
\Windows\ServicePackFiles\i386). The virus used this location so that Windows won't restore a clean version over it. But yes, it will fall again under the claws of the antivirus... I guess. Anyway, the antivirus offers to delete this file as well, and does so. - User is desperate. My coworkers are desperate. I am... mildly annoyed?
Once I found this line of thought… I went to a working machine, inserted my flash memory, and copied %system32%\rpcss.dll to it. Went back to a sick machine, and ran cmd. Then, it was just matter of copy f:\rpcss.dll c:\windows\system32, a simple reboot (it never hurts to reboot in Windows!), and problem solved!
Oh, as a side rant: I find it extremely annoying and sad that many people I know, sometimes with more experience as a computer operator/supporter than what I have of experience as a living human being, are so scared of using a command-line interface. They were dismayed at seeing no drag-and-drop and no copy/paste functionality were available! copy is not an option.
Anyway… Today was an experience on how a simple, mostly-harmless and quite-fertile virus is able to be terribly magnified by the presence of a trigger-happy antivirus.
Why won’t they give themselves a chance to try something else? Say, GNU/Linux? :-/
Comments
David Moreno 2009-01-17 09:47:17
On the post title,
On the post title, “viriguas” doesn’t require umlaut on the “u”.
Envite 2009-01-17 02:19:24
Qué duro
Te leo con asiduidad a través de Planet Debian.
Qué duro lo que cuentas.
¿Aprenderán alguna vez?
gwolf 2009-01-19 05:36:28
Of course it doesn’t
Still, it is there, and not by mistake. Any questions?
Roadmaster 2009-01-21 07:50:17
Ah the joys
May you always, for your own sanity, remain a Windows newbie.
Last time we had a massive virus infection, DGSCA unplugged us from the net and the tech support guys had to disinfect computer by computer (over 500).
When we reported the outbreak had been controlled, they restored network access, BUT THEY DID IT WRONG, and so the network kept crashing after a few seconds.
They blamed it on the Linux servers and were about to send a full-fledged security audit team (yeah as if they had anyone who knew what they were doing), when they realized they had done something wrong, fixed it, and everything was back to normal.
The fun part is, they just called off the security audit but didn’t even tell us and didn’t apologize for blaming us for their faults.
Morale of the story: if Windows hadn’t gotten infected none of this would have happened :)
vicm3 2009-01-16 19:10:28
Good windows support…
Well today I not at office at all, maybe monday had to see something like that, or worse, I wish support (on my work) take a course of good practices on maintaining windows ‘cause at the end of semester some of my students are more skilled than them, which is sad… Normally all antivirus on auto mode will be configured to repair/quarantine/delete on that order… :D, anyway is not supposed to be my work, but told a lot of tips on how use their new antivirus… I hope that today don’t have become a format/reinstall day just after had my students yesterday installing freemind to do mind mapping for their wednesday exam… this make me wonder why my suggestion of setting a clonezilla boot server to clean install as needed as not been implemented. Yea I know I’m professor and no sysadmin there ;D.
A tip get powertoys for xp and disable auto play for all units most of current malware is abusing autoexec.bat it’s really easy and keeps from modifying the windows register…