Threat Level: green Handler on Duty: Tom Webb

SANS ISC: Wipe, rinse and repeat - Internet Security | DShield SANS ISC InfoSec Forums


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
Wipe, rinse and repeat

Most of us have faced a time when a machine gets compromised with malware. In some cases it gets to the point where cleaning the infected computer is too time consuming or too difficult to clean, so the easy option is to wipe the machine and rebuild it.

Just before the forensic community (or some of my fellow handlers) lynch me for making this over generalised, evidence eliminating statement, allow me to elaborate.

“Nuke it from orbit”*

The format and rebuild statement normally comes from the following groups:

  • Management
  • Over worked IT staff
  • The owner who’s just spent the last hour on search engines on how to fix their “slow” (utterly infected) PC
  • The security team

The first three can be grouped as those that are not interested in analysing, understanding or knowing what happened on the particular machine. They just want their machine(s) back to normal ASAP as they can go about their business.

The security team, in contrast, have made this call as part of a calculated decision, after collecting the evident they need to get the business running safely again.

The decision to rebuild is considerably easier for those with a standard operating environment (SOE) or managed operating environment (MOE). This allows for a rapid deployment of a fully functional operation system with all the previous applications. This is a thing of beauty, bringing tears of joy to the most harden PC tech, as it’s a fast, reliable and easy completely re-deployment with a simple press of a few buttons. The assumption is - and I want to be very clear on this - that any user data is safely saved elsewhere, not on the PC about to be formatted and rebuilt.

The problem child

So what happens when you are confronted with a machine that needs to be wiped and re-built but no-one has a clue what’s on it and if it’s ever been backed up?

I like to call this the friend/family pc scenario or the forgotten machine, out back, that runs the company disaster-in-waiting issue.

Before even thinking about nuking this type of PC, there are normally two distinct areas to be worried about on these systems: data and applications

For the very wise or very paranoid amongst us, a full image of the troublesome system is the way to go. This provides a working image of the machine to refer back to quickly and avoids a great deal of painful conversations along the lines of “but you never mention that”.  Tools such as Sysinternals' Disk2vhd [1] makes a complete on line virtual image of the problem system. For those that run other virtualisation software it’s pretty easy to convert the Disk2vhd's .vhd file to other formats using your favourite virtualisation technology.

Close encounters

You have a backup, whether it is a virtual image, a standard backup or a copy of the PC's entire contents on an external drive; the next step is to know what you’re getting into.

An audit of all the known software on the machine, with first a verbal interrogation of the owner followed by a physical examination of the machine, provides a solid picture what needs to be on the clean system.  This is where recording your findings, conversation with the owner and processes to rebuild the machine can help in the future, should this happen again.

Dude, where’s my data?

Losing data doesn’t sound too bad until that data is someone’s child first steps or the company payroll. As a suggested list of files and folders to be sure you have:

  • Browser favourites and configuration files
  • Microsoft Office configuration
  • Email folders (.pst files and the like)
  • The entire My Documents folders
  • Game files
  • User profiles
  • File and folders saved in weird location only know to the owner or application

To alleviate some of the pain of manually hunting for these files, Microsoft offers a number of tools to export data off and these are well worth reviewing:

  • Office Save My Settings Wizard [2]
  • File and Settings Transfer Wizard [3]
  • User State Migration Tool [4]
  • Windows Easy Transfer [5]

Game over man, game over


Applications are just as important for any system, so ensuring you can get copies of the installation media the license keys for software, including the original operating system is a must.

For lost license keys, software such as The Magical Jelly Bean Keyfinder [6] can get back most standard products keys.

For those applications which the original installation media no longer exists and the vendor can’t supply a replacement copy, this may be an opportunity to upgrade or migrate to a new application.

As a final note, be aware that there may be Wacky hardware installed and the drivers for ancient ISDN/video/sound/modem/and so on cards were last seen back in the 90’s. The very of best luck with that.


As always, if you have any better suggestions, insights or tips please feel free to comment.


[1] http://technet.microsoft.com/en-us/sysinternals/ee656415.aspx
[2] http://support.microsoft.com/kb/312978
[3] http://support.microsoft.com/kb/293118
[4] http://technet.microsoft.com/en-us/library/dd560801(WS.10).aspx
[5] http://windows.microsoft.com/en-US/windows7/products/features/windows-easy-transfer
[6] http://www.magicaljellybean.com/keyfinder

*This frequently used phrase is taken from the movie Aliens and the actual quote from the character Ripley is: "I say we take off and nuke the entire site from orbit. It’s the only way to be sure."

Who knew James Cameron was really making a movie about the folly of poor incident response? Ripley is the lead incident handler dealing with this infection outbreak and she’s decided that Step 4 of the incident handling process [7], eradication, is the only real way forward. The business owner, Burke, disagrees; he later discovers he should have really taken Ripley expert advice to save him from, what is certainly, a very painful way to go.

[7] http://www.giac.org/resources/whitepaper/network/17.php

 

 

Chris Mohan --- Internet Storm Center Handler on Duty

Chris

105 Posts
ISC Handler
I agree 100% with list under "Nuke it from Orbit." Most shops have a get-r-done approach. When you have a pit boss on the tradefloor screaming that they are losing money without a PC, there is not much room for discussion. However, I can't tell you how many times I have saved the drive in the lockbox to find out later, that the employee ran Quickbooks or some financial software and stored the data on the loca drive. Over the years I have seen a proliferation of shoot first and ask questions later. "I don't know who is worse, I don't see them !@#$% each other over for a percentage." - Ripley
Anonymous
Why do I suddenly feel like watching Aliens?
Chavez243

15 Posts
... and also worth mentioning:

1) Install all security patches after the rebuild, or you're worse off than before.

2) Consider whether passwords (and credit cards) have been compromised and need to be changed. It helps if you happen to know the virus name and what it does, if it captures passwords or allows remote access, etc.

3) Enterprises should consider how the infection got in. Every infection and compromise indicates a potential gap in the current security posture, policy or procedures, and that often isn't fixed by a reinstall.

It seems to me that wiping the computer is recommended too often and without caveats. As you seem to suggest, not all malware deserves all that effort of rebuilding. Rebuilding a home system from an install CD takes hours or days and is never fun.

Unfortunately, antivirus today doesn't really do a good job of recommending remediation steps (e.g., "this virus captures passwords"), and usually doesn't distinguish between malware blocked before infection vs. malware removed after infection, when further remediation is needed.
Chavez243
8 Posts
J points out that anti-virus tells us nothing about the risks, and if the malware/virus executed and/or installed/embedded itself.

I have found using the new Symantec Endpoint Protection in a corporate environment that the latter is true. The critters always run before being "caught", and then a resident program many times will continue to run until t a reboot. Trace files left on the disk many times are not caught until the next full scan as well.

The older Symantec products nailed the virus on disk write. This no longer seems to be the case.

Big Anti-virus is basically useless by itself. Now a combination of IDS, patterning and smart techs in the data center is required. Sadly, this doesn't happen to work out money-wise. IT is being cut every day by more. ID-10-T comes to mind :-)

-Al
Al of Your Data Center

80 Posts
Personally... I like to pop the drive out of the infected windows box, attach it to a non-windows machine via usb and scan it a few times before moving files manually. I've used the time saving tools that you have mentioned however I have seen them transfer infected files from old images to new.

"not all malware deserves the effort of rebuilding"

Eh? If the machine has any malware, it likely has more. Putting your faith in that machine any longer is dangerous at best.
James P. Carter

1 Posts
I mostly agree with your groupings Chris, except that I am not overworked now that we outsource our IT Support to another company.
I am interested in knowing about Trojan/virus attack vectors, but I am not interested in trying to find out myself, I leave that to the companies who specialise in this area and I read their analysis. I would rather spend my time making music and animations.

I handle Home machines differently to Business machines.
For Business:-
I still go with the “nuke the machine”, going back 4-5 years or so I would have investigated since most trojans and virus were very simple to remove, but now days, they are very tricky ##### to get rid of.
Our Business has fully automated builds; Users do not have write permission to save data to their local hard disk and Users profiles are redirected to the network, so it is easy for us to wipe machines or even swap out for hot swappable machines.
For Home:-
I only now fixed my close families machines, I tend to have a few spare machines that I can build for them in a few hours with all their software they had before since I built their machines in the first place. This gets them back up and running quicker, it takes a little longer to get the data back.

Like Merge does, popping the infected drive into a non-windows box is a good way of recovering the data back.

I use to do regular Ghost Images of my family members machines to get them back to a know state, but use this less option less now days as I find rebuilding just as quick having all the software on my home NAS.
James P. Carter
4 Posts
I think (this class of) malware authors are wising up to the fact that their creation(s) will be discovered and removed in a shortening time-frame as anti-malware technologies improve.

Thus (this class of) malware tries to steal data as early in the process as possible, usually upon the point of infection.

Granted there are sometimes valid reasons to just reformat/reinstall, but most of the time people think "i dont have anything important on that machine." But consider a seemingly insignificant piece of data, eg a Cookie, could be used to pilfer money from bank-accounts, or even go on an eBay shopping spree.

Keys/passwords could be retrieved from the registry to allow access to things like VNC, RSA/DSA keys, access to WiFi networks, the list goes on..

I think people need to more seriously consider data-loss when dealing with malware. All an attacker needs is one critical piece of information the admin/user forgot to change post-infection, and it's game-over!

The odds are stacked in the attackers favour (not that that's anything new...) :)
Anonymous
It seems to me that, given the low cost of hard disks these days, the simple solution is to drop a new drive in the machine and then re-image it. Then, if it turns out that there's important data, all that is required is to mount it on a different OS, check the data for malware, and copy only the clean data to the new system. But maybe I'm missing something here.

-Don
Anonymous
Don, you are right, drop a new hard disk in or swapping the machine out is the quickies way to get the Users back up and running again, they are not interested in the technical details of the infection, mounting the original disk on another O/S to get the data back is a good approach.

Have a look at how "evercookie" works, its interest how they rebuild themselves as Users try and delete them. see "Bruce Schneier article on them" or listen to SecurityNow podcast.

VNC is not secure as it saves it password in clear text in the registry.

Another good read is how Stuxnet works (again SecurityNow Podcast), when the general trojan/virus writers start using these routine, it makes sense to wipe the machine as writers are specialist (or trying to be) in this areas and we are not.

And to keep with the theme, "Game Over, Man, Game Over" :-D

-Steve G
Anonymous
My experience is that you can't trust your AV. Neither do you know if all malware components were detected at all or not nor do you know if your AV named the treat correctly. Additionally some AV vendors (I'm looking at you, Symantec!) are too lazy to name the threats and instead prefer to return completely useless threat categorizations like 'W32.Backdoor' or 'W32.Trojan'. Other vendors might return a detailed naming information of the threat (like ‘Trojan.JS.Agent.bpu’) but will not provide technical details to that specific version of malware but only to the general class of threats or its first occurrence (e.g. technical information might only apply to Trojan.JS.Agent instead of Trojan.JS.Agent.bpu).

Either way, I am going to build myself an automated malware analysis station (Minibis) for exactly that reason. I try to keep spare HDDs around so that my colleagues can do the nuking part on a different hardware without having to wait for my evidence collecting to complete.

Regarding the AVs can’t clean infections anymore story: If a threat is already running with admin privileges it is truly “Game Over, Man, Game Over” for the Antivirus. It might be able to remove the threat but it resembles a race condition: whoever removes the other first wins, threat or AV. Thus you have to catch the threat either before it is run (on the HDD, on file shares, USB Sticks and other media etc.) or when it tries to enter your RAM (Host IPS functionality should aid here in theory)
Anonymous
If the computer has been compromised at the system/admin/root level we require a rebuilt to regain system integrity. You can't trust the system until it is rebuilt. Rebuilding protects the user as well as the company from potential damage.
@Miss_Sudo

12 Posts
I still take some odd pleasure in rooting out all the little components of malware and removing them from a system, as long as it's not so infected as to be nearly unusable. Most malware authors seem to try to keep machines functioning these days in order to keep them as good zombies. A broken PC won't display scareware alerts.
In a home environment I usually start by looking at what's running on startup, identifying the suspect executable then pointing procmon at it and looking at what it calls. In a corporate, managed environment I'll identify the malware and capture whatever I need for analysis of infection vector etc. then, well... nuke it.
@Miss_Sudo
12 Posts

Sign Up for Free or Log In to start participating in the conversation!