You may have seen the reports that the New Zealand Ambulance service had to revert to manual processing of calls after a worm affected a number of their systems (http://computerworld.co.nz/news.nsf/news/mystery-virus-disrupts-st-johns-ambulance-service). This got me thinking about what needs to happen in order to deal with this kind of situation, but first lets set the scene. 

Most organisations will have the basic security controls in place.  They will have policies, firewalls, Antivirus on the desktop and maybe on the servers. Scanning software on email and web traffic, possibly even USB control. So how did the worm get in in the first place?  Now this is purely speculation, based on past experiences and in no way relates at all to the NZ ambulance case. We are talking hypothetically here. So What could possibly have happened?

There are a few attack vectors I can think of and no doubt you can add to this.

• Option 1: A laptop has been off the corporate network for a while, may not have been patched or kept up to date with patches and AV.  It is infected when connected to the internet at an insecure location. When brought back into the corporate environment (e.g. plugged into the network or connected via VPN) the malware did a little jump for joy and started spreading.
• Option 2: User browses a web page and is the victim of a drive by.  The malware is downloaded and starts spreading. 
• Option 3: An email is opened and malware is downloaded and executed.

Any of the three above options are possible in most environments.  AV products whilst good, are far from infallible and it is easy enough to create malicious payloads that sail past most antivirus products.  Once the malware is in, it can do its thing and start attacking the rest of the infrastructure.

So if prevention is difficult, you may have to face the reality that what happened to NZ Ambulance can happen to you.  If you can't prevent you must detect. How can you identify the fact that you have an issue?  Worst case scenario, a third party tells you.  At the Storm Centre we often contact ISPs, Corporations and yes sometimes Government agencies to give them some bad news, usually they are a tad surprised. It is much better to find these things your self. It makes explanations to CEOs that much more comfortable.

What should you be looking for?  You may look at firewall logs to see what traffic from inside the network is bouncing off the firewall. Examine proxy logs to look for connections to interesting locations (insert your favourite countries here). Look for multiple connections from multiple devices in your network to a few target locations.  Examine server and AD logs to find log in attempts. You may receive complaints that things are slow, so monitor help desk calls. Systems that stop working may be a clue as well. If you can spend an hour, 30 minutes, even less to look at your logs on a daily basis, then you will be in a better position to identify weirdness.  Once detected you can react. 

You've found the worm, now what? turning the device off will contain it, but it is unlikely to make management happy, especially if you start switching off critical servers.  So you may need to do something else.  Workstations may be a bit easier to contain.  You could move them to a sandbox or walled garden environment.  Place them on this contained vlan and they can do less damage to the rest of the organisation. Ideally this is an automated process, but someone with quick fingers could in a pinch achieve this as well. If you find it is leaving your environment, you might need to change firewall rules or IDS/IPS rules.

For eradication, realistically the only safe option is to rebuild.  Re-image, redeploy the system from known good media. You could attempt a removal process documented by an AV vendor or other organisation, just remember it wasn't picked up in the first place. Since the state of the machine is unknown you are really better off to rebuild, sorry.


Putting all the above in the context of the incident handling process

Preparation
In addition to the policies and base security controls mentioned above, you may want to consider the following:

• No local admin privileges
• Segmentation in the network
• IDS/IPS
• Log Monitoring and analysis (ACLs, or internal firewalls)
• Private VLANS
• Darknet

Identification

• Look for unusual network activity
• Examine log files
• Become familiar with your environment

Containment

• Move the device to a sandbox VLAN
• Switch it off
• Implement firewall rules, ACLs other configuration changes to reduce the ability to do damage.

Eradication

• Unfortunately rebuild is the safest option.
• Some vendors may have a removal process
• Identify how they got in and develop strategies to plug the hole

Recovery

• Put systems back in a controlled fashion.
• Monitor activities, watch for their return

Lessons Learned

• Learn the lessons :-)
• Fix the issues identified
• Implement the controls that allow you to ideally prevent, but at least detect it next time.

the above is by no means complete so if you have anything to add, feel free to add a comment or let us know via the contact form.

Mark H - Shearwater