You may have seen the reports that the New Zealand Ambulance service had to revert to manual processing of calls after a worm affected a number of their systems (http://computerworld.co.nz/news.nsf/news/mystery-virus-disrupts-st-johns-ambulance-service). This got me thinking about what needs to happen in order to deal with this kind of situation, but first lets set the scene.
Most organisations will have the basic security controls in place. They will have policies, firewalls, Antivirus on the desktop and maybe on the servers. Scanning software on email and web traffic, possibly even USB control. So how did the worm get in in the first place? Now this is purely speculation, based on past experiences and in no way relates at all to the NZ ambulance case. We are talking hypothetically here. So What could possibly have happened?
There are a few attack vectors I can think of and no doubt you can add to this.
So if prevention is difficult, you may have to face the reality that what happened to NZ Ambulance can happen to you. If you can't prevent you must detect. How can you identify the fact that you have an issue? Worst case scenario, a third party tells you. At the Storm Centre we often contact ISPs, Corporations and yes sometimes Government agencies to give them some bad news, usually they are a tad surprised. It is much better to find these things your self. It makes explanations to CEOs that much more comfortable.
What should you be looking for? You may look at firewall logs to see what traffic from inside the network is bouncing off the firewall. Examine proxy logs to look for connections to interesting locations (insert your favourite countries here). Look for multiple connections from multiple devices in your network to a few target locations. Examine server and AD logs to find log in attempts. You may receive complaints that things are slow, so monitor help desk calls. Systems that stop working may be a clue as well. If you can spend an hour, 30 minutes, even less to look at your logs on a daily basis, then you will be in a better position to identify weirdness. Once detected you can react.
For eradication, realistically the only safe option is to rebuild. Re-image, redeploy the system from known good media. You could attempt a removal process documented by an AV vendor or other organisation, just remember it wasn't picked up in the first place. Since the state of the machine is unknown you are really better off to rebuild, sorry.
the above is by no means complete so if you have anything to add, feel free to add a comment or let us know via the contact form.
Mark H - Shearwater
Nov 16th 2011
8 years ago
Having lived throught this a couple of times it is a challenge at best. Usually you have a worm or malware that hits the corporate file server that is storing 4TB of data. To run and AV scan on this box is painful. I highly recommend when an incident is upon you to enable Network File scanning om your AV product. This is a temp thing because it will hit most ofmthe boxes hard. However this scanning will catch the bug as it comes over the wire instead of waiting for on access scans.
I do agree that rebuilds are needed but we first have to figure out who else the box talked to. Examine the infected box and look for traces of contacting other workstations. Netstat, NBTSTAT, etc. if the bug is designed to call home and let the bad guy install software chances are they hoped over to another machine and are now running silent and deep.
I cannot stress enough, enable the firewall on the local machine and write rules that block all traffic from user subnets. if one machine gets infected its hard to jump from machine to machine. Now you can monitor the server subnets for interesting traffic. it can cut your investigation time in half. Regarding VPN's and laptop that have not checked in for a while, this is where you need you AV product to send you an email with a report of machines that have not checked in for 15 days.
Oh and you are running 802.1x on the wifi and the wire...right? If not, start working on management ASAP and convince desktop they can image the workstations on a private network.
Nov 17th 2011
8 years ago