Memory dumping for incident response is nothing new, but ever since they locked down access to direct memory (/dev/mem) on Linux, I’ve had bad experiences dumping memory.  I usually end up crashing the server about 60 percent of the time while collecting data with Fmem.

 

A new version of Linux memory dumping utility rekall (previous called Winpmem) has recently came out. I’ve been testing it on the latest versions of Ubuntu and Redhat EL 5 and have not run into any issues with collection.  

 

If you are fortunate enough to have an environment where you have groups of servers with the same patch levels, you should run the following steps on a non-compromised server.  Additionally, if the compromised system is a VM, you can clone it and perform these actions on the clone. Make sure you collect all other volatile data (MACtimes, LSOF, PS ect..) before you dump memory as this may still cause instability to the system and you do not want to lose this data.

 

Preparing for collection

Install Linux Kernel Headers

Ubuntu

>sudo apt-get install linux-headers-server zip

CentOS/Redhat

>yum install kernel-headers gcc

 

Download and Compile rekall

When you run the makefile, it will automatically create part of the profile for the server. This will need to be copied off the server for analysis.

>wget http://downloads.rekall.googlecode.com/git/Linux/linux_pmem_1.0RC1.tgz

>tar -zxvf linux_pmem_1.0RC1.tgz

>cd linux

>make

 

Note:For Redhat/CentOS systems you will need to adjust the Makefile KHEADER variable.

 

Copy this file to your Volatility analysis machine under your volatility directory /plugins/overlays/linux/.

Load the Kernel Driver

>sudo insmod pmem.ko

>sudo lsmod |grep pmem

pmem                   12680  0

 

Collect Memory

Now that the drive is loaded, a new device is accessible /dev/pmem. We want to copy the memory to an external device/share.

#Items in {} need to be changed per incident to be useful for analysis

>dcfldd if=/dev/pmem bs=512 conv=noerror,sync  of=/{USBDRIVE}/ mount/{servername.date}.memory.dd hash=md5,sha256  hashlog=/{USBDRIVE}/{servername.date}.memory.dd-hash.log

 

Unload driver

>sudo rmmod pmem.ko

Analysis using Rekall

Now that collection is completed, we need to be able to examine the memory dump.  Copy the memory image to your analysis workstation.

 

Install Rekall

>sudo apt-get install python-pip python-dev

>sudo pip install rekall

 

Build Rekall Profile

We now need to create a profile that will work with Rekall. Convert the file that was copied from the server and name it something useful for future analysis.

>rekal.py convert_profile 3.5.0-45-generic.zip Ubuntu3.5.0-45-generic.zip

>rekal.py --profile ./Ubuntu3.5.0-45-generic.zip -f /media/mem.dd  pslist

 

To enter the interactive shell, you do not add a plugin on the commandline

>rekal.py --profile ./Ubuntu3.5.0-45-generic.zip -f /media/mem.dd

 

To list the available plugins, use the interactive shell:

>rekal.py info[tab][tab]

 

plugins.arp              plugins.check_idt        plugins.convert_profile  plugins.dwarfparser      plugins.info             plugins.lsof             plugins.null             plugins.psaux            plugins.vmscan

plugins.banner           plugins.check_modules    plugins.cpuinfo          plugins.fetch_pdb        

…..  

 

To get more info about a specific plugin use a ? after plugin name

mem 12:38:31>plugins.pslist?

 

Some of the more useful plugins are:

• plugins.bash -searches for bash history

• plugins.check_modules- List loaded modules

• plugins.dmesg - Gathers dmesg buffer

• plugins.lsof

• plugins.netstat

• plugins.pslist

 

Optional (If you want to use Volatility for analysis)

I haven’t spent much time on this, but Volatility will not be able to use the rekall default profile. You also have to do the steps below to read the memory dump with Volatility.  I’m guessing only a small change in the file is needed, but I have dug any deeper at this time.

 

>sudo apt-get install dwarfdump

>wget  https://volatility.googlecode.com/files/volatility-2.3.1.tar.gz

>tar -zxvf volatility-2.3.1.tar.gz

>cd volatility-2.3.1/tools/linux

>make

>zip Ubuntu{Kernel ver}.zip ./module.dwarf  /boot/System.map-`uname -r`

 

For more information on Rekall

http://docs.rekall.googlecode.com/git/tutorial.html

 

For more info on Volatility Linux analysis

https://code.google.com/p/volatility/wiki/LinuxMemoryForensics

 

--

Tom Webb