Apache Tomcat is a java based web service that is used for different applications. While you may have it running in your environment, you may not be familiar with its workings to provide adequate incident response  when the time come. This article will walk through an incident where Tomcat is used and what critical artifacts you should collect.

This articles assumes that you have already collected MACtimes (hxxps://digital-forensics.sans.org/blog/2011/12/07/digital-forensic-sifting-super-timeline-analysis-and-creation) and other volatile data.

Locating Install Directory

Tomcat can be installed in several different locations, but the most common are for Redhat: /usr/share/apache-tomcat-(version) and for Ubuntu: /usr/share/tomcat or /var/lib/tomcat. To be sure you have the right locations track down the Tomcat process and analyze it.

    #ps -auxwww |fgrep -i tomcat

    0 S root     31847     1  0  80   0 - 1124641 futex_ 2015 ?       02:36:33 /usr/bin/java  -classpath /usr/share/apache-tomcat-7.0.65/bin/bootstrap.jar …..

Here you can see that it is running from /usr/share/apache-tomcat-7.0.65.  Depending on the install this will likely contain all the artifacts for tomcat that we need but we should still verify. If you are not getting an entire copy of the system, make sure that you get a backup of this directory along with the other critical ones.

Tomcat Logs

On the system I dealt with, the logs were in a subfolder called logs (e.g. /TOMCAT_HOME/logs). Check to make sure you have stuff in it, if not it is likely using syslog or the attacker cleared it out.

Catalina.out file is output from the tomcat console into this file. It is a multi-line log file, but has time and date stamps. This file shows if modules are added and other output that is useful for IR. Access_log is in common Apache format, but it is a simple format that does not have the user agent information. The manager.log and host-manager.log really did not have any useful information in my case.

For more information about Tomcat loging visit hxxps://tomcat.apache.org/tomcat-7.0-doc/logging.html

Web Applications Locations

Web applications are located on TOMCAT_HOME/webapps/. To deploy a web app, you must use the Tomcat web manager which you will see GETS and POSTS to /manager/ folder in the access logs.

For more info on Tomcat app deployment visit hxxps://tomcat.apache.org/tomcat-4.1-doc/appdev/deployment.html

Config File Locations

The Tomcat configurations are located in the TOMCAT_HOME/config directory.  Server.xml is the main server config file. Tomcat-users.xml is a list of users and passwords for the system. The default admin password is:admin:password.

The Incident

Now that we know where to look, let’s go over the incident.  A system was discovered to be compromised so I started our IR process. When looking at the processes running, a process was quickly changing its name and running as root.  When looking at the list of open files for the process, I got a hint that tomcat might have something to do with the compromise.

#ps -auxwww

10346 root 20 0 21852 952 212 S 25.9 0.0 7103:14 qymasclksx

#lsof -p 10346

qymasclks 10346 root   DIR 253,6 4096 275525 /usr/share/apache-tomcat-7.0.65/bin

qymasclks 10346 root rtd DIR 253,0 4096 2 /

qymasclks 10346 root txt REG 253,6 619090 3154 /usr/bin/qymasclksx

Lets see if any files have changed in the Tomcat directory recently to get an idea of possible time of compromise. The file in /usr/bin/qymasclksx is being deleted and recreated every few seconds, so that will not be helpful for initial compromise time.

 

Tue Dec 01 2015 05:58:38,493137,mac.,-rw-r--r--,0,0,0,"/usr/share/apache-tomcat-7.0.65/webapps/eei.war"

Tue Dec 01 2015 05:58:38,69334,.ac.,-rw-r--r--,0,0,0,"/usr/share/apache-tomcat-7.0.65/webapps/eei/a.jsp"

 

There is a new file eei.war that has been created. Lets take a look at the log files and see what we can get from that time frame.  

#fgrep “Dec 01, 2015” Catalina.out

Dec 01, 2015 5:58:38 AM org.apache.catalina.startup.HostConfig deployWAR

INFO: Deploying web application archive /usr/share/apache-tomcat-7.0.65/webapps/eei.war

Dec 01, 2015 5:58:38 AM org.apache.catalina.startup.HostConfig deployWAR

INFO: Deployment of web application archive /usr/share/apache-tomcat-7.0.65/webapps/eei.war has finished in 118 ms

 

You can see that a new application has been deployed, which means the attacker had access to the Tomcat admin.  Let look at the access_logs to see if we can get more detail.

 

#fgrep “01/Dec/2015” access_logs

122.236.51.194 - - [01/Dec/2015:05:58:08 -0500] "GET /manager/html HTTP/1.1" 401 2474

122.236.51.194 - admin [01/Dec/2015:05:58:09 -0500] "GET /manager/html HTTP/1.1" 200 19270

122.236.51.194 - admin [01/Dec/2015:05:58:39 -0500] "POST /manager/html/upload?org.apache.catalina.filters.CSRF_NONCE=4C0343589816E985E2010C618944EF5A HTTP/1.1" 200 20940

122.236.51.194 - - [01/Dec/2015:05:58:43 -0500] "GET /eei/ HTTP/1.1" 200 3319

122.236.51.194 - - [01/Dec/2015:05:58:45 -0500] "POST /eei/ HTTP/1.1" 200 6383

122.236.51.194 - - [01/Dec/2015:05:58:49 -0500] "GET /eei/?action=command HTTP/1.1" 200 2677

122.236.51.194 - - [01/Dec/2015:05:58:55 -0500] "POST /eei/?action=command HTTP/1.1" 200 2736

122.236.51.194 - - [01/Dec/2015:05:58:58 -0500] "POST /eei/?action=command HTTP/1.1" 200 2725

Here you can see that the attacker accessed the Tomcat management as the admin user and uploaded a file. Then the attacker access another page that accepts a command via a POST. Let’s see how the attacker was able to gain access as the admin user to the manager site. By viewing the tomcat-users.xml file, we can see that a very common username/password is being used.

 

/usr/share/apache-tomcat-7.0.65/conf/tomcat-users.xml

-->

 

The package that was installed was  “jsp File browser 1.1a”. This allowed the attacker to install his backdoor/DDOS tool called Xorddos. Mandint did a write-up about this and mostly what I found was similar to their findings. (hxxps://www.fireeye.com/blog/threat-research/2015/02/anatomy_of_a_brutef.html)

Other IOC(s)

MD5 filehash:        

 968c4e06ff32d97b1f2b3f2ce3bcbb7e, gfty

 a568167a5950ae55f6f442d959de4476, libkill.so

 cac58ebacb036f706d58ec9f47ee64cc, eei.war

 

filename:      

 c.rar

 libkill.so

 cmd.n

 eei.war

 cron.sh

 kill.sh

 

filepath:        

/usr/share/apache-tomcat/webapps/eei

/lib

/etc/cron.hourly/

/lib/udev

 

IPS:

222.186.21.166 (Download site w/ wget)

122.236.51.194 (Attacker IP)

23.234.60.143 (C.rar download every 30 min)

--

Tom Webb