The easy way to analyze huge amounts of PCAP data

Published: 2017-09-28
Last Updated: 2017-09-28 07:04:22 UTC
by Xavier Mertens (Version: 1)
4 comment(s)

When you are investigating a security incident, there are chances that, at a certain point, you will have to dive into network traffic analysis. If you’re lucky, you’ll have access to a network capture. Approximatively one year ago, I wrote a quick diary[1] to explain how to implement a simple FPC or “Full Packet Capture” solution based on a Docker container. It’s nice to capture all the traffic in PCAP files but then? PCAP files are not convenient to process and they consume a lot of disk space (depending on the captured traffic of course).

The next step is to have a tool ready to use to inspect the network traffic at any time. I tested multiple tools but my favourite one remains: Moloch[2]. It is a very good candidate to achieve this task: It helps you to capture traffic (optional), store/index flows in a database (at large scale because based on ElasticSearch) and, of course, search them. I’m also using Docker containers to run Moloch and ElasticSearch because they can be quickly deployed to investigate any PCAP even while on the road. They are two nice containers ready to use[3][4]. To deploy them, I use the following commands:

# docker run -d --name elasticsearch -v /data/elasticsearch:/usr/share/elasticsearch/data elasticsearch:5.2.2-alpine
# docker run -d --name moloch --link elasticsearch:elasticsearch -p 8005:8005 -v /data/tcpdump:/data/pcap:rw danielguerra/docker-moloch

Replace </data/elasticsearch> and <data/tcpdump> with your local directories that will contain, respectively, the ElasticSearch data and PCAP files to analyse.

By default, Moloch can sniff traffic passing by the default configured interface (eth0) but this is not possible with the Docker. If you configure the docker to have access to the local network resources (via ‘—net=host’), you won’t be able to access ElasticSearch. I’m using Moloch in two cases: during one-shot investigations (PCAP files provided by third-parties) or analysis of my own traffic. To achieve this, Moloch provides a nice tool called moloch-capture to ingest PCAP files.

In the first case, just process the provided PCAP file:

# docker exec -it moloch moloch-capture -r /data/pcap/evidence.pcap -t case_CUSTOMER_caseNO_YYYYMMYY

The tag (-t) is useful to search for packets related to a specific investigation.

In the second case, you must process new PCAP files when they are created:

# docker run -d --name moloch --link elasticsearch:elasticsearch -p 8005:8005 -v /data/tcpdump:/data/pcap:rw danielguerra/docker-moloch capture -m -R /data/pcap

Now, it will watch the directory /data/pcap for new “closed” PCAP files and index them. I would recommend you to use a tcpdump command like this:

/usr/sbin/tcpdump -i p1p1 -s 0 -G 1800 -W 96 -w /data/tcpdump/dump-%Y%m%d%H%M.pcap

tcpdump will create a new PCAP file every 1800" (30 mins) and keep an history of 96 * 30 mins (2 days).

Moloch is available through port 8005 and provides a nice interface:

It’s easy to analyse in deep protocols and extract interesting information like files from HTTP flows from the TCP headers up to the data present at the application layer:

All data being indexed in an ElasticSearch instance, you can build nice queries to search for specific pieces of information. Examples: 

protocols == dns && host.dns == /.*\.top/ 
port.dst == 80 && http.uri == /.*admin.*/

Other features that I like:

  • You can also add tags to interesting flows for further references or to share with other analysts. 
  • You can generate a permalink to give access to a specific flow to your colleagues
  • You can re-export interesting flows in other PCAP files

What about building a distributed full packet capture? Let’s imagine that you have hosts with sniffing capabilities on your network.  If you run the initial Docker container to collect PCAP files via those hosts, you can send them to the central Moloch server with a simple cron job like:

find /data/tcpdump -name “*.pcap” -mmin 5 -print | while read PCAP
do
    scp $PCAP moloch-server:/data/pcap
done

Files will be stored in the central repository and made available for moloch-capture, which will automatically index them. Once indexed, you don't need to keep the original PCAP files and ElasticSearch will take care of the long-term retention. 

Happy sniffing!

[1] https://isc.sans.edu/forums/diary/Full+Packet+Capture+for+Dummies/21679
[2] http://molo.ch/
[3] https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
[4] https://github.com/danielguerra69/docker-moloch

Xavier Mertens (@xme)
ISC Handler - Freelance Security Consultant
PGP Key

4 comment(s)
ISC Stormcast For Thursday, September 28th 2017 https://isc.sans.edu/podcastdetail.html?id=5688

Comments

What's this all about ..?
password reveal .
<a hreaf="https://technolytical.com/">the social network</a> is described as follows because they respect your privacy and keep your data secure:

<a hreaf="https://technolytical.com/">the social network</a> is described as follows because they respect your privacy and keep your data secure. The social networks are not interested in collecting data about you. They don't care about what you're doing, or what you like. They don't want to know who you talk to, or where you go.

<a hreaf="https://technolytical.com/">the social network</a> is not interested in collecting data about you. They don't care about what you're doing, or what you like. They don't want to know who you talk to, or where you go. The social networks only collect the minimum amount of information required for the service that they provide. Your personal information is kept private, and is never shared with other companies without your permission
https://thehomestore.com.pk/
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> public bathroom near me</a>
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> nearest public toilet to me</a>
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> public bathroom near me</a>
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> public bathroom near me</a>
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> nearest public toilet to me</a>
<a hreaf="https://defineprogramming.com/the-public-bathroom-near-me-find-nearest-public-toilet/"> public bathroom near me</a>
https://defineprogramming.com/
https://defineprogramming.com/
Enter comment here... a fake TeamViewer page, and that page led to a different type of malware. This week's infection involved a downloaded JavaScript (.js) file that led to Microsoft Installer packages (.msi files) containing other script that used free or open source programs.
distribute malware. Even if the URL listed on the ad shows a legitimate website, subsequent ad traffic can easily lead to a fake page. Different types of malware are distributed in this manner. I've seen IcedID (Bokbot), Gozi/ISFB, and various information stealers distributed through fake software websites that were provided through Google ad traffic. I submitted malicious files from this example to VirusTotal and found a low rate of detection, with some files not showing as malware at all. Additionally, domains associated with this infection frequently change. That might make it hard to detect.
https://clickercounter.org/
Enter corthrthmment here...

Diary Archives