Threat Level: green Handler on Duty: Tom Webb

SANS ISC: Analyzing HTTP Packet Captures - Internet Security | DShield SANS ISC InfoSec Forums


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
Analyzing HTTP Packet Captures

There are plenty of tools to extract files that are transmitted via HTTP. For example Jim Clausing's brilliant perl script [1], or the Wireshark "export features among many others (chaosreader, xplico, network miner ...).

However, I am sometimes faced with a different problem: You do have a network capture of a set of HTTP requests, and you are trying to "replay" them in all of their beauty, which includes all headers and POST data if present.

There are two parts to this challenge:

- extracting the HTTP requests from the packet capture.
- sending the packet capture to a web server

"tcpreplay" may appear like the right tool, but it will just blindly replay the traffic, and the web server will not actually establish a connection.

"wireshark" can be used to extract the data using the tcp stream reassembly feature, but this can't easily be scripted. "tshark" does not have a simple feature to just extract the http requests. You can only extract individual headers easily or the URLs.

The probably easiest way to parse the packet capture, and extract the request, is the perl module "Sniffer::HTTP". This module will not only reassemble the TCP streams, it will also extract HTTP requests:

#!/usr/bin/perl                                                                                                               

use Sniffer::HTTP;
my $VERBOSE=0;
my $sniffer = Sniffer::HTTP->new(
  callbacks => {
      request  => sub { my ($req,$conn) = @_; print $req->as_string,"n" if $req },
  }
);

$sniffer->run_file("/tmp/tcp80");

Will read packets from the file "/tmp/tcp80", and print the HTTP requests. The output could now be used to pipe it to netcat (or directly send it from perl).

 

[1] http://handlers.sans.org/jclausing/extract-http.pl

------
Johannes B. Ullrich, Ph.D.
SANS Technology Institute
Twitter

I will be teaching next: Defending Web Applications Security Essentials - SANS Munich July 2019

Johannes

3532 Posts
ISC Handler
I find this an easy to use/fun GUI for HTTP traffic:
http://thesz.diecru.eu/content/york.php
mbrownnyc

19 Posts
Regarding the statement: " "tshark" does not have a simple feature to just extract the http requests"...
I thought you could use the filter parameter, -R, and just copy-paste the filter used in Wireshark.

i.e.
-R "(http.request == True)"
should do it, if I remember correctly.
mbrownnyc
1 Posts
tshark filters/extracts the packets. But it does not easily extract the http request data. For example, -T fields -e http will just extract the string "http". You need to extract each header line individually. The best way I found with tshark is to convert the pcap file to tshark's XML version (-T pdml) and then post process that. But Sniffer::HTTP does it all in one step and the request should be compatible with LWP too.
Johannes

3532 Posts
ISC Handler
Have you tried httpry?

isc.sans.edu/…
Guy

439 Posts
ISC Handler
Sniffer::HTTP is great, but I don't think it does de-chunking. I used HTTP::Parser in StreamDB (code.google.com/p/streamdb/) to automatically gunzip and dechunk TCP streams from Vortex and carve out the objects (JPG, exe, etc.).
Guy
13 Posts
Yea...tshark can do this pretty slick like ;)
James

34 Posts

Sign Up for Free or Log In to start participating in the conversation!