SANS ISC: Capture and Analysis of User Agents

• SANS Site Network
  • Current Site
  • SANS Internet Storm Center
  • Other SANS Sites Help
  • Graduate Degree Programs
  • Security Training
  • Security Certification
  • Security Awareness Training
  • Penetration Testing
  • Industrial Control Systems
  • Cyber Defense Foundations
  • DFIR
  • Software Security
  • Government OnSite Training

SANS ISC InfoSec Forums

ISC collects web logs which also includes User-Agents. If you are running a honeypot or a web server, it is fairly easy to quickly use some Regex to parse the logs and get a count of what is most commonly seen. This is some of the activity I have observed over the past week, some well know user-agent associated with valid browser versions and some custom that are telltale to hacking tools:

86 User-Agent: Mozilla/5.0
15 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7
14 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
13 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
11 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
10 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
9 User-Agent: Hello, World
8 User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-US) AppleWebKit/532.8 (KHTML, like Gecko) Chrome/4.0.302.2 Safari/532.8
6 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0
3 User-Agent: Go-http-client/1.1
2 User-Agent: Mozilla/5.0 zgrab/0.xAccept: */*
2 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36
1 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
1 User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0
1 User-Agent: Mozilla/5.0(WindowsNT6.1;rv:31.0)Gecko/20100101Firefox/31.0
1 User-Agent: Mozilla/5.0 (Linux; Android 8.1.0; Pixel 2 Build/OPM2.171019.029.B1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36
1 User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)

This is the regex I used to parse my honeypot logs:

cat tcp-honeypot-*.log | sed 's/.*\(User-Agent.*\)Content.*/\1/g' | sed 's/.*\(User-Agent.*\)\Accept.*/\1/g' | sed 's/.*\(User-Agent.*\)\Connection.*/\1/g' | sed 's/.*\(User-Agent.*\)\Host.*/\1/g' | sed 's/\\r\\n//g' | sort | uniq -c | sort -h -r > agent.txt

If you are interested in participating in this ISC project, you can follow this link which explains how to setup a honeypot to participate.

[1] https://developers.whatismybrowser.com/useragents/explore/software_name/safari/
[2] https://developers.whatismybrowser.com/useragents/explore/software_name/chrome/2
[3] https://developers.whatismybrowser.com/useragents/explore/software_name/internet-explorer/
[4] https://developers.whatismybrowser.com/useragents/explore/layout_engine_name/gecko/107
[5] https://developers.whatismybrowser.com/useragents/explore/software_name/
[6] https://github.com/zmap/zgrab
[7] https://isc.sans.edu/weblogs/
[8] https://isc.sans.edu/honeypot.html

-----------
Guy Bruneau IPSS Inc.
Twitter: GuyBruneau
gbruneau at isc dot sans dot edu