Improving Bash Forensics Capabilities

Bash is the default user shell in most Linux distributions. In case of incidents affecting a UNIX server, they are chances that a Bash shell will be involved. Bash keeps an history to help the user to search (and reuse) his last commands:

$ history | tail -5
 1993  pwd
 1994  whoami
 1995  cd
 1996  cd /tmp
 1997  history | tail -5
$ !1996
cd /tmp
$ ^tmp^opt
cd /opt
$

When terminated, the Bash shell will saves the current history to a file. From a forensics point of view, this is a gold mine! Being able to browse the history of commands executed by a user is a must have. A few days ago, a friend published a blog post about a tip to improve the history of Bash shells. By defining the “HISTTIMEFORMAT” environment variable, you enable time stamps in the history:

$ export HISTTIMEFORMAT="%d/%m/%y %T "
$ history | tail -5
 1997  27/03/16 10:58:22 history | tail -5
 1998  27/03/16 10:58:33 cd /tmp
 1999  27/03/16 10:58:42 cd /opt
 2000  27/03/16 11:00:26 export HISTTIMEFORMAT="%d/%m/%y %T "
 2001  27/03/16 11:00:29 history | tail -5
$

While discussing with him about this topic, I realized that there are some ways to fine tune the history of commands to improve forensics investigations. In 2009, I also wrote a blog post about Bash which gave some ideas to send a Bash command history to a remote Syslog server. I checked my web logs and this blog post remains popular with more than 1000 visits for the last 30 days! Note that my blog post is outdated: Since the version 4.1, Bash supports Syslog natively but in most distribution, it is not enabled. To use this feature, you need to recompile your shell. I found this not very convenient but the good point is that it cannot be disabled by the user (except if he switches his shell to another shell or another Bash binary). You just have to define "SYSLOG_HISTORY" in config-top.h:

$ vi config-top.h
#define SYSLOG_HISTORY
#if defined (SYSLOG_HISTORY)
#  define SYSLOG_FACILITY LOG_USER
#  define SYSLOG_LEVEL LOG_INFO
#endif

./configure
make install

When a Bash shell is started, it loads the existing history from a flat file (by default $HOME/.bash_history) and saves it when it exists. Besides the "HISTTIMEFORMAT" environment variables, there are others which can also affect the way the logging of commands is performed. (and help investigations). Here is the list:

HISTFILE	The name of the file to which the command history is saved. The default value is `~/.bash_history`.
HISTFILESIZE	The maximum number of lines contained in the history file. When this variable is assigned a value, the history file is truncated, if necessary, to contain no more than that number of lines by removing the oldest entries. The history file is also truncated to this size after writing it when a shell exits. If the value is 0, the history file is truncated to zero size. Non-numeric values and numeric values less than zero inhibit truncation. The shell sets the default value to the value of `HISTSIZE` after reading any startup files.
HISTIGNORE	A colon-separated list of patterns used to decide which command lines should be saved on the history list. Each pattern is anchored at the beginning of the line and must match the complete line (no implicit ‘``’ is appended). Each pattern is tested against the line after the checks specified by `HISTCONTROL` are applied. In addition to the normal shell pattern matching characters, ‘`&`’ matches the previous history line. ‘`&`’ may be escaped using a backslash; the backslash is removed before attempting a match. The second and subsequent lines of a multi-line compound command are not tested, and are added to the history regardless of the value of `HISTIGNORE`. `HISTIGNORE` subsumes the function of `HISTCONTROL`. A pattern of ‘`&`’ is identical to `ignoredups`, and a pattern of ‘`[ ]`’ is identical to `ignorespace`. Combining these two patterns, separating them with a colon, provides the functionality of `ignoreboth`.
HISTCONTROL	A colon-separated list of values controlling how commands are saved on the history list. If the list of values includes ignorespace, lines which begin with a space character are not saved in the history list. A value of ignoredups causes lines matching the previous history entry to not be saved. A value of ignoreboth is shorthand for ignorespace and ignoredups. A value of erasedups causes all previous lines matching the current line to be removed from the history list before that line is saved.
HISTSIZE	The maximum number of commands to remember on the history list. If the value is 0, commands are not saved in the history list. Numeric values less than zero result in every command being saved on the history list (there is no limit). The shell sets the default value to 500 after reading any startup files.
HISTTIMEFORMAT (already discussed)	If this variable is set and not null, its value is used as a format string for strftime to print the time stamp associated with each history entry displayed by the `history` builtin. If this variable is set, time stamps are written to the history file so they may be preserved across shell sessions. This uses the history comment character to distinguish timestamps from other history lines.

You can also affect the way logging is performed with the “shopt” command. The following command will force Bash to append the current history to the history file instead of overwriting the current one:

$ shopt -s histappend

The cmdhist shell option, if enabled, causes the shell to attempt to save each line of a multi-line command in the same history entry, adding semicolons where necessary to preserve syntactic correctness:

$ shopt -s cmdhist

Finally, lithist saves the command with embedded newlines instead of semicolons. This could be helpful for further processing:

$ shopt -s lithist

Also interesting, when you use the HISTTIMEFORMAT environment variable, the .bash_history file format is changed and timestamps (in UNIX epoch format) are added:

$ tail -10 .bash_history
#1458933529
history|less
#1458933544
vi .bash_history
#1458933792
wc -l .bash_history
#1458976122
more .bash_history
#1458976132
echo foobar

You can add the environment variable in /etc/bash.bashrc or, per user, in $HOME/.bashrc. Note that these environment variables do not prevent a malicious user to disable the Bash history! Just spawn another shell (zsh, ksh) and you will escape the logging features. If you really want to track what users are doing, have a look at psacct which runs in the background to track users activity (not only from Bash).

Happy Easter break!

Xavier Mertens
ISC Handler - Freelance Security Consultant
PGP Key

I will be teaching next: Reverse-Engineering Malware: Malware Analysis Tools and Techniques - SANS Amsterdam August 2020 Part 2

Xme

533 Posts
ISC Handler

Mar 28th 2016

Note also that all commands preceded with white space will not be included in the history at all. Does this also affect syslog history?

Anonymous

Actually the white space trick no longer works. I just tested to double check, but both my Mac (OS X 10.11.4, bash versio 3.1.57) as well as a Centos 6.7 system (bash 4.1.2) and a CentOS 7.2 system (bash 4.2.46) will log commands that are preceded by a space.

Johannes

3882 Posts
ISC Handler

Although not totally foolproof, I guess you could declare HISTFILE as read only (ex: readonly HISTFILE) as well. I've seen several automated exploits that try to unset HISTFILE or change it to /dev/null before doing anything else.

DR

4 Posts