Using The Coroner's Toolkit : Harvesting information with grave-robber

Implementation Details
Applies to the practice:
Analyze all available information to characterize an intrusion

Applicable technologies:
Sun Solaris 2.x; UNIX operating systems and derivatives
 
Introduction In the aftermath of a network break-in, system administrators are often asked to explain what happened. The Coroner's Toolkit (TCT) is a collection of tools that gather and analyze data on a UNIX system and help the administrator answer that question. grave-robber is the central application of the toolkit used to achieve this goal. This program controls several other tools in an attempt to capture as much information as possible about a potentially compromised system and its files. 

As the TCT authors point out: "If there was a theme, it would be the reconstruction of the past - determining as much as possible what happened with a static snapshot of a system." Certainly such activities require an experienced and committed system administrator during the forensic investigation phase of an intrusion. No software can replace someone who knows his or her system, but TCT is a start. 

In addition to grave-robber, TCT contains two tools, unrm and lazarus,which can work together to help  identify what happened after a break-in. These tools can also be used to recover  files deleted by accident. While unrm makes a copy of all accessible, unallocated (free) disk space, lazarus analyzes this copy and tries to determine each block of data and its type (text versus graphics, etc.). Their use is explained in another implementation.

Using the TCT tools can require a great deal of time and effort. You need to review all documentation carefully and test all parts before use so that you can understand and take full advantage of their features. In particular, when using grave-robber, you need to read the file help-when-broken-into which comes as part of the TCT documentation. 

The installation of the TCT toolkit is explained in the implementation Installing The Coroner's Toolkit and using the mactime utility.

This implementation discusses the use of one TCT tool grave-robber on the Sun Solaris operating system, version 2.x. You can use this approach with other UNIX operating systems and hosts.
Effort Estimates The time needed to retrieve all data collectable by the TCT tools depends on the underlying system, its processor, and required disk space. Using these tools can take considerable time and can create a large volume of output data. The technical analysis of this output can easily take hours or days.
Prerequisites The output of grave-robber can take considerable space. For example, for a system with 1.0 GB of data,  grave-robber will typically produce anywhere from 50 to 200 KB of output. You need to make sure that you have set aside sufficient disk space to store the output. During forensic analysis, changes to the system being analyzed should be as small as possible. Therefore we recommend against storing the grave-robber output data on the disks being analyzed. In addition, if you need to search for deleted files, any write access to the disks can destroy the valuable data you are seeking. 

Make sure that spare disks are reserved and available when you need them to store the grave-robber output. Make sure you know how to connect such disks to a live system such that minimal changes occur on the system being analyzed. In particular, ensure that no reboot occurs to avoid killing running processes before you have a chance to grab information about them with grave-robber. 

To reliably analyze any system, you must use unmodified, authentic tools. Therefore, use write-protected media to store tools like the TCT and others used during forensic analysis.
Understanding the grave-robber output grave-robber can be used in three different modes. The output will vary depending on the mode used as explained below. 
  1. Life system, default data collection:

    2. By default, all data is collected except data only available by dumping the memory content of system processes. 
  2. Life system, maximal data collection:

    3. In addition to the default settings, information from active processes is collected. 
  3. Backup copy or mounted disk of another system (corpse):

    4. As no life system is available for analysis purposes, only the available disk space is analyzed.
We recommended running grave-robber from a simple terminal if a life system is being analyzed. The program may kill a windows system while retrieving data from system processes!
Directory structure of the grave-robber output.
  The availability of specific information depends on the running operating system. TCT relies on native programs to collect as much data as possible. 

Several directories and files are created within the directory used to store all grave-robber output: 

Files:

  1. coroner.log

    2. This file lists all program executions that were initiated with time stamp information and arguments. 
  2. error.log

    3. This file contains all error messages that occurred during the data collection. 
  3. body

    4. This is the mactime database which lists all attributes of all files examined such as its MD5 checksum, inode information, and access time stamps. 
  4. body.S

    5. This file has the same structure as body but contains information about SetUID files only. 
  5. MD5_all

    6. This file lists all MD5 checksums for all output files generated by grave-robber. 
  6. MD5_all.md5

    7. This file lists the MD5 checksum of the file MD5_all only. You can identify changes to the MD5 database file by comparing the actual MD5 checksum with this file.
Directories:

All files contained within these subdirectories are protected with a MD5 checksum file. The name of this file is created by taking the original file name used by grave_robber and adding .md5 as an extension. 

  1. command_out

    2. This subdirectory keeps the output of most of the programs that are executed from within grave-robber. The files are named based on the command run and its arguments. 
  2. conf_vault

    3. All files that are of interest for grave-robber are copied into this directory. This includes configuration files, critical files, and directories, etc. The list of files is controlled by several configuration files: 
    • save_these_files
    • coroner.cf
    • grave-robber.cf
  3. pcat

    4. pcat is used to recover images of running processes. Much important information can be found in process memory such as IP addresses, passwords, etc. 
  4. proc

    5. This directory also contains images of running processes but based on the proc file system. 
  5. removed_but_running

    6. This directory contains all deleted files that were still open or running at the time of data gathering. 
  6. trust

    7. Within this directory, all files related to "trust" and trusted relationships that could be exploited are copied. Currently this will grab .rhosts, forward files, and dumps crontab and at output as well as xhost / xauth results. 
  7. user_vault

    8. For all users, copies of sensitive files found while analyzing the file system (such as SSH key files, history files for various shells, etc.) are stored here.
Running grave-robber on a life system In order to start grave-robber, you need to mount your secured copy of TCT. (The creation of a secured copy based on a write-protected floppy-disk is explained in the implementation Installing The Coroner's Toolkit and using the mactime utility.) 

This step assumes that the directory /tct is already available on the system and that a spare disk has been mounted as /tct-data.  Create this directory on all systems that may be used for forensic analysis in advance of any incident. This will avoid your having to create this directory in order to be able to mount the spare disk. 

After the TCT toolkit disk is mounted, change to the data directory on the spare disk. Start script to maintain a detailed log of all information displayed on the terminal. 

# mount /dev/diskette /tct
# cd /tct-data
# script tct-20001031.log

Now start grave-robber to collect the default set of data. Option -d directs the tool to use the actual directory (/tct-data) as the directory to store all output. -v directs the tool to create a more verbose explanation of its progress. The last argument / controls which directory is used as the starting point for any disk analysis: 

# /tct/bin/grave-robber -d . -v /
# exit

If you want to collect the maximum set of data, you need to additionally provide option -E as shown below. 

# /tct/bin/grave-robber -d . -E -v /

While running  grave-robber, avoid any other activities on the system being analyzed.
Running grave-robber on a corpse system A clean and secure system should be used if a corpse system is being analyzed. It is not necessary to use a new spare disk to collect the grave-robber output if enough local disk space is available. For this step, we assume that the hard disk being analyzed is already mounted as /tct-corpse. Alternatively, a copy of the file system of a potentially compromised system can be copied to this location for further analysis. 

Create a data directory for the grave-robber output and change to it. Start script to maintain a detailed log of all information displayed on the terminal. 

# mkdir /tct-data
# cd /tct-data
# script tct-20001031.log

Now start grave-robber with option -c <path-to-corpse-directory> to collect the set of data available for the analysis of corpses. Option -d directs the tool to use the actual directory (/tct-data) as the directory to store all output. -v directs the tool to create a more verbose explanation of its progress. The -o option tells the tool which operating system was running on the system to adjust for specific differences: 

# /tct/bin/grave-robber -d . -v -c /tct-corpse -o SUNOS5
# exit

Option -o can be used with the following keywords indicating a supported operating system: 

  • SUNOS5 and SUNOS4
  • LINUX2
  • BSDI2 and BSDI3
  • OPENBSD2
  • FREEBSD2 and FREEBSD3