SANS ISC: Analysis of Competing Hypotheses (ACH part 1) - Internet Security | DShield

• SANS Site Network
  • Current Site
  • Internet Storm Center
  • Other SANS Sites Help
  • Graduate Degree Programs
  • Security Training
  • Security Certification
  • Security Awareness Training
  • Penetration Testing
  • Industrial Control Systems
  • Cyber Defense Foundations
  • DFIR
  • Software Security
  • Government OnSite Training

SANS ISC InfoSec Forums

In threat intelligence, by definition, an analyst will most of the times have to perform assessments in an environment of incomplete information, and/or with information that is being produced with the purpose of misleading the analyst.

One of the well-known methodologies is the Analysis of Competing Hypotheses (ACH) [1], developed by Richards J. Heuer, Jr., a former CIA veteran. ACH is an analytic process that identifies a set of alternative hypotheses, and assesses whether data available are either consistent or inconsistent with each hypothesis. The hypotheses with most inconsistent data will be rejected. To quote Heuer’s words

ACH is a tool to aid judgment on important issues requiring careful weighing of alternative explanations or conclusions. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve.

It is very important to note that the goal of ACH is to reject as many hypotheses as possible, not to confirm any.

One of the advantages of using ACH is that it reduces analysts’ confirmation bias. In fact, a common pitfall during analysis usually is to form a hypothesis on our head and to look for evidences that support it, confirming indeed our initial line of thought following the “most likely” hypothesis. However, such approach does not take into account possible alternate hypotheses, as well as what other data is missing that should be there if that given hypothesis would be true. This is achieved by imposing the analyst to identify, and then refuse, as many competing hypotheses as possible using all data available.

How it works

ACH requires the analyst to collect all the related information and organize them in a matrix: all the hypotheses on the top (first row), and all the relevant information on the left side (the first column). In this way, every piece of information can be evaluated against each of the hypotheses, by assessing if it’s consistent or inconsistent. Heuer describes the full process in eight steps, which could be summarized as follow:

1. Identify all hypotheses. Ideally, all hypotheses should be mutually exclusive, meaning that if one is true all the others must be false.
2. Lists evidences and arguments relevant for and against each hypothesis. This has to include also assumptions and logical deductions.
3. Create a matrix as described above and analyze each evidence against every hypotheses by defining whether is Consistent, Inconsistent or Not applicable / Not relevant, in an attempt to disprove as many hypotheses as possible. In Heuer words, analyze the “diagnosticity” of the evidence. Moreover, it is important to provide each evidence with a credibility and relevance level (i.e. high, medium or low), as it will help in the confidence of the final assessment.
4. Refine the matrix: review the findings, identify any gaps, collect any additional evidence needed, delete evidences that have no diagnostic value.
5. Draw tentative conclusions about the relative likelihood of each hypothesis, trying to disprove them. Less consistency implies a lower likelihood.
6. Analyze the sensitivity of the analysis to few critical evidences.
7. Report conclusions based on the likelihood of all hypotheses, not only the most likely one, presenting also a summary of alternatives that were considered and why they were rejected.
8. Identify milestones for future observation

The following is an example of the final matrix from a post by Scott J. Roberts, where he applies ACH to understand whether or not the Republican National Committee and Donald J. Trump for President, Inc were victims of similar attacks as the Democratic National Committee [2]. In his post he goes step-by-step on how he created and refined his ACH matrix.

Conclusions

ACH is just one of the possible structured analytic techniques available. I personally like it very much and find it quite useful. But remember that while the matrix helps in creating a model for the analysis of problems with conflicting information and it generates a definitive mathematical total for each hypothesis, at the end it is still up to the analyst to use his/her judgment to make the final conclusion.

This was a very brief introduction of ACH, and in my next diary I will apply ACH to a practical recent case. But I definitely encourage those interested to go through Heuer’s book for a deeper explanation of such model.

Happy Hunting,
Pasquale

References:

[1] – Richards J. Heuer, Jr.; “Psychology of Intelligence Analysis”, Center for the Study of Intelligence, Central Intelligence Agency. https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis
[2] – Scott J. Roberts, “ACH Analysis of a Trump Campaign Compromise”, https://sroberts.github.io/2016/12/12/rnc-hack/

Pasquale Stirparo, Ph.D.
@pstirparo