Analysis of Competing Hypotheses (ACH part 1) - Internet Security

Analysis of Competing Hypotheses (ACH part 1)
In threat intelligence, by definition, an analyst will most of the times have to perform assessments in an environment of incomplete information, and/or with information that is being produced with the purpose of misleading the analyst. One of the well-known methodologies is the Analysis of Competing Hypotheses (ACH) [1], developed by Richards J. Heuer, Jr., a former CIA veteran. ACH is an analytic process that identifies a set of alternative hypotheses, and assesses whether data available are either consistent or inconsistent with each hypothesis. The hypotheses with most inconsistent data will be rejected. To quote Heuer’s words ACH is a tool to aid judgment on important issues requiring careful weighing of alternative explanations or conclusions. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve. It is very important to note that the goal of ACH is to reject as many hypotheses as possible, not to confirm any. One of the advantages of using ACH is that it reduces analysts’ confirmation bias. In fact, a common pitfall during analysis usually is to form a hypothesis on our head and to look for evidences that support it, confirming indeed our initial line of thought following the “most likely” hypothesis. However, such approach does not take into account possible alternate hypotheses, as well as what other data is missing that should be there if that given hypothesis would be true. This is achieved by imposing the analyst to identify, and then refuse, as many competing hypotheses as possible using all data available. How it works ACH requires the analyst to collect all the related information and organize them in a matrix: all the hypotheses on the top (first row), and all the relevant information on the left side (the first column). In this way, every piece of information can be evaluated against each of the hypotheses, by assessing if it’s consistent or inconsistent. Heuer describes the full process in eight steps, which could be summarized as follow: Identify all hypotheses. Ideally, all hypotheses should be mutually exclusive, meaning that if one is true all the others must be false. Lists evidences and arguments relevant for and against each hypothesis. This has to include also assumptions and logical deductions. Create a matrix as described above and analyze each evidence against every hypotheses by defining whether is Consistent, Inconsistent or Not applicable / Not relevant, in an attempt to disprove as many hypotheses as possible. In Heuer words, analyze the “diagnosticity” of the evidence. Moreover, it is important to provide each evidence with a credibility and relevance level (i.e. high, medium or low), as it will help in the confidence of the final assessment. Refine the matrix: review the findings, identify any gaps, collect any additional evidence needed, delete evidences that have no diagnostic value. Draw tentative conclusions about the relative likelihood of each hypothesis, trying to disprove them. Less consistency implies a lower likelihood. Analyze the sensitivity of the analysis to few critical evidences. Report conclusions based on the likelihood of all hypotheses, not only the most likely one, presenting also a summary of alternatives that were considered and why they were rejected. Identify milestones for future observation The following is an example of the final matrix from a post by Scott J. Roberts, where he applies ACH to understand whether or not the Republican National Committee and Donald J. Trump for President, Inc were victims of similar attacks as the Democratic National Committee [2]. In his post he goes step-by-step on how he created and refined his ACH matrix. Conclusions ACH is just one of the possible structured analytic techniques available. I personally like it very much and find it quite useful. But remember that while the matrix helps in creating a model for the analysis of problems with conflicting information and it generates a definitive mathematical total for each hypothesis, at the end it is still up to the analyst to use his/her judgment to make the final conclusion. This was a very brief introduction of ACH, and in my next diary I will apply ACH to a practical recent case. But I definitely encourage those interested to go through Heuer’s book for a deeper explanation of such model. Happy Hunting, Pasquale References: [1] – Richards J. Heuer, Jr.; “Psychology of Intelligence Analysis”, Center for the Study of Intelligence, Central Intelligence Agency. https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis [2] – Scott J. Roberts, “ACH Analysis of a Trump Campaign Compromise”, https://sroberts.github.io/2016/12/12/rnc-hack/ Pasquale Stirparo, Ph.D. @pstirparo	Pasquale Stirparo 18 Posts ISC Handler
Subscribe	May 28th 2017 2 years ago
Interesting idea for analysis methodology. I'll be looking forward to the second part. The image is missing.	Rastech 18 Posts
Quote	May 28th 2017 2 years ago
Thank you for the feedback and for the notice, image has been fixed. Part 2 should be online in the coming days, stay tuned.	Pasquale Stirparo 18 Posts ISC Handler
Quote	May 28th 2017 2 years ago

In threat intelligence, by definition, an analyst will most of the times have to perform assessments in an environment of incomplete information, and/or with information that is being produced with the purpose of misleading the analyst.

One of the well-known methodologies is the Analysis of Competing Hypotheses (ACH) [1], developed by Richards J. Heuer, Jr., a former CIA veteran. ACH is an analytic process that identifies a set of alternative hypotheses, and assesses whether data available are either consistent or inconsistent with each hypothesis. The hypotheses with most inconsistent data will be rejected. To quote Heuer’s words

ACH is a tool to aid judgment on important issues requiring careful weighing of alternative explanations or conclusions. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve.

It is very important to note that the goal of ACH is to reject as many hypotheses as possible, not to confirm any.

One of the advantages of using ACH is that it reduces analysts’ confirmation bias. In fact, a common pitfall during analysis usually is to form a hypothesis on our head and to look for evidences that support it, confirming indeed our initial line of thought following the “most likely” hypothesis. However, such approach does not take into account possible alternate hypotheses, as well as what other data is missing that should be there if that given hypothesis would be true. This is achieved by imposing the analyst to identify, and then refuse, as many competing hypotheses as possible using all data available.

How it works

ACH requires the analyst to collect all the related information and organize them in a matrix: all the hypotheses on the top (first row), and all the relevant information on the left side (the first column). In this way, every piece of information can be evaluated against each of the hypotheses, by assessing if it’s consistent or inconsistent. Heuer describes the full process in eight steps, which could be summarized as follow:

Identify all hypotheses. Ideally, all hypotheses should be mutually exclusive, meaning that if one is true all the others must be false.
Lists evidences and arguments relevant for and against each hypothesis. This has to include also assumptions and logical deductions.
Create a matrix as described above and analyze each evidence against every hypotheses by defining whether is Consistent, Inconsistent or Not applicable / Not relevant, in an attempt to disprove as many hypotheses as possible. In Heuer words, analyze the “diagnosticity” of the evidence. Moreover, it is important to provide each evidence with a credibility and relevance level (i.e. high, medium or low), as it will help in the confidence of the final assessment.
Refine the matrix: review the findings, identify any gaps, collect any additional evidence needed, delete evidences that have no diagnostic value.
Draw tentative conclusions about the relative likelihood of each hypothesis, trying to disprove them. Less consistency implies a lower likelihood.
Analyze the sensitivity of the analysis to few critical evidences.
Report conclusions based on the likelihood of all hypotheses, not only the most likely one, presenting also a summary of alternatives that were considered and why they were rejected.
Identify milestones for future observation

The following is an example of the final matrix from a post by Scott J. Roberts, where he applies ACH to understand whether or not the Republican National Committee and Donald J. Trump for President, Inc were victims of similar attacks as the Democratic National Committee [2]. In his post he goes step-by-step on how he created and refined his ACH matrix.

Conclusions

ACH is just one of the possible structured analytic techniques available. I personally like it very much and find it quite useful. But remember that while the matrix helps in creating a model for the analysis of problems with conflicting information and it generates a definitive mathematical total for each hypothesis, at the end it is still up to the analyst to use his/her judgment to make the final conclusion.

This was a very brief introduction of ACH, and in my next diary I will apply ACH to a practical recent case. But I definitely encourage those interested to go through Heuer’s book for a deeper explanation of such model.

Happy Hunting,
Pasquale

References:

[1] – Richards J. Heuer, Jr.; “Psychology of Intelligence Analysis”, Center for the Study of Intelligence, Central Intelligence Agency. https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis
[2] – Scott J. Roberts, “ACH Analysis of a Trump Campaign Compromise”, https://sroberts.github.io/2016/12/12/rnc-hack/

Pasquale Stirparo, Ph.D.
@pstirparo

Pasquale Stirparo