Computer-based test interpretation in psychological assessment

Present status

Currently, CBTI programs fall into one of two categories: actuarial assessment programs or automated assessment programs. Actuarial assessment programs are based on statistical or actuarial prediction (e.g., statistical analyses, linear regression equations and Bayesian rules), which is empirically based while automated assessment programs consist of a series of if-then statements derived by expert clinicians and informed by published research and clinical experience. For the purposes of this article, both types will be referred to as computer-based test interpretations (CBTIs). The use of CBTIs is found in a variety of psychological domains (e.g., clinical interviewing and problem rating), but is most commonly utilized in personality and neuropsychological assessments. This article will focus on the use of CBTIs in personality assessment, most commonly using the MMPI and its subsequent revised editions.

Reliability

The ability for CBTIs to eliminate human-error is considered a benefit, and as a result reliability of CBTIs is considered to be better than those of clinician interpretations. However, CBTIs have demonstrated poor reliability. Research regarding the equivalence of CBTIs and paper-and-pencil measures has been found to be equivocal (for reviews see). Further, CBTI research has been criticized for failure to assess inter-rater (comparing the interpretation of one protocol by two different programs) and internal consistency reliability (comparing the reliability of different sections of the same interpretation). On the other hand, test-retest reliability of CBTIs is considered perfect (i.e., the same protocol will repeatedly yield the same interpretation), if the same program is used.

Validity

Research on the validity of CBTIs tends to utilize three types of studies: external criterion studies (comparing the CBTI report to some external criterion measure of the construct, such as a self-report or behavioral measure), consumer satisfaction studies (asking clients whether the reports are accurate representations of themselves), and comparison with clinical conclusions (comparing CBTI reports to clinician interpretations). Comprehensive reviews of CBTI validity can be found elsewhere (e.g.,). In general, the validity of CBTIs has not been demonstrated, and the validity of individual CBTI systems has been found to vary. However, many validity studies are flawed due to small samples, criterion contamination, the Barnum effect, inadequate input data to generate powerful statistical prediction rules, unreliability of measures and the practice of generalizing across testing situations and populations without considering potential moderators.

Strengths and weaknesses

CBTI programs can be found for nearly every type of personality assessment available today. CBTI programs arguably have many benefits over traditional hand-scored assessments and clinician interpretations which may contribute to their popularity. For example, CBTI programs save time and eliminate human responding and scoring errors. Further, CBTI programs are often more comprehensive than clinician interpretation, tend to be more reliable than clinician interpretation, are cost effective, and more objective which may allow clients to be more accepting of feedback. Despite these benefits, there are significant limitations of CBTIs to consider. For example, CBTI reports may suggest an unwarranted impression of scientific precision and reports may be too general to provide differential information. Additionally, CBTIs may promote exceedingly cavalier attitudes towards clinical assessment and interpretation, and as they are increasingly available to inadequately trained evaluators, the potential for misuse is high. Clinicians are cautioned to educate themselves before using CBTI programs, not to blindly interpret computer-generated reports as true or use CBTIs as a way to circumvent their responsibilities as a clinician to integrate multiple sources of data.

Future

As our healthcare system and society as a whole becomes increasingly reliant on technology, it is inevitable that the availability and use of CBTI software will also expand. The potential of the internet for extending the use of CBTIs has been recognized, although the potential problems associated with this modality have yet to be fully understood and will need to be addressed before the use of internet-based CBTI utilization proliferates. In addition, the application of computer-adaptive testing, although successfully applied in other assessment domains (i.e., ability and aptitude), provides a promising, yet under researched addition to personality assessment. Lastly, there is a call for the more effective integration of clinical and computer-based prediction methods, beginning with a partnership between clinicians and researchers in the development of CBTI programs.