information as the operator does and fuses the information to produce an environmental estimate in more comprehensible form, i.e., probability of attack. Therefore, a Lens model was developed, called hybrid Lens model, by combining a triple system and the hierarchical system designs. Associated Lens model parameters were redefined to reflect a variety of characteristics of the judgment agents; human operator, ADA or human judgment with ADA.
In this environment described above, human operators’ understanding about the ADA is critical in making decisions whether to accept or reject the decision aid’s estimates. This leads us to another important issue in human machine interaction: human operators’ trust in ADA. To support human judgment performance and/or calibration of their trust in ADA, the cognitive feedback was used to provide some information about the ADA. The cognitive information feedback of the ADA was provided to increase human operators’ understanding of the inner workings of the ADA.
Empirical results showed that participants were significantly affected by the validity of the decision aid’s estimates (Re_ADA). More importantly, those provided with the feedback information outperformed participants only with the decision aid in all Lens model parameters. Also, participants were able to utilize the feedback information to calibrate their trust in the decision aid in finer detail.
In conclusion, this study investigated the effect of automated decision aid’s various characteristics on operators’ judgment performance and trust in the system. A Lens model of human judgment performance with the decision aid was developed to identify the precise effect of ADA on judgment performance. Additionally, the cognitive information feedback was utilized as a way to increase judges’ understanding of the decision aid. Results showed that the cognitive information feedback of the decision aid was useful in supporting judges’ judgment performance and calibrating their trust in the decision aid. Further, the effect became greater as the “quality” of the decision aid became worse.
Can Decision Errors Be Predicted Before They Happen? Application of a New Measure of Skilled Performance
James Shanteau, along with Rickey Thomas, Brian Friel, John Raacke, Shawn Farris (all at Kansas State University, USA), David Weiss (California State University, Los Angeles, USA), and Julia Pounds (FAA, Oklahoma City, USA)
We conducted a two-month longitudinal study of the development of expertise in a simulated air- traffic-control microworld task. The simulation (CTEAM developed by the FAA) requires continuous, dynamic control of multiple aircraft through an air sector. To analyze results, we applied a new moving-window measure of skilled performance. The measure (C-CWS for Continuous-CWS) is an extension of CWS (Cochran-Weiss-Shanteau), developed originally for static analysis of single-case judgments. Our goal was to track the acquisition of expertise over an extended period.
Method Twelve operators were trained for eight weeks in a single-sector version of CTEAM. Six scenarios were generated by crossing three levels of aircraft density with two levels of airspace restrictions. Each scenario lasts 5 to 8 min. To allow for comparisons, the same 12 aircraft were embedded in all scenarios. The operator’s task was to route all aircraft from their origin to their destination. The dependent measures were number of control actions and time to move planes through the sector; similar results were obtained with the two measures.
Results C-CWS was sensitive to variations in performance associated with both aircraft density and restrictions in airspace. C-CWS was moderately correlated with two “objective measures" of performance - number of separation errors and number of barrier incursions. However, C-CWS outperformed these objective measures in two respects. First, C-CWS revealed performance improvements even after objective measures reached asymptote. Second, C-CWS showed increased sensitivity over sessions, whereas the objective measures became less sensitive as skill improved.
Additional analyses revealed that skill development was closely tied to internal consistency and discrimination (which make up C- CWS). Thus, the components of C-CWS were validated. However, skill development was only marginally related to consensus (which is not part of C-CWS). Most other approaches to assessing
Newsletter 2002 page 23 of 28