A Validational Analysis of Methods of Utility Estimation
Radhika Nath Rockefeller College, State University of New York at Albany E-mail: email@example.com
For my Ph.D dissertation I examined the validity of three different utility elicitation methods used in medical decision making. These instruments (Visual analog scale, Time tradeoff and Standard gamble)
are used to elicit inform physician
patient preferences decision making.
which in turn Where most
previous research looked health state, in this study
at preferences for a single the multiple methods were
used to different
estimate preferences for each of three health states (blindness, stroke and AIDS).
This facilitated a evaluation of the
thorough MultiTrait MultiMethod
questions about before, I believe
instruments have is the first study to
been raised undertake a
instruments I chose to study did not achieve
specific validity. through
represent and how they other words, the results
relate to obtained
from using these methods caution if at all, in medical
should be used decision making
Further, I also examined if specific
individual characteristics (such as a facility with numbers) had any affect on responses to specific methods as hypothesized by other scholars. Using a structural equation model I found that, of the various characteristics studied, only those related to risk (specifically physical and professional risk) had any affect on responses to the time tradeoff and the standard gamble instruments.
Predicting search strategies in simple heuristics and using feedback in MCPL
Ben Newell and David Shanks University College London E-mail: firstname.lastname@example.org
Continuing the program of research reported in last year’s newsletter David Shanks and I have been furthering our empirical investigations of fast-and- frugal heuristics, with a particular focus on search strategies. In work with Tim Rakow of the University of Essex, we have been examining how decision- makers learn about cue properties in environments with objective criteria. In such environments, cues can be evaluated on the basis of three properties: validity (the probability that a cue identifies the correct choice if cue values differ between alternatives); discrimination rate (the proportion of
occasions on which a cue has differing values); and success (the expected proportion of correct choices when only that cue can be used).
It turns out that validity and discrimination rate are often negatively correlated. For example, in the German cities task (identify the city from a pair that has the largest population) the ‘capital city cue’ (knowing if either city in the pair is the capital) has a very high validity, because normally a capital has a very large population, but a very low discrimination rate because there is only one capital. This inverse relationship is potentially problematic for heuristics such as “Take-The-Best” (Gigerenzer & Goldstein, 1996), which use a validity-ordered search rule. This because the first few cues looked up by the heuristic, though highly valid will often not discriminate between alternatives and therefore not be able to be used as a basis for a decision. In environments where information is costly such a non-frugal search rule appears maladaptive. Using a stock market prediction task (see Newell & Shanks, 2003) we tested our hypothesis that people would learn a sensitivity to both the validity and the discrimination rate of cues and that their search patterns would follow a pattern that was a function of these two properties - namely success. In two experiments we found support for this hypothesis. The patterns of search through the cues were more closely associated with the search ordering predicted by success than by discrimination rate or validity. Furthermore, when asked to rate each cue for ‘usefulness’ at the end of the experiment these ratings again were more closely associated with the success ordering than the other two predicted orderings. These results provide strong support for the claim that search through cues in environments with objective criteria is determined by the success rate of cues, and not by validity as the search rule of the “Take-the-Best” heuristic states.
In a new project, unrelated to the fast-and- frugal heuristics, we have been examining the role of feedback in discrete cue MCPL. Using a four cue environment we compared performance of a group given task information feedback with a group given only outcome feedback. The findings mirrored those seen in continuous cue MCPL with the group given task information feedback performing better than the group given only outcome feedback. However, in a follow-up experiment we changed the task structure such that the cues and criterion were presented on the same dimension. Now the difference in performance between the feedback groups largely disappeared due to improvements in the outcome feedback only group. The results are consistent with Harries and Harvey’s (2000) contention that having cues and criterion on the same dimension provides participants directly with information
Newsletter 2002 page 20 of 28