Third, interviews tend to gather what people say, but not always what they do. Fourth, interviews can be subject to interviewer bias, for example if there is a large difference in age or socio-economic status between interviewee and interviewer.
3.2.3 Controlled Experiments and Case Studies
Controlled experiments can be very useful for understanding privacy behavior and trust determinants. However, it can be difficult to design experiments that are both plausible and elicit realistic responses from credible privacy threats or concerns. One precaution taken by Kindberg et al. was to avoid explicitly mentioning privacy and security to the participants at the outset of the study . The rationale was to avoid leading participants into specific privacy concerns, and rather probe the “natural” concerns of the users. We are not aware of any research proving that participants of studies on privacy should not be explicitly “led into” privacy or security. However, we believe that this is good precautionary practice, and that the topic of privacy can always be brought up after the experiment.
While conducting user studies, it is important to ensure that the tasks used are as realistic as possible, to give greater confidence of the validity of the results. In particular, participants need to be properly motivated to protect their personal information. Participants should also be put in settings that match expected usage. In their evaluation of PGP, Whitten and Tygar asked people to role-play, acting out in a situation that would require secure email . While it is clear that PGP had serious usability defects, it is also possible that participants could have been more motivated if they had a more personal stake in the matter, or could have performed better if they had been placed in an environment with multiple active users of PGP.
As another example, in Egelman et al’s evaluation of Privacy Finder, they discovered that individuals were willing to spend a little more money for privacy, by having participants purchase potentially embarrassing items [91, 121]. To make the purchase as realistic as possible, they had participants use their own credit cards (though participants also had the option of shipping the purchased items to the people running the study). This tradeoff in running realistic yet ethical user studies of privacy and security is an ongoing topic of research.
Some researchers have advocated using ethnographic methods, including contextual inquiry , to address the weaknesses of interviews. The basic idea is to observe actual users in situ, to understand their current practices and to experience their social and organizational context firsthand. Ethnographic methods have been successfully used to study privacy in the context of everyday life [130, 162, 201]. However, committing to this methodological approach requires the researcher to take an exploratory stance which may be incompatible with the tight process requirements of typical IT development.
end-user-privacy-in-human-computer-interaction-v57.docPage 27 of 85