Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi
Eric–Jan Wagenmakers, Ruud Wetzels, Denny Borsboom, & Han van der Maas
University of Amsterdam
Does psi exist? In a recent article, Dr. Bem conducted nine studies with over a thousand participants in an attempt to demonstrate that future events retroactively affect people’s responses. Here we discuss several limitations of Bem’s experiments on psi; in particular, we show that the data analy- sis was partly exploratory, and that one-sided p-values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem’s data using a default Bayesian t-test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem’s p-values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
Keywords: Confirmatory Experiments, Bayesian Hypothesis Test, ESP.
In a recent article for Journal of Personality and Social Psychology, Bem (in press) presented nine experiments that test for the presence of psi.1 Specifically, the experiments were designed to assess the hypothesis that future events affect people’s thinking and peo- ple’s behavior in the past (henceforth precognition). As indicated by Bem, precognition—if it exists—is an anomalous phenomenon, because it conflicts with what we know to be true about the word (e.g., weather forecasting agencies do not employ clairvoyants, casino’s
1The preprint that this article is based on was downloaded September 25th, 2010, from http://dbem. ws/FeelingFuture.pdf.
This research was supported by Vidi grants from the Dutch Organization for Scientific Research (NWO). Correspondence concerning this article may be addressed to Eric–Jan Wagenmakers, University of Amster- dam, Department of Psychology, Roetersstraat 15, 1018 WB Amsterdam, the Netherlands. Email address: email@example.com. We thank Rogier Kievit and Jan de Ruiter for constructive discussions. Note that this is a revised version of a previous draft that was accepted pending revision for Journal of Personality and Social Psychology.