NO EVIDENCE FOR PSI
result is firm belief that psi is indeed possible. On the one hand, 2000 participants seems excessive; on the other hand, this is but a small subset of participants that have been tested in the field of parapsychology during the last decade. Of course, this presupposes that the experiment under consideration was 100% confirmatory, and that it has been conducted with the utmost care.
Guidelines for Confirmatory Research
As discussed earlier, exploratory research is useful but insufficiently compelling to change the mind of a skeptic. In order to provide hard evidence for or against an empirical proposition, one has to resort to strictly confirmatory studies. The degree to which the scientific community will accept semi-confirmatory studies as evidence depends partly on the plausibility of the claim under scrutiny: again, extraordinary claims require extraor- dinary evidence. The basic characteristic of confirmatory studies is that all choices that could influence the result have been made before the data are observed. We suggest that confirmatory research in psychology observes the following guidelines:
1. Fishing expeditions should be prevented by selecting participants and items before the confirmatory study takes place. Of course, previous tests, experiments, and ques- tionnaires may be used to identify those participants and items who show the largest effects—this method increases power in case the phenomenon of interest really does exist; however, no further selection or subset testing should take place once the con- firmatory experiment has started.
2. Data should only be transformed if this has been decided beforehand. In confirmatory studies, one does not “torture the data until they confess”. It also means that—upon failure—confirmatory experiments are not demoted to exploratory pilot experiments,
and that—upon success matory experiments.
exploratory pilot experiments are not promoted to confir-
In simple examples, such as when the dependent variable is success rate or mean re- sponse time, an appropriate analysis should be decided upon before the data have been collected.
It is prudent to report more than a single statistical analysis. If the conclusions from p-values conflict with those of, say, Bayes factors, then this should be clearly stated. Compelling results yield similar conclusions, irrespective of the statistical paradigm that is used to analyze the data.
In our opinion, the above guidelines are sufficient for most research topics. However, the researcher who wants to convince a skeptical community of academics that psi exists may want to go much further. In the context of psi, Price (1955, p. 365) argued that “(...) what is needed is something that can be demonstrated to the most hostile, pig-headed, and skeptical of critics.” This is also consistent with Hume’s maxim that “(...) no testimony is sufficient to establish a miracle, unless the testimony be of such a kind, that its falsehood would be more miraculous, than the fact, which it endeavours to establish (...)” (Hume, 1748, Chapter 10). What this means is that in order to overcome the skeptical bias against