The Bonferonni and Šidák Corrections
same set of data. For example, brain imaging researchers will rou- tinely run millions of tests to analyze an experiment. Running so many tests increases the risk of false alarms. To illustrate, imagine the following “pseudo-experiment":
I toss 20 coins, and I try to force the coins to fall on the heads. I know that, from the “binomial test," the null hypothesis is rejected at the α = .05 level if the number of heads is greater than 14. I repeat this experiment 10 times.
Suppose that one trial gives the “significant" result of 16 heads versus 4 tails. Did I influence the coins on that occasion? Of course not, because the larger the number of experiments, the greater the probability of detecting a low-probability event (like 16 versus 4). In fact, waiting long enough is a sure way of detecting rare events!
2.1 Probability in the family
A family of tests is the technical term for a series of tests performed on a set of data. In this section we show how to compute the prob- ability of rejecting the null hypothesis at least once in a family of tests when the null hypothesis is true.
For convenience, suppose that we set the significance level at α=.05. For each test (i.e., one trial in the example of the coins) the probability of making a Type I error is equal to α = .05. The events “making a Type I error" and “not making a Type I error" are com- plementary events (they cannot occur simultaneously). Therefore the probability of not making a Type I error on one trial is equal to
1 − α = 1 − .05 = .95 .
Recall that when two events are independent, the probability of observing these two events together is the product of their proba- bilities. Thus, if the tests are independent, the probability of not making a Type I error on the first and the second tests is
.95 × .95 = (1 − .05)2 = (1 − α)2 .