Confidence Level: % certainty that a difference of the size listed below is not due to chance

90%

Power: % certainty that a smaller difference than the given difference IS due to chance (in other words that you 80% are not missing a true difference)

State A

50%

State B

50%

estimated compliance rate of the universe

#

of facilities needed to determine that a finding that State

## A's performance is

than

## State B's performance is not due to chance

56

90%

50%

90%

90%

70%

50%

70%

50%

70%

70%

70%

50%

34

82

49

46

65

107

90%

90%

90%

95%

95%

95%

80%

90%

90%

80%

80%

90%

#

of facilities needed to determine that a finding that State

A's performance is

than

State B's performance is not due to chance

#

of facilities needed to determine that a finding that

State A's performance is than State B's performance is not due to chance

100

225

60

146

88

135

328

197

82

115

190

185

260

428

The highlighted columns show the level of certainty (confidence level) and precision (confidence interval) that are possible given the number of inspections the project states said they could realistically afford to complete (between 34 – 67). The table shows that:

î‚¾

If the observed compliance rates are in the 50% range, 56 inspections would be needed to detect a 20% point difference, at a 90% confidence level.

î‚¾

Unless states inspected more than 80 facilities each, the highest difference that states could realistically hope to detect is 15%, and that would only be achieved if both states had relatively high observed compliance rates (as shown in column 2 with 90% and 70% estimated compliance rates).

# Decision on sample size for comparing performance across states

More inspections were required to compare performance levels across states than to benchmark performance within a state. The project states decided that by inspecting at least 56 SQGs per state, the results would provide sufficient precision at a 90% confidence level. Using a higher confidence level would allow the project to detect smaller differences in performance but would require more inspections and resources. If, after conducting inspections, observed performance levels were greater than 50%, it would be possible to detect whether smaller differences in state performance levels were statistically significant.

## The States Common Measures Project Final Report

51

95%

90%

70% 70%

95

137

309