4.3.4 The Effect of Confidence Levels, Number of Inspections and Confidence Intervals on the Usefulness of the Data for Decision Making
Measurement projects such as the States Common Measures Project are not undertaken just for the sake of measuring something. These measurements are needed to determine if the facility performance in a state is “good enough” to meet the state’s policy objectives for the regulatory program and to identify any oversight practices that appear to be associated with higher performance levels. To the extent that the findings are reliable, the states are able to use the findings to make better decisions about efficient and effective programs.
When making choices on the basis of data, decision makers generally consider two factors. One is the confidence level in the results -- the likelihood that the observed data accurately reflects the conditions in the world. The other is the precision of the results -- the confidence interval, or the range above and below the observed value within which the group’s performance actually falls. In general, the larger the number of inspections, the greater the precision of the results and the confidence that the results accurately reflect conditions in the world.
In many fields, a 90% confidence level (which indicates that there is a 10% chance that the observed results do not reflect the actual performance of the group) is considered the lowest “acceptable” level for drawing conclusions about the behavior of a group, and whether that behavior is “statistically different” than that of another group. However, achieving a precise measurement at a high level of confidence may require more inspections than a state can “afford” and perhaps may need. A state may be comfortable being only 85% certain that their results are within the specified confidence interval, if the consequences of being wrong are not serious. A state may be able to base a decision on a very wide confidence interval if “good enough” or “not good enough” performance is within that wide range, or the state only needs to be able to identify very large differences in performance levels. Furthermore, policy decisions sometimes need to be based on whatever amount of information can be obtained with the level of resources available to measure.
To explore the issue of what is “good enough” data, the States Common Measures Project analyzed the data at three different confidence levels. The results (shown in the charts below) illustrate how confidence intervals are affected by confidence levels, the numbers of inspections and the observed performance. This provides some insight into the level of confidence and precision and therefore the number of inspections that a state decision maker may “need” to make choices about program design.
Note: The confidence interval is calculated from a formula based on three factors: the confidence level, the observed performance rate and the sample size. The relationship among these three factors is the same regardless of the indicator being measured or the universe size. Therefore, the charts below use different indicators from different states to illustrate this relationship.
The States Common Measures Project Final Report