performance. See Exhibit 3.1 in Section 3.3 for the list of SQG indicators. See Exhibit 5.1 in Section 5.4 for the list of Auto Body indicators. They include both regulatory requirements and voluntary beyond compliance practices.
What the Project Will Measure
State SQG Mean Facility Score on 1) All Indicators, 2) Regulatory Indicators and 3) Beyond Compliance Indicators: A facility score is the proportion of the performance indicators to which a particular facility is subject, which the facility was observed to achieve (i.e., was either in compliance with the regulatory requirement or was implementing the beyond compliance practice). This is expressed as a number from 1 to 10: A score of “1” means that the facility was achieving 10% of the applicable indicators, a score of “5” signifies that the facility achieved 50% of the applicable indicators, and a score of 10 means that the facility achieved 100% of the applicable indicators. The state mean facility score is the average score achieved by all facilities in the sample. Scores can be calculated for all or a subset of indicators.
State SQG Achievement Rate By Indicator: The percentage of the facilities in the state that were achieving each performance indicator. This was calculated for each indicator for each state.
State SQG Facility Score Distribution: The percentage of facilities in the state that achieved each facility score (e.g., the percentage of facilities that achieved facility score of 1, the percentage of facilities that scored 2, the percentage of facilities that scored 3, etc.).
Interstate Comparisons: Statistically significant differences in state SQG performance on achievement rates and mean facility scores at varying confidence levels.
OBSERVATIONS AND LESSONS LEARNED FROM PHASE 2
The degree to which there were differences in state definitions of the selected groups was surprising. Summer work helped focus discussion, identified, and resolved definitional differences.
It was helpful to not pick indicators before statistical training. This allowed for context on how data will be used. This also helped in the design of the indicators.
A state participant suggested it would be helpful to have a more in depth training module available to help explain important measurement concepts to project participants as well as other decision makers within their state.
It is beneficial to have a common approach to universe identification when doing multi-state comparisons.
States that initially planned to use existing data on groups decided it was more efficient and effective to get new data for the project than try to meet the data standards needed to use existing data.
The States Common Measures Project Final Report