Data collection methodologies.
The characteristics of the group being measured.
The indicators used to measure performance.
This background was provided prior to selecting the groups and indicators in the belief that it would lead to agreement on groups, indicators and data collection methodologies that were conducive to reliable measurement.
Phase 2) Making Decisions on Groups and Indicators
In Phase 2, project states applied the “lessons learned” about measurement to select two groups to measure and to define the set of indicators that all states would use to evaluate the performance of each group. While the grant commitment was to complete measurement on at least one group, the states decided, through a combination of “summer work” assignments for each state, in person meetings, and telephone conference calls, to measure two groups. Colorado, Connecticut, Maine, Massachusetts, New Hampshire, New York, Rhode Island, and Vermont all agreed to measure the performance of Small Quantity Generators of Hazardous Waste (SQG). In addition, the project states decided that to the extent that there was sufficient time during the grant period and a state had the capacity to take on an additional measurement project, project states could choose to work on the auto body sector as well. The project states successfully agreed on a set of common indicators for both groups. The indicators included both “regulatory requirements” and desirable best practices that went “beyond compliance.”
Phase 3) Data Collection, Field Observer Training and Statistical Methods
Data collection was the focus of Phase 3. In this phase, participating states developed and implemented common approaches for identifying the universe of facilities in each group, and for selecting a random sample of facilities to inspect. The project states also agreed on the ways the results would be analyzed and presented. An automated data analysis and presentation tool developed for the Massachusetts and Colorado ERP programs, the “ERP Performance Analyzer,” was augmented for use in this project.
A “sample-size calculator” tool developed for the Massachusetts ERP program was used to analyze the effect of sample size on the precision and reliability of the results. The project states decided that 56 was the optimal number of inspections for each state to conduct in light of available resources. In order to ensure data comparability across states, a common inspection checklist was developed for both sectors. All of the individuals that conducted field observations were trained in the use of the checklist and data quality assurance procedures. All eight states that participated in measuring SQG performance completed their inspections by the fall of 2008. However, due to budget, time constraints and universe composition, the number of inspections successfully completed per state varied from a low of 22 to a high of 57. Upon completion of the SQG field observations, the data collected on the inspection checklists was entered into the ERP Performance Analyzer. In addition to the SQG inspections, New York and Washington State completed a portion of their planned auto body inspections. However
The States Common Measures Project Final Report