3.3 Quality appraisal (QA) Studies which met the eligibility criteria were assessed for methodological quality using a process developed by the authors based generally on previous work (Franche et al., 2004; Cote, 2002; Drummond and Jefferson, 1996; Jadad, 1998; Oxman and Guyatt, 1991; Smith et al., 2000; Zaza et al., 2000; Kuhn et al., 1999; van Tulder et al., 2003; van Tulder et al., 1997; Abenhaim et al., 2000; Thomas et al., 2003; Tompa et al., 2004; Sculpher et al., 2000).
Although many of the reviewed quality assessment tools incorporated about a dozen questions, the tool developed for this review emphasized parsimony with an aim to streamlining the consensus procedure. This meant that issues which were often covered in more than one question, are covered in only one (for example all issues related to selecting and maintaining the sample were covered in one question). As well, the tool was designed to focus on internal validity.
The quality appraisal (QA) form and guide to reviewers can be found in Appendices C.1 (Primary QA form) and E (Guide to quality assessment and data extraction). The methodological quality of each study was rated independently by two reviewers, who then met for consensus. If consensus could not be reached, one or more other reviewers were consulted as needed.
The QA questions were structured around the type of evidence(s) examined by a study. As noted in section 3.2, there were three types of evidence included in this systematic review: (1) implementation, (2) effectiveness (on intermediate outcomes, final OHS outcomes, and/or economic outcomes), and (3) facilitators and/or barriers (using either quantitative or qualitative methods). It is important to note that the review’s quality appraisal of the evidence refers to the quality of the research studies and reports, and not to the quality of the OHSMS interventions themselves. Furthermore, the quality was appraised from the point of view of the review’s questions about OHSMSs. A study may in some cases have been assigned a higher quality if the research question had been different.
3.3.1 Quality appraisal of quantitative evidence For all types of quantitative evidence, reviewers were asked to rate the studies on i) selection and maintenance of the sample and ii) potential confounders. The ratings were based on a three-point scale (meets criteria - Yes, Partially, No). There were three additional quality criteria that were considered separately for each type of quantitative evidence: measurement methods, appropriateness of statistical analyses, and other issues (including contamination of comparison group with exposure to the intervention). The quality ratings for these criteria were assessed separately for each type of evidence.
Effectiveness of Occupational Health & Safety Management Systems: A Systematic Review