It is difficult to generalize about common limitations with respect to the quality of the evidence on implementation, intermediate outcomes, economic outcomes, and quantitative facilitators/barriers because of the small number of studies. However, if all effectiveness outcomes are considered together, an absence of any control for confounders and inappropriate statistical tests again appear to be the most frequent quality problems.
As noted in section 4.1, the search retrieved three articles containing facilitator/barrier evidence that had been based on qualitative data, i.e., data from interviews, focus groups, document review, or observations. In two articles, the interviews were part of an audit procedure. Although the means of selecting interviewees was sometimes reported, these audits were not described as containing an explicit inquiry into facilitating features or barriers to implementation, nor were the methods of analysis of the interview data described. In the third article, which involved case studies of two firms, data collection included taped interviews and observation, and the focus was not on OHSMSs but on the directness of employee involvement in management systems. In every case, an absence of any description of the studies’ methodologies and/or the anecdotal nature of the results described led to a rating of the research methods as “Low” or “Very low”.
The quality of studies also varied according to whether the OHSMS was voluntary or mandatory. More studies of a mandatory OHSMS met the QA criteria than did those of voluntary OHSMSs (see Figure 4.1.2). Specifically, the studies of mandatory OHSMSs were better at considering confounders, controlling for them, and discounting them than those of voluntary OHSMSs (see Table 4.2.1).
Effectiveness of Occupational Health & Safety Management Systems: A Systematic Review