There were also issues in these studies arising from their limited handling of potential confounding factors and their use of self administered questionnaires and self reports from telephone interviews. However, the most crucial limitation for these studies relates to the cross-sectional design which two of them had, making temporal causality difficult to establish.
The two Canadian studies had more sophisticated study designs and used pooled times series data analyses. However, these studies were limited for reasons primarily related to their units of analyses and an inability to account for confounding factors. Dufour et al. (1998) incorporated industry-level data from a variety of publicly available sources and there was concern that the findings may have been affected by factors acting at lower levels. Thus, their finding of a significant positive impact of prevention programs on productivity growth based on aggregate data at the industry level may be subject to ecological fallacy.24 Similarly, the findings of Lewchuk et al. (1996) that Bill 70 led to a significant reduction in work-related injury rates in manufacturing must be interpreted in light of possible co-intervention or confounding effects.
Although all five studies reported positive findings, the overall conclusions are more equivocal. The Norwegian investigations showed effects across a variety of outcomes and over time, indicating that the IC regulation was having some impact. However, the limitations related to study design and sampling leaves open the possibility that the studies had over-selected organizations that were complying with the regulation and managers and workers felt they should report positive effects. The studies of the Ontario and Quebec regulations are limited as the regulations themselves had only a limited focus on OHSMSs and likely used other regulatory actions to impact OHS. The inability of these studies to measure other regulatory activities, and other confounders impacting OHS performance made it difficult to feel confident that the findings reflected only the impact of OHSMSs.
4.5 Evidence synthesis Sections 4.3 and 4.4 of the report described the individual studies of voluntary and mandatory OHSMSs, and summarized the results for each grouping. This section gives a higher level synthesis of each group of studies and discusses what this means in terms of the level of evidence for OHSMS effectiveness.
One of the original aims of this study was to examine the relative effectiveness of voluntary and mandatory OHSMS interventions. The ideal research design for studies addressing that question would involve direct comparisons of mandatory and voluntary OHSMS interventions. No studies incorporated such a design.
24 A situation that can occur when a researcher makes an inference about outcomes at a lower level based on aggregate data obtained a higher level of observation.
Institute for Work & Health