Description of the Checklist
The following checklist is designed to provide a guide for evaluating conjoint studies that can be used by conjoint researchers in conducting their own work and by readers and users of conjoint studies. The ten-point checklist is presented in Table 1. In the remaining sections, we describe issues to be considered in evaluating each of these ten items and provide references for researchers to consult to better understand these issues.
i. Research question
The first item in the checklist is common to all scientific research and relates to the research question. Specifically “Was a well-defined research question stated and is conjoint analysis an appropriate method for answering it?”
Following generally accepted research practices, a conjoint analysis study must clearly state a well-defined research question that defines what the study will attempt to measure (8). For example, a conjoint analysis might be undertaken to quantify patients’ preferences for cost, risk of complications, and service location of a health care service. In addition to defining the research question, researchers should indicate the hypothesis(es) to be tested in the study. The hypothesis tests may be implicit in the research question itself. For example, in a conjoint analysis estimating the rate at which subjects are willing to trade off between two attributes, the testable null hypothesis is that the preference weight for one level of the attribute is not statistically significantly different from the preference weight for a different level of that attribute.
In other words, the hypothesis test is designed to infer whether a change in the level of one attribute (for example, a change in surgical wait time from one month to two months) is statistically significant. If the null hypothesis is rejected for a given attribute, then the change in the attribute level is statistically significant indicating that subjects are not willing to trade other attributes for changes in that attribute.
Second, the research question should define the study perspective including any relevant decision-making or policy context. The research question, “What are patients willing to pay for treatment to reduce the rate of relapse in multiple sclerosis?” includes both the items to be measured – the tradeoff between cost and reduction in relapse rates – the perspective and decision-context of the analysis – and the study perspective in the context of the patients’ decision about multiple sclerosis treatment.
In a health policy context, the research question might look something like, “What level of increase in waiting time for non-emergency surgery is the public willing to accept to reduce the rate of surgical errors?” Here, the items to be measured include waiting periods for non-emergency surgery and the rate of surgical errors. The perspective is that of the government or health service provider in which the preferences of the general public are estimated in the context of health policy for the delivery of non-emergency surgical health care services.
A conjoint analysis study should include reasons to justify that conjoint methods are appropriate to answer the research question. Conjoint analysis studies are well suited to the valuation of services or products that differ in their component attributes and where decisions-makers are willing to trade off among these component attributes. Because the examples of research questions presented above involve explicit tradeoffs between measureable attributes, conjoint methods are appropriate in each case.
ISPOR Conjoint Analysis in Health Task Force Report