House, ER Blowback: Consequences of Evaluation for Evaluation. American Journal of Evaluation. 2008 December 29 (4), 416-426.
For some, the Randomized Controlled Trial (RCT) has the mystique of separating the researcher from the method and therefore guaranteeing research objectivity. However, in a 2008 article in the American Journal of Evaluation, Ernest House disputes this myth. He describes many examples of how bias has been introduced into the RCTs for new drugs, which are funded primarily by drug companies (over 70%, according to the article). He talks about suppression of negative results as one problem with drug-company sponsored research; but he also describes ways that drug trials can actually be manipulated to influence results favoring the drug company’s products. For example, the drugs under investigation may be compared to a lower dosage of the competitor’s drug in the control group or the competitor’s drug may be administered in a less effective manner. Studies also may be conducted on younger subjects who generally tend to show fewer side effects or conducted for short periods of time even though the drug was developed for long-term use.
House also notes that, as an evaluation tool, RCTs are very limited in providing all information needed to judge a new drug’s value. Usually the drug group is compared to a control group that gets no treatment instead of the typical dosage of the closest competitor drug. So, while consumers may know the tested drug is superior to no treatment, we know nothing about the cost-benefit or “clinical effectiveness” of a new (often more expensive) drug.
This article provides some important take-home messages for evaluators as well as for consumers of drugs. First of all, no method guarantees objectivity: even the highly acclaimed RCT can be manipulated (deliberately or unconsciously) to influence desired results. Second, evaluation – finding the value of products, services, or programs – usually involves multiple issues that must be investigated through mixed methods. Finally, evaluators need to be aware that they are not completely objective and methods cannot protect them from their own subjectivity. We need to be transparent about our data collection and analysis process and be open to feedback from peers and stakeholders.