Background: While randomized controlled trials (RCTs) inform the efficacy and effectiveness of treatments, we need to understand that even RCTs can be associated with sub-optimal execution. This is of special pertinence to eating disorders given the majority of treatment studies involving cognitive behaviour therapy are of poor quality with respect to managing risk of bias adequately. Methods: The current paper outlines the components of a good RCT for psychotherapy, and examines ways to improve the conduct, interpretation, and usefulness of RCTs. Results: This includes managing reporting bias, recognizing the limits of randomization, applicability, and ethical considerations. Conclusions: We highlight a number of strategies for future research, including issues related to utilizing a variety of designs to examine treatment outcomes, integrity, openness and reproducibility.