The external validity of controlled clinical trials of psychotherapy for depression and anxiety: A naturalistic study

Psychotherapy researchers have increasingly called for clinical practice and training to focus on empirically supported therapies tested in randomized controlled clinical trials (RCTs). In this paper, we report data from a naturalistic study of successful treatments in clinical practice that bear on the external validity of ESTs for three disorders. Participants were 242 experienced doctoral‐level clinicians who reported on their last successfully treated patient seeking treatment for clinically significant depression, panic, or anxiety without panic. Successful treatments typically take substantially longer than the 8–16 sessions characteristic of efficacy trials for these disorders, even for the briefest treatments (cognitive‐behavioural). Of particular relevance for generalizability from RCTs, most patients in clinical practice present with multiple problems other than a single Axis I disorder, which clinicians of all theoretical orientations recognize and treat, and these co‐occurring conditions have a substantial impact on treatment length in everyday practice. The data suggest the importance of effectiveness research in bridging research and practice. They also point to the utility of distinguishing two complementary ways in which effectiveness research can be understood and implemented: by starting with efficacy trials and then testing treatments with promising results in the laboratory using broader community samples; or by starting with everyday clinical practice, examining patterns of covariation between specific interventions and outcomes at clinically meaningful follow‐up intervals with diverse and ecologically valid samples and using these data to generate prototypes of treatments that can be used to guide the next generation of experimental studies.     

Quality of reporting internal and external validity data from randomized controlled trials evaluating stents for percutaneous coronary intervention

Background  

Stents are commonly used to treat patients with coronary artery disease. However, the quality of reporting internal and external validity data in published reports of randomised controlled trials (RCTs) of stents has never been assessed.     

Randomised controlled trials in primary care: scope and application.

There is now widespread acknowledgement of the absence of a sound evidence base underpinning many of the decisions made in primary care. Randomised controlled trials represent the methodology of choicefor determining efficacy and effectiveness of interventions, yet researchers working in primary care have been reluctant to use intervention studies, favouring observational study designs. Unfamiliarity with the different trial designs now available, and the relative advantages and disadvantages conferred by each, may be one factor contributing to this paradox. In this paper, we consider the principal trial designs available to primary care researchers, discussing the contexts in which a particular design may prove most useful. This information will, we hope, also prove useful to primary care clinicians attempting to interpret trial findings.     

Pragmatic trials in primary care. Methodological challenges and solutions demonstrated by the DIAMOND-study

Background  

Pragmatic randomised controlled trials are often used in primary care to evaluate the effect of a treatment strategy. In these trials it is difficult to achieve both high internal validity and high generalisability. This article will discuss several methodological challenges in designing and conducting a pragmatic primary care based randomised controlled trial, based on our experiences in the DIAMOND-study and will discuss the rationale behind the choices we made. From the successes as well as the problems we experienced the quality of future pragmatic trials may benefit.     

Conducting randomised controlled trials in primary care: lessons from an obesity management trial

BACKGROUND: A UK trial ending in 2002 reported that a training intervention to improve the management of obesity in primary care had no impact. Process analysis showed that the intervention was taken up by very few of the practitioners in the participating practices. AIM: The aim of the current study was to explore both the reasons behind low levels of implementation and the context in which the trial was delivered. DESIGN: In-depth qualitative interviews. SETTING: General practices in the North East of England. METHOD: Interviews with 13 practitioners (GPs and practice nurses) and 10 patients, representing seven of the eight intervention practices in the largest centre of the original trial. RESULTS: While patients were clear that they had participated in a trial few of those interviewed had any recollection of the intervention. Most staff were positive about the training, resources to use with patients and the weight management model, but not all training needs were met. High initial expectations of the trial gave way to disillusionment, exacerbated by significant misunderstanding by some practice staff of their role in implementation. CONCLUSIONS: Frustration among practitioners managing obesity in primary care combined with unrealistic expectations of and misunderstanding about an intervention designed to improve care in the field, appeared to have resulted in disillusionment with and consequent disengagement from the trial.     

A randomized controlled trial of IPT versus CBT in primary care: with some cautionary notes about handling missing values in clinical trials

A randomized controlled trial is reported in which three treatments were compared for the management of depression in Primary Care. The treatments were Treatment As Usual (TAU) carried out by the General Practitioners, Cognitive-Behaviour Therapy (CBT) or Interpersonal Psychotherapy (IPT). Measurements of depressive symptomatology were taken at Baseline (Time1), at end of treatment (Time2), and at 5-month follow-up (Time3). An initial analysis of the longitudinal data revealed that there were a significant number of missing values, especially in the Time3 follow-up for the TAU group. That is, the missing data were not missing at random within the dataset, which is one of the considerations for usual procedures for replacement of missing values (RMV). The paper presents, therefore, the outcome of different approaches to RMV and their consequences for conclusions about the relative efficacy of the treatment conditions. The results showed that clients in all conditions improved significantly, with at least some analyses showing superiority of IPT and CBT at end of treatment Time 2. However, by the follow-up clients in all conditions performed equally well.     

Vasectomy techniques for male sterilization: systematic Cochrane review of randomized controlled trials and controlled clinical trials

BACKGROUND: The review aimed to compare the effectiveness, safety and acceptability of vasectomy techniques for male sterilization. METHODS: We searched five computerized databases and reference lists of relevant articles and book chapters for randomized controlled trials (RCTs) and controlled clinical trials (CCTs) comparing vasectomy techniques. Two reviewers independently extracted data from eligible articles. RESULTS: Two poor-quality trials compared vas occlusion with clips versus a conventional technique, and four poor-quality trials examined vas irrigation with water versus no irrigation or irrigation with euflavine. No significant differences regarding the primary outcome of time to azoospermia were found. However, one trial reported fewer median number of ejaculations to azoospermia with euflavine rather than water irrigation. An interim report of a high-quality trial comparing vasectomy with and without fascial interposition found more azoospermia with fascial interposition but also more surgical difficulties. CONCLUSIONS: No conclusions can be reached regarding the effectiveness, safety and acceptability of vas occlusion techniques or vas irrigation since only low-quality, underpowered studies were available. Fascial interposition had improved vasectomy success but also increased surgical difficulty. High-quality, adequately reported RCTs are required. More work is also needed in the standardization of follow-up protocols, evaluation of vasectomy success and failure, recanalization and analytical methods.     

Issues in the design and analysis of controlled clinical trials

Several issues that pertain to the analysis of data from repeated measurements designs are addressed. The necessity for an assumption concerning uniformity of correlations (symmetry) can be avoided by calculating linear and/or nonlinear trend scores. In the usual repeated measurements ANOVA the baseline covariate does not affect within-subject sums of squares, such as that for the groups × times interaction. However, the separately calculated trend scores have the advantage of accounting for most of that interaction and permitting covariance-correction for baseline differences. Chance baseline differences in a repeated measurements design otherwise can contribute materially to probabilities of erroneous conclusions. A conservative criterion for use of one-sided tests of significance is proposed.     

Ruling out coronary heart disease in primary care: external validation of a clinical prediction rule

Background

The Marburg Heart Score (MHS) aims to assist GPs in safely ruling out coronary heart disease (CHD) in patients presenting with chest pain, and to guide management decisions.

Aim

To investigate the diagnostic accuracy of the MHS in an independent sample and to evaluate the generalisability to new patients.

Design and setting

Cross-sectional diagnostic study with delayed-type reference standard in general practice in Hesse, Germany.

Method

Fifty-six German GPs recruited 844 males and females aged ≥35 years, presenting between July 2009 and February 2010 with chest pain. Baseline data included the items of the MHS. Data on the subsequent course of chest pain, investigations, hospitalisations, and medication were collected over 6 months and were reviewed by an independent expert panel. CHD was the reference condition. Measures of diagnostic accuracy included the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, likelihood ratios, and predictive values.

Results

The AUC was 0.84 (95% confidence interval [CI] = 0.80 to 0.88). For a cut-off value of 3, the MHS showed a sensitivity of 89.1% (95% CI = 81.1% to 94.0%), a specificity of 63.5% (95% CI = 60.0% to 66.9%), a positive predictive value of 23.3% (95% CI = 19.2% to 28.0%), and a negative predictive value of 97.9% (95% CI = 96.2% to 98.9%).

Conclusion

Considering the diagnostic accuracy of the MHS, its generalisability, and ease of application, its use in clinical practice is recommended.     

Counselling in primary care: A randomised controlled trial

A randomised controlled trial of the effectiveness of counselling (In-House Counselled (IHC) group) compared to routine advice from General Practitioners (Routine Treatment (RT) group) was conducted with three practices and a total of 188 patients in East Sussex. Changes in interpersonal problems using the 32 item Inventory of Interpersonal Problems, symptoms of psychological distress using the Symptom Index and self esteem using Repertory Grids were compared between groups at four months and after a further four month follow-up. A questionnaire monitoring patient service satisfaction was given to those who had received In-House counselling. The number of counselling sessions, early withdrawals and refusers was also monitored. In order to facilitate interprofessional communication, the three counsellors and a GP representative from each practice met monthly for an Action Learning group, led by an external facilitator to provide a forum to discuss working practices. The group met six times for two and a half hours. An audit of the participants' medical notes was carried out at the end of the study to establish any changes in subsequent use of medical services and prescribing patterns. The results show that patients within both groups improved considerably, in line with similar studies. The IHC group was significantly less likely to be referred out to mental health services. However, there was no statistical difference between the groups on any of the measures or in changes in subsequent service use or prescribing patterns. This may have been a result of Action Learning Group producing more psychologically minded GPs. Patients in the IHC group were overwhelmingly in favour of counselling and stated that it had helped them with a variety of psychological problems.     

Bias in treatment assignment in controlled clinical trials

Controlled clinical trials of the treatment of acute myocardial infarction offer a unique opportunity for the study of the potential influence on outcome of bias in treatment assignment. A group of 145 papers was divided into those in which the randomization process was blinded (57 papers), those in which it may have been unblinded (45 papers), and those in which the controls were selected by a nonrandom process (43 papers). At least one prognostic variable was maldistributed (P less than 0.05) in 14.0 per cent of the blinded-randomization studies, in 26.7 per cent of the unblinded-randomization studies, and in 58.1 per cent of the nonrandomized studies. Differences in case-fatality rates between treatment and control groups (P less than 0.05) were found in 8.8 per cent of the blinded-randomization studies, 24.4 per cent of the unblinded-randomization studies, and 58.1 per cent of the nonrandomized studies. These data emphasize the importance of keeping those who recruit patients for clinical trials from suspecting which treatment will be assigned to the patient under consideration.     

Mirroring everyday clinical practice in clinical trial design: a new concept to improve the external validity of randomized double-blind placebo-controlled trials in the pharmacological treatment of major depression

ABSTRACT: BACKGROUND: Randomized, double-blind, placebo-controlled trials constitute the gold standard in clinical research when testing the efficacy of new psychopharmacological interventions in the treatment of major depression. However, the blinded use of placebo has been found to influence clinical trial outcome and may bias patient selection. DISCUSSION: To improve clinical trial design in major depression so as to reflect clinical practice more closely we propose to present patients with a balanced view of the benefits of study participation irrespective of the assignment to placebo or active treatment. In addition every participant should be given the option to finally receive the active medication. A research agenda is outlined to evaluate the impact of the proposed changes on the efficacy of the drug to be evaluated and on the demographic and clinical characteristics of the enrolment fraction with regard to its representativeness of the eligible population. SUMMARY: We propose a list of measures to be taken to improve the external validity of double-blind, placebo-controlled trials in major depression. The recommended changes to clinical trial design may also be relevant for other psychiatric as well as medical disorders in which expectations regarding treatment outcome may affect the outcome itself.     

The internal and external validity of the Major Depression Inventory in measuring severity of depressive states

BACKGROUND: We have developed the Major Depression Inventory (MDI), consisting of 10 items, covering the DSM-IV as well as the ICD-10 symptoms of depressive illness. We aimed to evaluate this as a scale measuring severity of depressive states with reference to both internal and external validity. METHOD: Patients representing the score range from no depression to marked depression on the Hamilton Depression Scale (HAM-D) completed the MDI. Both classical and modern psychometric methods were applied for the evaluation of validity, including the Rasch analysis. RESULTS: In total, 91 patients were included. The results showed that the MDI had an adequate internal validity in being a unidimensional scale (the total score an appropriate or sufficient statistic). The external validity of the MDI was also confirmed as the total score of the MDI correlated significantly with the HAM-D (Pearson's coefficient 0.86, P < or = 0.01, Spearman 0.80, P < or = 0.01). CONCLUSION: When used in a sample of patients with different states of depression the MDI has an adequate internal and external validity.     

Checklist for the qualitative evaluation of clinical studies with particular focus on external validity and model validity

Background  

It is often stated that external validity is not sufficiently considered in the assessment of clinical studies. Although tools for its evaluation have been established, there is a lack of awareness of their significance and application. In this article, a comprehensive checklist is presented addressing these relevant criteria.     

Collaborative care for depression in UK primary care: a randomized controlled trial

BACKGROUND: Collaborative care is an effective intervention for depression which includes both organizational and patient-level intervention components. The effect in the UK is unknown, as is whether cluster- or patient-randomization would be the most appropriate design for a Phase III clinical trial. METHOD: We undertook a Phase II patient-level randomized controlled trial in primary care, nested within a cluster-randomized trial. Depressed participants were randomized to 'collaborative care' - case manager-coordinated medication support and brief psychological treatment, enhanced specialist and GP communication - or a usual care control. The primary outcome was symptoms of depression (PHQ-9). RESULTS: We recruited 114 participants, 41 to the intervention group, 38 to the patient randomized control group and 35 to the cluster-randomized control group. For the intervention compared to the cluster control the PHQ-9 effect size was 0.63 (95% CI 0.18-1.07). There was evidence of substantial contamination between intervention and patient-randomized control participants with less difference between the intervention group and patient-randomized control group (-2.99, 95% CI -7.56 to 1.58, p=0.186) than between the intervention and cluster-randomized control group (-4.64, 95% CI -7.93 to -1.35, p=0.008). The intra-class correlation coefficient for our primary outcome was 0.06 (95% CI 0.00-0.32). CONCLUSIONS: Collaborative care is a potentially powerful organizational intervention for improving depression treatment in UK primary care, the effect of which is probably partly mediated through the organizational aspects of the intervention. A large Phase III cluster-randomized trial is required to provide the most methodologically accurate test of these initial encouraging findings.