Estimands and Their Role in Clinical Trials

The National Research Council (NRC) highlighted the need to more clearly distinguish between the target of estimation (“estimand”) and the method of estimation (“estimator”) in clinical trials. While the NRC report on “The Prevention and Treatment of Missing Data in Clinical Trials” focuses on issues arising due to missing data, a framework to coherently align trial objectives and corresponding estimands is valuable in a broader sense. The International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH) has reinforced this by tasking a working group to develop an addendum to the ICH-E9 guideline “Statistical Principles for Clinical Trials.” In this article, we motivate the need for change, propose a structured framework to bridge trial objectives with proper inference tools, and discuss how it may impact the role of statisticians involved in clinical trial design and analysis.     

Aligning Treatment Policy Estimands and Estimators—A Simulation Study in Alzheimer’s Disease

Abstract

Draft ICH E9(R1) Addendum on “Estimands and Sensitivity Analysis in Clinical Trials” provides different strategies for addressing intercurrent events in defining an estimand and describing the treatment effect that is targeted. The set of considered intercurrent events will depend on the specific therapeutic setting and trial objectives. This article considers a case study of a long-term prevention trial investigating the treatment effect of a new drug in asymptomatic subjects who are at risk for developing Alzheimer’s dementia to illustrate the definition of different estimands, which correspond to different scientific questions of interest. The potential intercurrent events are identified. This article shows how the selection of various strategies for intercurrent events translates into different estimators. A simulation investigation is included, which explores the properties of several estimators aligned with estimands that apply a treatment policy strategy for the intercurrent event of treatment discontinuation. Different scenarios are considered for the on-treatment versus the off-treatment mean efficacy trajectory, under a potential range of retrieval rates of off-treatment data. This simulation exercise illustrates how the selection of the estimators for an estimand could have a strong impact on the estimates of the treatment effect and, consequently, on the decision making in a clinical trial.     

Treatment Effects,Comparisons, and Randomization

Abstract

The causal estimands of interest in a clinical trial are treatment effects, and a treatment effect is a comparison of the outcomes for the same patients, or for similar groups of patients, on different treatments. The draft addendum to ICH E9, Statistical Principles for Clinical Trials discusses five possible strategies for defining treatment effects to be estimated, on a more or less equal footing. Indeed, they may appear almost as items on a menu. However, the situations in which they can usefully be applied may be distinct. Furthermore, the classification into five categories may obscure some fairly close relationships between categories and some important differences within categories. Especially, the category of hypothetical estimands is so broad as to include useful estimands that are nothing like anything in the other categories, useful estimands that are notably similar to something else, and possibly useless estimands that are not clearly distinguished from useful ones. This article will discuss the strategies in the addendum. It will offer some advice on when different strategies for constructing estimands may be appropriate. It will classify strategies to make some differences and some similarities clearer; especially it will unpack the hypothetical category. The three classes to be considered are external comparisons, nonrandomized comparisons, and randomized comparisons.     

Estimation of causal effects in clinical endpoint bioequivalence studies in the presence of intercurrent events: noncompliance and missing data

In clinical endpoint bioequivalence (BE) studies, the primary analysis for assessing equivalence between a generic and an innovator product is based on the observed per-protocol (PP) population (usually completers and compliers). However, missing data and noncompliance are post-randomization intercurrent events and may introduce selection bias. Therefore, PP analysis is generally not causal. The FDA Missing Data Working Group recommended using “causal estimands of primary interest.” In this paper, we propose a principal stratification causal framework and co-primary causal estimands to test equivalence, which was also recommended by the recently published ICH E9 (R1) addendum to address intercurrent events. We identify three conditions under which the current PP estimator is unbiased for one of the proposed co-primary causal estimands – the “Survivor Average Causal Effect” (SACE) estimand. Simulation shows that when these three conditions are not met, the PP estimator is biased and may inflate Type 1 error and/or change power. We also propose a tipping point sensitivity analysis to evaluate the robustness of the current PP estimator in testing equivalence when the sensitivity parameters deviate from the three identified conditions, but stay within a clinically meaningful range. Our work is the first causal equivalence assessment in equivalence studies with intercurrent events.     

A Conservative Approach for Analysis of Noninferiority Trials With Missing Data and Subject Noncompliance

Abstract

Noninferiority clinical trials aim to show an experimental treatment is therapeutically no worse than standard of care, particularly if the new treatment is preferred for reasons such as cost, convenience, safety, and so on. Noninferiority trials are by nature less conservative than superiority studies: protocol violations may increase bias toward the alternative hypothesis of noninferiority. Our objective was to compare multiple imputation, a linear mixed model, and other methods for analyzing a longitudinal trial with missing data in intention-to-treat and per-protocol populations. We simulated trials with missing data and noncompliance due to treatment inefficacy under varying trial conditions (e.g., trajectory of treatment effects, correlation between repeated measures, and missing data mechanism), assessing each approach by estimating bias, Type I error, and power. We found that multiple imputation using auxiliary data on noncompliance in the imputation model performed best. A hybrid intention-to-treat/per-protocol multiple imputation approach with a missing not at random imputation model produced low Type I error, was unbiased and maintained reasonable power to detect noninferiority. We conclude that the anti-conservatism of noninferiority trial estimands conforming with the intention-to-treat principle may be offset by imputation models that include variables on intercurrent events. Supplementary materials for this article are available online.     

Analysis of Longitudinal Trials with Protocol Deviation: A Framework for Relevant,Accessible Assumptions,and Inference via Multiple Imputation

Protocol deviations, for example, due to early withdrawal and noncompliance, are unavoidable in clinical trials. Such deviations often result in missing data. Additional assumptions are then needed for the analysis, and these cannot be definitively verified from the data at hand. Thus, as recognized by recent regulatory guidelines and reports, clarity about these assumptions and their implications is vital for both the primary analysis and framing relevant sensitivity analysis. This article focuses on clinical trials with longitudinal quantitative outcome data. For the target population, we define two estimands, the de jure estimand, “does the treatment work under the best case scenario,” and the de facto estimand, “what would be the effect seen in practice.” We then carefully define the concept of a deviation from the protocol relevant to the estimand, or for short a deviation. Each patient's postrandomization data can then be divided into predeviation data and postdeviation data. We set out an accessible framework for contextually appropriate assumptions relevant to de facto and de jure estimands, that is, assumptions about the joint distribution of pre- and postdeviation data relevant to the clinical question at hand. We then show how, under these assumptions, multiple imputation provides a practical approach to estimation and inference. We illustrate with data from a longitudinal clinical trial in patients with chronic asthma.     

The Implications of Current Methodological Issues for International Guidance

We describe a number of methodological issues that have arisen in recent years with respect to the clinical trial work included in marketing applications for new medicinal products. The issues are then used as a vehicle to discuss the relevant guidance provided in the ICH E9 guideline entitled “Statistical Principles for Clinical Trials.” We explore whether there are apparent deficiencies in current guidance and whether there is potential new subject matter to cover. The aspects of clinical trial methodology addressed in this way include interim analysis, statistical analysis plans, the analysis population, missing data, multiplicity of end-points, subgroup analysis, and prespecification. We conclude that the ICH E9 guideline is still fully relevant as it stands, but that more might usefully have been said on some topics. In addition, some new methodological challenges have emerged since it was written. Consensus agreement should be sought on a number of new topics and, if achieved, the appropriate place to record the agreement should be identified, either in new guidance documents or in an updated ICH E9.     

An alternative way to classify missing data mechanism in clinical trials--a dialogue on missing data

Statisticians in pharmaceutical field are constantly challenged by missing data caused by patient dropout in clinical trials. What the targeted population parameter for statistical inference should be when missing data are present has been a much-debated point. Many missing data methods aim at the so-called hypothetical parameter, i.e., treatment effect of a drug assuming no patients dropout from a clinical trial for the drug. Other methods intend to combine all dropout information into the treatment effect estimate. We believe that patient dropouts should not be treated equally when determining the population parameter of treatment effect. The objective of clinical trials, after all, is to evaluate a drug's effect on patients. Dropouts due to drug-related reasons such as drug-induced adverse experience are part of the drug's attributes, while dropout due to non-drug-related reasons, such as protocol deviation, are not inherent characteristics of the drug. Hence we propose to classify the patient dropouts into two classes: intrinsic (drug-related) and extrinsic (non-drug-related) dropouts. The former should be taken into account when defining the population parameter of the treatment effect, while the latter should not be. This classification will help determine a target population parameter that depicts a fair picture of a drug's effect, while the common classification of missing data as missing completely random (MCAR), missing at random (MAR), and missing not at random (MNAR) will help define appropriate statistical approach to analysis when missing data exist. Other related issues, such as statistical inference under this classification and implementing the classification in real clinical trials, are also touched upon here.     

Modeling Missingness for Time-to-Event Data: A Case Study in Osteoporosis

Abstract

Clinical trials of long duration are often hampered by high dropout rates, making statistical inference and interpretation of results difficult. Statistical inference should be based on models selected according to whether missingness is independent of response [missing completely at random (MCAR)], or depends on response either through observed responses only [missing at random (MAR)] or through unobserved responses [nonignorable missing (NIM)]. If the dropout rate is high and little is known about the dropout mechanism, plausible nonignorable missing scenarios should be investigated as a sensitivity tool, offering the data analyst an understanding of the robustness of conclusions. Modeling missingness is illustrated by an analysis of an interval censored time-to-event outcome from a 5-year clinical trial on fracture response in osteoporosis in which the overall dropout rate was substantial. In this article, we provide an overview of a reanalysis accounting for possible nonignorable missingness, emphasize the importance of modeling the dropout and response mechanisms jointly, and highlight critical points arising in missing data problems.     

Handling missing data in vaccine clinical trials for immunogenicity and safety evaluation

In clinical trials, study subjects are usually followed for a period of time after treatment, and the missing data issue is almost inevitable due to various reasons, including early dropout or lost-to-follow-up. It is important to take the missing data into consideration at the study design stage to minimize its occurrence throughout the study and to prospectively account for it in the analyses. There are many methods available in the literature that are designed to handle the missing data issue under various settings. Vaccines are biological products that are primarily designed to prevent infectious diseases, and are different from pharmaceutical products, which traditionally have been chemical products designed to treat or cure diseases. While a lot of similarities exist between clinical trials for vaccines and those for pharmaceutical products, there are some unique issues in vaccine trials, including how to handle the missing data, which calls for special considerations. In this report we present a variety of statistical approaches for analyses of vaccine immunogenicity and safety trials in the presence of missing data. The methods are illustrated with numerical simulations and vaccine trial examples.     

A Comparison of Frequentist and Bayesian Model Based Approaches for Missing Data Analysis: Case Study with a Schizophrenia Clinical Trial

Missing data are common in clinical trials and could lead to biased estimation of treatment effects. The National Research Council (NRC) report suggests that sensitivity analysis on missing data mechanism should be a mandatory component of the primary reporting of findings from clinical trials, and regulatory agencies are requesting more thorough sensitivity analyses from sponsors. However, recent literature research showed that missing data were almost always inadequately handled. This is partially due to the lack of standard software packages and straightforward implementation platform. With recent availability of flexible Bayesian software packages such as WinBUGS, SAS Proc MCMC, and Stan, it is relatively simple to develop Bayesian methods to address complex missing data problems while incorporating the uncertainty. In this article, we present a case study from the DIA Bayesian Scientific Working Group (BSWG) on Bayesian approaches for missing data analysis. We illustrate how to use Bayesian approaches to fit a few commonly used frequentist missing data models. The properties, advantage, and flexibility of the Bayesian analysis methods will be discussed using a case study based on a schizophrenia clinical trial. Supplementary materials for this article are available online.     

新药临床试验期间药物警戒和风险控制研究二:临床试验方案、研究者手册和知情同意书监管要求研究

目的：对比临床试验期间国内外临床试验方案、研究者手册、知情同意书这三个关键要素中关于药物警戒监管的差异,为我国科学监管提出建议,从而加强新药临床试验期间安全性风险控制及受试者保护。方法：通过文献和法规研究,对比临床试验期间国内外临床试验方案、研究者手册、知情同意书这三个关键要素中关于药物警戒监管的差异,就如何更好应用临床试验方案、研究者手册、知情同意书保护受试者展开讨论。结果与结论：临床试验方案、研究者手册、知情同意书是临床研究中关于风险控制和受试者保护的关键安全性管理要素。我国临床试验期间上述三项管理要求的指导原则尚待完善,目前规定大多仅基于ICH的普遍要求,缺少基于国内行业现状的监管及指导。     

Inference for Clinical Trials with Some Protocol Amendments

ABSTRACT

The use of adaptive methods in clinical development has become very popular in recent years due to its flexibility in modifying trial procedures and/or statistical procedures of on-going clinical trials. Modifications to trial procedures are usually documented by protocol amendments. However, the actual patient population after protocol amendments could deviate from the originally targeted patient population. In addition, protocol amendments made based on accrued data of the on-going trial may distort the sampling distribution of the statistic designed for the case of no protocol change. In this article, we model the population deviations due to protocol amendments using some covariates and study how to develop a valid statistical inference procedure. An example concerning an asthma trial is presented for illustration.     

The analysis, interpretation, and presentation of quality of life data

All too often in clinical trials the assessment of quality of life is seen as a bolt-on study. Consequently insufficient consideration is often given to its design, collection, analysis and presentation, and its impact on the trial results and on clinical practice is minimal. In many trials quality of life is a key endpoint, and it is vital that quality of life expertise is involved as soon as possible in the design. Setting a priori quality of life hypotheses will focus the decisions regarding which questionnaire to use, when to administer it, the sample size required, and the primary analyses. Nevertheless quality of life data are complex, and require much skill in determining how to deal with multi-dimensional and longitudinal data, much of which is often missing. There are no agreed standard ways of analysing and presenting quality of life data, but there are guidelines, which if followed, will add transparency to the way results have been calculated. Understanding the impact of treatments on their quality of life is vital to patients, and it is up to us, as statisticians and trialists, to present the data as clearly as we can.     

Challenges of risk-based monitoring of clinical trials

Abstract

The oversight of monitoring activities in clinical trials generally comes from the ICH GCP Guidelines and covers a wide range of responsibilities: trial progress oversight, adherence to the Study Protocol, Standard Operating Procedures, Good Clinical Practice, applicable regulatory requirement(s) and Source Data verification vs accuracy and completeness of the Case Report Form entries. Risk-based monitoring was developed and adopted by Sponsors, Investors, and CROs to decrease the costs of clinical trials and make study management more effective. Both the EMA and the FDA support such an approach with their papers. Interestingly, the review of the EMA “Annual report of the GCP Inspectors working group in 2012” has consistently shown persistence of a significant number of findings in fields/areas of monitoring that cannot be fully or partially captured with a centralized or targeted monitoring approach and cannot be identified, such as essential documents, presence and adherence to SOPs, trainings, and the quality of source documentation. Such results open up new challenges for Sponsors, CROs, and other stakeholders. As long as all current ICH GCP Guidelines are a cornerstone of clinical research, monitoring plans and risk assessments will include overseeing a significant pool of additional aspects, apart from the SDV.     

Missing Data in Noninferiority Trials

Missing data in clinical trials has been widely discussed in the literature, but issues specific to missing data in noninferiority trials have rarely been addressed. The goal of this article is to present missing data issues that are particularly important in noninferiority trials. Issues of assay sensitivity and the constancy assumption are affected by missing data. Importantly, these issues are not solved by per protocol analyses which remove patient data based on postrandomization criteria. We advocate collecting data to the extent possible for sensitivity analyses. We discuss some other issues that remain unresolved in assessing the impact of missing data in noninferiority trials. A simulation analysis of different strategies for assessing noninferiority in the presence of missing data is reported for a clinical trial comparing two treatments. Single imputation procedures and observed case analyses resulted in reduced power due to missing data and occasionally in inflation in Type I error rate or bias in estimates of treatment effect. The mixed-effect model repeated measures approach resulted in a method that controlled the Type I error rate when data are missing at random, and often with higher power than the other two methods. Further work on multiple imputation procedures is desired.     

Patient-reported outcome measurement in drug discovery: a tool to improve accuracy and completeness of efficacy and safety data

ABSTRACT

Introduction: Patient-reported outcomes (PROs) reflect how patients feel and function as conveyed directly by patients themselves, for example symptoms and physical functioning. PRO measures can be included in any phase of product development as primary, secondary, or exploratory endpoints to understand the impact of treatment on the patient experience.

Areas covered: In this review, the authors describe approaches commonly used to assess PROs in drug development programs, including how to select or develop outcomes and measures, implement these in trials, and analyze data. Barriers and facilitators for effectively including PROs in clinical trials are discussed, and strategies for making labeling claims based on PRO data are noted.

Expert opinion: Early planning with PRO experts is recommended to assure a successful PRO strategy in a product development program. Outcomes that are meaningful to patients should be rationally identified early in a product development program based on qualitative work, literature search and/or assumed mechanism of action. Measures corresponding to those outcomes should be selected or developed, with demonstration of robust psychometric properties in a related patient population. Ideally, these measures will be tested prior to a pivotal trial to support the design and analysis in the pivotal trial. Selection of timing and mode of assessments should be specified a priori and justified. If a labeling claim is sought, a statistical plan should be pre-specified. Strategies to minimize and handle missing data in analyses should be planned up front.     

药物临床试验质量影响因素调查分析

目的：探讨药物临床试验实施过程中质量影响的关键因素。方法：以药物临床试验质量管理规范（Good Clinical Practice,GCP）培训、方案设计、知情同意书签署、受试者依从性、临床试验记录和报告、试验用药品管理、不良事件（Adverse Events,AE）/严重不良事件（Serious Adverse Events,SAE）上报、项目质控措施和比例等内容为参照制定调查问卷,对药物临床试验相关人员进行调查分析。结果：合计收到191份问卷,其中有效问卷183份,有效率95.8%。临床试验中常见方案设计与医院常规操作相违背（61.20%）、知情同意书信息填写有误/不规范（34.97%）、访视超窗（38.80%）、试验数据/报告填写/修改不规范（55.74%）、试验用药品相关管理记录不完整（36.61%）、AE/SAE漏报（40.44%）等问题,以及GCP培训、质控力度、研究团队、申办方质量管理体系建设等都是影响药物临床试验质量的关键因素。结论：加强药物临床试验参与各方的培训、提高GCP认知是保障临床试验整体质量的有力措施。     

Are We Really That Blind?

ABSTRACT

In double-blind randomized clinical trials, it is common practice to randomize patients using randomly permuted blocks. In this article, it is demonstrated that before unblinding statistical inference of the treatment effects can be conducted, yielding consistent and rather precise estimates even in the presence of an additive block effect. With an even greater precision, the within-group standard deviation on which power calculation are usually based can be inferred from blinded data. The use of blocks of random lengths as suggested by ICH-E9 in the (unlikely) case that previous treatment allocation can be guessed by strong pharmacological effects, merely complicates the analysis but blinded inference can still be conducted without much extra loss of information. On the one hand, one might argue that this possibility of blinded inference takes away the need of conducting interim analyses for administrative or business reasons or for sample size reestimation. On the other hand, however, it most probably will have a disputable, positive or negative effect on the conduct of the remainder of the trial. If regulators and the pharmaceutical world at large want to avoid this possibility, then other unrestricted, biased coin, or more general dynamic allocation randomization procedures may be less controversial alternatives. It at least provides another strong argument in favor of using large blocks as the precision of blinded inference decreases with increasing block lengths.

If blinded inferences are deemed a useful replacement of interim analyses in nonpivotal trials, then further guidelines will be needed on consequent decision-making aspects.