Model selection versus model averaging in dose finding studies

A key objective of Phase II dose finding studies in clinical drug development is to adequately characterize the dose response relationship of a new drug. An important decision is then on the choice of a suitable dose response function to support dose selection for the subsequent Phase III studies. In this paper, we compare different approaches for model selection and model averaging using mathematical properties as well as simulations. We review and illustrate asymptotic properties of model selection criteria and investigate their behavior when changing the sample size but keeping the effect size constant. In a simulation study, we investigate how the various approaches perform in realistically chosen settings. Finally, the different methods are illustrated with a recently conducted Phase II dose finding study in patients with chronic obstructive pulmonary disease. Copyright © 2016 John Wiley & Sons, Ltd.     

Comparison of Bayesian model averaging and stepwise methods for model selection in logistic regression

Logistic regression is the standard method for assessing predictors of diseases. In logistic regression analyses, a stepwise strategy is often adopted to choose a subset of variables. Inference about the predictors is then made based on the chosen model constructed of only those variables retained in that model. This method subsequently ignores both the variables not selected by the procedure, and the uncertainty due to the variable selection procedure. This limitation may be addressed by adopting a Bayesian model averaging approach, which selects a number of all possible such models, and uses the posterior probabilities of these models to perform all inferences and predictions. This study compares the Bayesian model averaging approach with the stepwise procedures for selection of predictor variables in logistic regression using simulated data sets and the Framingham Heart Study data. The results show that in most cases Bayesian model averaging selects the correct model and out-performs stepwise approaches at predicting an event of interest.     

An objective Bayesian analysis of a crossover design via model selection and model averaging

Inference about the treatment effect in a crossover design has received much attention over time owing to the uncertainty in the existence of the carryover effect and its impact on the estimation of the treatment effect. Adding to this uncertainty is that the existence of the carryover effect and its size may depend on the presence of the treatment effect and its size. We consider estimation and testing hypothesis about the treatment effect in a two‐period crossover design, assuming normally distributed response variable, and use an objective Bayesian approach to test the hypothesis about the treatment effect and to estimate its size when it exists while accounting for the uncertainty about the presence of the carryover effect as well as the treatment and period effects. We evaluate and compare the performance of the proposed approach with a standard frequentist approach using simulated data, and real data. Copyright © 2016 John Wiley & Sons, Ltd.     

Bayesian analysis of case-control studies

A Bayesian approach to the estimation of an odds ratio from case-control data is considered. The exact posterior density of the odds ratio and its moments are derived. A log-normal approximation to the density is shown to be adequate for practical purposes. Mechanisms for setting prior parameters are discussed and some examples are presented.     

Bayesian variable and model selection methods for genetic association studies

Variable selection is growing in importance with the advent of high throughput genotyping methods requiring analysis of hundreds to thousands of single nucleotide polymorphisms (SNPs) and the increased interest in using these genetic studies to better understand common, complex diseases. Up to now, the standard approach has been to analyze the genotypes for each SNP individually to look for an association with a disease. Alternatively, combinations of SNPs or haplotypes are analyzed for association. Another added complication in studying complex diseases or phenotypes is that genetic risk for the disease is often due to multiple SNPs in various locations on the chromosome with small individual effects that may have a collectively large effect on the phenotype. Hence, multi-locus SNP models, as opposed to single SNP models, may better capture the true underlying genotypic-phenotypic relationship. Thus, innovative methods for determining which SNPs to include in the model are needed. The goal of this article is to describe several methods currently available for variable and model selection using Bayesian approaches and to illustrate their application for genetic association studies using both real and simulated candidate gene data for a complex disease. In particular, Bayesian model averaging (BMA), stochastic search variable selection (SSVS), and Bayesian variable selection (BVS) using a reversible jump Markov chain Monte Carlo (MCMC) for candidate gene association studies are illustrated using a study of age-related macular degeneration (AMD) and simulated data.     

A Bayesian partition model for case-control studies on highly polymorphic candidate genes

We present a new statistical model for the analysis of case-control or cohort studies examining a highly polymorphic candidate disease susceptibility gene. Many genotypes are possible for such a gene. Consequently, the average number of subjects having each genotype will be modest. If analyzed separately, the risks associated with most genotypes will be estimated imprecisely. Our Bayesian partition model clusters genotypes according to risk, only allowing partitions that satisfy a particular assumption about the joint effect of the two alleles making up a genotype. This assumption is genetically plausible, imposes structure on the set of genotype risks, and still leaves a highly flexible model. By Bayesian model averaging over partitions, the model becomes, in effect, a semiparametric model for genotype risk. It allows borrowing of strength, i.e., estimates of risk for one genotype are informed by the risk estimates of all the genotypes. We present the results of fitting the model to two datasets, one simulated and one genuine case-control study of the NAT1 gene and lung cancer, and compare it in a simulation study with a haplotype relative risk model. The partition model enables genotype risks to be estimated more accurately and the alleles to be ranked according to risk.     

Reporting and selection bias in case-control studies of congenital malformations.

Retrospective studies of congenital malformations frequently rely on exposures reported by study subjects. Differential error in exposure reporting by cases and controls, which has alternatively been referred to as "recall bias" and "reporting bias," may result in a biased effect measure. Some authors have attempted to avoid reporting bias by comparing exposures between two malformed groups, rather than between cases and nonmalformed controls. This approach, however, may introduce its own bias, which we call selection bias. Both reporting bias and selection bias are shown to be algebraically equivalent to bias arising from exposure misclassification. The magnitudes of these biases are compared for a range of plausible parametric values. The case-control design is sensitive to both differential reporting and selection bias, and the choice of study design involves balancing these two sources of bias.     

A Bayesian model averaging approach for cost-effectiveness analyses

We consider the problem of assessing new and existing technologies for their cost-effectiveness in the case where data on both costs and effects are available from a clinical trial, and we address it by means of the cost-effectiveness acceptability curve. The main difficulty in these analyses is that cost data usually exhibit highly skew and heavy-tailed distributions so that it can be extremely difficult to produce realistic probabilistic models for the underlying population distribution, and in particular to model accurately the tail of the distribution, which is highly influential in estimating the population mean. Here, in order to integrate the uncertainty about the model into the analysis of cost data and into cost-effectiveness analyses, we consider an approach based on Bayesian model averaging: instead of choosing a single parametric model, we specify a set of plausible models for costs and estimate the mean cost with a weighted mean of its posterior expectations under each model, with weights given by the posterior model probabilities. The results are compared with those obtained with a semi-parametric approach that does not require any assumption about the distribution of costs.     

Incorporating model uncertainty in cost-effectiveness analysis: a Bayesian model averaging approach

Recently, several authors have proposed the use of linear regression models in cost-effectiveness analysis. In this paper, by modelling costs and outcomes using patient and Health Centre covariates, we seek to identify the part of the cost or outcome difference that is not attributable to the treatment itself, but to the patients' condition or to characteristics of the Centres. Selection of the covariates to be included as predictors of effectiveness and cost is usually assumed by the researcher. This behaviour ignores the uncertainty associated with model selection and leads to underestimation of the uncertainty about quantities of interest. We propose the use of Bayesian model averaging as a mechanism to account for such uncertainty about the model. Data from a clinical trial are used to analyze the effect of incorporating model uncertainty, by comparing two highly active antiretroviral treatments applied to asymptomatic HIV patients. The joint posterior density of incremental effectiveness and cost and cost-effectiveness acceptability curves are proposed as decision-making measures.     

Control selection strategies in case-control studies of childhood diseases

To address concerns regarding the representativeness of controls in case-control studies, two selection strategies were evaluated in a study of childhood leukemia, which commenced in California in 1995. The authors selected two controls per case: one from among children identified by using computerized birth records and located successfully, the other from a roster of friends; both were matched on demographic factors. Sixty-four birth certificate-friend control pairs were enrolled (n = 128). Additionally, 192 "ideal" controls were selected without tracing from the birth records. Data on parental ages, parental education, mother's reproductive history, and birth weight were obtained from the birth certificates of all 320 subjects. For all variables except birth weight, the differences between birth certificate and ideal controls were smaller than those between friend and ideal controls. None of the differences between birth certificate and ideal controls was significant, whereas two factors were significantly different between friend and ideal controls. These findings suggest that friend controls may be less representative than birth certificate controls. Despite difficulty in tracing and a seemingly low participation rate (49.0% for 560 enrolled birth certificate controls), using birth records to recruit controls appears to provide a representative sample of children and an opportunity to assess the representativeness of controls.     

Variable selection: current practice in epidemiological studies

Selection of covariates is among the most controversial and difficult tasks in epidemiologic analysis. Correct variable selection addresses the problem of confounding in etiologic research and allows unbiased estimation of probabilities in prognostic studies. The aim of this commentary is to assess how often different variable selection techniques were applied in contemporary epidemiologic analysis. It was of particular interest to see whether modern methods such as shrinkage or penalized regression were used in recent publications. Stepwise selection methods remained the predominant method for variable selection in publications in epidemiological journals in 2008. Shrinkage methods were not used in any of the reviewed articles. Editors, reviewers and authors have insufficiently promoted the new, less controversial approaches of variable selection in the biomedical literature, whereas statisticians may not have adequately addressed the method’s feasibility.     

Variable selection for proportional odds model

In this paper we study the problem of variable selection for the proportional odds model, which is a useful alternative to the proportional hazards model and might be appropriate when the proportional hazards assumption is not satisfied. We propose to fit the proportional odds model by maximizing the marginal likelihood subject to a shrinkage-type penalty, which encourages sparse solutions and hence facilitates the process of variable selection. Two types of shrinkage penalties are considered: the LASSO and the adaptive-LASSO (ALASSO) penalty. In the ALASSO penalty, different weights are imposed on different coefficients such that important variables are more protectively retained in the final model while unimportant ones are more likely to be shrunk to zeros. We further provide an efficient computation algorithm to implement the proposed methods, and demonstrate their performance through simulation studies and an application to real data. Numerical results indicate that both methods can produce accurate and interpretable models, and the ALASSO tends to work better than the usual LASSO.     

Posterior maximization and averaging for Bayesian working model choice in the continual reassessment method

The continual reassessment method (CRM) is a method for estimating the maximum tolerated dose in a dose-finding study. Traditionally, use is made of a single working model or 'skeleton' idealizing an underlying true dose-toxicity relationship. This working model is chosen either by discussion with investigators or published data, before the beginning of the trial or simply on the basis of operating characteristics. To overcome the arbitrariness of the choice of such a single working model, Yin and Yuan (biJ. Am. Statist. Assoc. 2009; 104:954-968) propose a model averaging over a set of working models. Here, instead of averaging, we investigate some alternative Bayesian model criteria that maximize the posterior distribution. We propose three adaptive model-selecting CRMs using the Bayesian model selection criteria, in which we specify in advance a collection of candidate working models for the dose-toxicity relationship, especially initial guesses of toxicity probabilities, and adaptively select the only one working model among the candidates updated by using the original CRM for each working model, based on the posterior model probability, the posterior predictive loss or the deviance information criteria, during the course of the trial. These approaches were compared via a simulation study with the model averaging approach.     

Dose-response in case-control studies.

The evidence provided by a case-control study on the association between a disease and some factor is strengthened if the extent of exposure to the factor is categorised into several groups or measured on a continuous scale. Then dose-response relationships can be estimated. The methods available are illustrated by application to data on lung cancer and chrysotile asbestos exposure from Quebec in which there were three matched controls for each case. Regression-type models were fitted assuming that the relative risk of lung cancer was linearly related to an exposure measure; a covariate, smoking, was also included in the analysis. The data were first analysed ignoring the matching and secondly taking account of the matching. The methodology for the latter analysis has only recently been developed; formerly, matched studies were of necessity analysed as unmatched. Although, in this particular example, the unmatched and matched analyses gave similar results, this is not always the case and it is argued that, now that the methodology is available, matched case-control studies should be analysed taking proper account of the matching.     

Variable selection in the functional linear concurrent model

We propose methods for variable selection in the context of modeling the association between a functional response and concurrently observed functional predictors. This data structure, and the need for such methods, is exemplified by our motivating example: a study in which blood pressure values are observed throughout the day, together with measurements of physical activity, location, posture, affect or mood, and other quantities that may influence blood pressure. We estimate the coefficients of the concurrent functional linear model using variational Bayes and jointly model residual correlation using functional principal components analysis. Latent binary indicators partition coefficient functions into included and excluded sets, incorporating variable selection into the estimation framework. The proposed methods are evaluated in simulations and real‐data analyses, and are implemented in a publicly available R package with supporting interactive graphics for visualization. Copyright © 2017 John Wiley & Sons, Ltd.     

Integrative variable selection via Bayesian model uncertainty

We are interested in developing integrative approaches for variable selection problems that incorporate external knowledge on a set of predictors of interest. In particular, we have developed an integrative Bayesian model uncertainty (iBMU) method, which formally incorporates multiple sources of data via a second‐stage probit model on the probability that any predictor is associated with the outcome of interest. Using simulations, we demonstrate that iBMU leads to an increase in power to detect true marginal associations over more commonly used variable selection techniques, such as least absolute shrinkage and selection operator and elastic net. In addition, iBMU leads to a more efficient model search algorithm over the basic BMU method even when the predictor‐level covariates are only modestly informative. The increase in power and efficiency of our method becomes more substantial as the predictor‐level covariates become more informative. Finally, we demonstrate the power and flexibility of iBMU for integrating both gene structure and functional biomarker information into a candidate gene study investigating over 50 genes in the brain reward system and their role with smoking cessation from the Pharmacogenetics of Nicotine Addiction and Treatment Consortium. Copyright © 2013 John Wiley & Sons, Ltd.     

Objective Bayesian model selection for Cox regression

There is now a large literature on objective Bayesian model selection in the linear model based on the g‐prior. The methodology has been recently extended to generalized linear models using test‐based Bayes factors. In this paper, we show that test‐based Bayes factors can also be applied to the Cox proportional hazards model. If the goal is to select a single model, then both the maximum a posteriori and the median probability model can be calculated. For clinical prediction of survival, we shrink the model‐specific log hazard ratio estimates with subsequent calculation of the Breslow estimate of the cumulative baseline hazard function. A Bayesian model average can also be employed. We illustrate the proposed methodology with the analysis of survival data on primary biliary cirrhosis patients and the development of a clinical prediction model for future cardiovascular events based on data from the Second Manifestations of ARTerial disease (SMART) cohort study. Cross‐validation is applied to compare the predictive performance with alternative model selection approaches based on Harrell's c‐Index, the calibration slope and the integrated Brier score. Finally, a novel application of Bayesian variable selection to optimal conditional prediction via landmarking is described. Copyright © 2016 John Wiley & Sons, Ltd.     

Evaluation of random digit dialing as a method of control selection in case-control studies.

Control groups selected by random digit dialing are frequently used in case-control studies. Concern about the potential for bias in these control groups has been expressed, primarily because of low response rates. This study compares the characteristics of a hypothetical control group consisting of 341 men and women aged 40-74 years, selected by random digit dialing and participating in an interview in 1990, with the characteristics of 15,563 men and women aged 40-74 years who participated in a privately conducted census in the same upstate New York county in 1989. For most measures, no differences were seen between the random digit dialing sample and the census population. However, the hypothetical control group was more likely to have had their cholesterol checked in the past 2 years and was somewhat more likely to have had other screening tests as well. In addition, the hypothetical control group was somewhat better educated. The results suggest that, at least in this setting, control groups selected by random digit dialing are representative of the general population in most respects; however, caution should be used when studying the relation between screening tests and disease occurrence by means of case-control studies using controls selected by random digit dialing.