Publishing nutrition research: a review of multivariate techniques--part 1

This article is the seventh in a series reviewing the importance of research design, analyses, and epidemiology in the conduct, interpretation, and publication of nutrition research. Although there are a variety of factors to consider before conducting nutrition research, the techniques used to conduct the statistical analysis are fundamental for translating raw data into interpretable findings. The statistical approach must be considered during the design phase of any study and often involves the use of multivariate analytical techniques. Multivariate analytical techniques represent a variety of mathematical models used to measure and quantify an exposure–disease or an exposure–outcome association, taking into account important factors that can influence this relationship. The primary purpose of this review is to introduce the more commonly used multivariate techniques, including linear and logistic regression (simple and multiple), and survival analyses (Kaplan Meier plots and Cox regression). These techniques are described in detail, providing basic definitions and practical examples with nutrition relevancy. An appreciation for the general principles within and presented previously in this article series is vital for enhancing the rigor in which nutrition-related research is implemented, reviewed, and published.     

The validity of subscores for a credentialing test

Subscores resulting from the administration of high-stakes tests to candidates for credentials in the health professions are desirable for two reasons. First, failing candidates want a profile of performance to plan future remedial studies. Second, training institutions want a profile of performance for their graduates to better evaluate their training. The validity of the interpretation or use of subscores depends on a summative judgment based on a combination of reasoning and empirical analyses, known as validation. We describe this reasoning process and show that with a large credentialing test the validity of any subscore interpretation or use can and should be studied systematically. Validity evidence should be established to support the interpretation and use of subscores that we intend to report. Some principles arise in this study related to the validity of subscores, and some procedures are proposed to help testing program personnel better validate the use of subscores.     

The analysis of treatment effects for recurring episodic conditions

Many chronic disease processes feature acute episodic conditions which warrant therapeutic intervention to alleviate symptoms or reduce the risk of further complications. Examples of such disease processes arise in fields such as neurology, where migraineurs experience recurrent attacks of migraine, and respirology, where patients suffering from asthma, cystic fibrosis, or chronic obstructive pulmonary disease may experience recurrent exacerbations. In randomized clinical trials, patients suffering from diseases of this sort are often randomized to one of several treatments and followed over a fixed period of time, during which any episodes are treated with the assigned treatment. When the outcome of interest is a response to treatment at each episode, the data have a similar structure to longitudinal data from studies with prescheduled follow‐up assessments, and it is commonplace for analyses to be based on the corresponding methodology. However, this approach ignores the fact that the timing of episodes, and hence the number observed in any given period, is stochastic. In this tutorial we demonstrate the biases that result from naive analyses, discuss analyses that account for the complete stochastic nature, and use a recent migraine trial for illustration. We conclude with some considerations for the design of randomized trials where the unit of analysis is the episode rather than the patient. Copyright © 2010 John Wiley & Sons, Ltd.     

An exact method for analysis following a two-stage phase II cancer clinical trial

This paper presents an exact method for the analysis of a phase II cancer clinical trial conducted using a two-stage design in which early stopping may be allowed for either futility or efficacy. The method provides a point and interval estimate of the response probability associated with the treatment under investigation and a p-value for testing whether this exceeds some standard null value. Two-stage designs are often used in phase II trials in oncology for reasons of ethics and efficiency, but this design feature is seldom taken into account when the results are analyzed. After any two-stage design or multi-stage design, the method for analysis should take into account the previous interim analyses performed, otherwise the results will be biased. In this paper, an approximate approach based on a log-odds ratio parameterisation will be compared with an exact method through the calculation of the precise coverage probabilities of each approach.     

Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation

Propensity score methodology can be used to help design observational studies in a way analogous to the way randomized experiments are designed: without seeing any answers involving outcome variables. The typical models used to analyze observational data (e.g., least squares regressions, difference of difference methods) involve outcomes, and so cannot be used for design in this sense. Because the propensity score is a function only of covariates, not outcomes, repeated analyses attempting to balance covariate distributions across treatment groups do not bias estimates of the treatment effect on outcome variables. This theme will the primary focus of this article: how to use the techniques of matching, subclassification and/or weighting to help design observational studies. The article also proposes a new diagnostic table to aid in this endeavor, which is especially useful when there are many covariates under consideration. The conclusion of the initial design phase may be that the treatment and control groups are too far apart to produce reliable effect estimates without heroic modeling assumptions. In such cases, it may be wisest to abandon the intended observational study, and search for a more acceptable data set where such heroic modeling assumptions are not necessary. The ideas and techniques will be illustrated using the initial design of an observational study for use in the tobacco litigation based on the NMES data set.     

A general statistical principle for changing a design any time during the course of a trial

A general method is presented that allows the researcher to change statistical design elements such as the residual sample size during the course of an experiment, to include an interim analysis for early stopping when no formal rule for early stopping was foreseen, to increase or reduce the number of planned interim analyses, to change time points and the type I error spending function for the further design of interim analyses, or to change the test statistic, the outcome measure, etc. At the time of a pre-planned interim analysis for early stopping or at any time of an interim look without spending part of the type I error level the method offers the option to completely redesign the remaining part of the trial, without affecting the type I error level. The method is described in the usual Brownian motion model and extended to the general context of statistical decision functions. It is based on the conditional rejection probability of a decision variable.     

Evaluation of Community-Level Effects of Communities That Care on Adolescent Drug Use and Delinquency Using a Repeated Cross-Sectional Design

The Communities That Care (CTC) prevention system has shown effects on reducing incidence and prevalence of problem behaviors among a panel of youth followed from 5th through 12th grade. The present report examines whether similar intervention effects could be observed using a repeated cross-sectional design in the same study. Data were from a community-randomized trial of 24 US towns. Cross-sectional samples of sixth, eighth, and tenth graders were surveyed at four waves. Two-stage ANCOVA analyses estimated differences between CTC and control communities in community-level prevalence of problem behaviors for each grade, adjusting for baseline prevalence. No statistically significant reductions in prevalence of problem behaviors were observed at any grade in CTC compared to control communities. Secondary analyses examined intervention effects within a “pseudo cohort” where cross-sectional data were used from sixth graders at baseline and tenth graders 4 years later. When examining effects within the pseudo cohort, CTC compared to control communities showed a significantly slower increase from sixth to tenth grade in lifetime smokeless tobacco use but not for other outcomes. Exploratory analyses showed significantly slower increases in lifetime problem behaviors within the pseudo cohort for CTC communities with high, but not low, prevention program saturation compared to control communities. Although CTC demonstrated effects in a longitudinal panel from the same community-randomized trial, we did not find similar effects on problem behaviors using a repeated cross-sectional design. These differences may be due to a reduced ability to detect effects because of potential cohort effects, accretion of those who were not exposed, and attrition of those who were exposed to CTC programming in the repeated cross-sectional sample.     

A comparison of three daily coital diary designs and a phone-in regimen

Barrier contraceptive trials and disease intervention studies often utilize coital diaries to measure sexual exposures: dates and frequency of intercourse, product use, additional or alternative contraceptive use, and menstrual bleeding. The validity of these self-reported data is a matter of debate, but if used, better diary designs are sought. We studied 3 different coital diaries, plus a phone-in regimen (none or weekly) in a 3 x 2 factorial design to compare participant ratings and promptness of recording. Our underlying presumption was that ease of and satisfaction with use, and promptness of data collection, are associated with greater accuracy. A self-completed questionnaire at the end of the study collected comparative retrospective data. Diary 1 captured information about a single day on one page and had three columns, for up to three possible acts of intercourse. Diary 2 had the same question format as the first diary, but contained 7 days per page. Diary 3 had 7 days on a page, but instead of a column for each act, participants enumerated the number of acts, the types of contraception used, and condom use details. Half of the women in each diary group phoned in their data weekly. Phone-in improved participants' satisfaction with the diary design as reflected by higher ratings of diary features. Phone-in did not improve recall of data at the end of the study for any of the diaries. There were no differences in the promptness of diary completion. Diaries 1 and 2 showed good concordance with recalled data, and participants expressed a preference for the layout of Diary 2. Women assigned to Diary 3 expressed dissatisfaction with the design and were worse at recalling data at the end of the study, probably due to the complexity of that diary design.     

Estimation of treatment effects following a sequential trial of multiple treatments

When a clinical trial is subject to a series of interim analyses as a result of which the study may be terminated or modified, final frequentist analyses need to take account of the design used. Failure to do so may result in overstated levels of significance, biased effect estimates and confidence intervals with inadequate coverage probabilities. A wide variety of valid methods of frequentist analysis have been devised for sequential designs comparing a single experimental treatment with a single control treatment. It is less clear how to perform the final analysis of a sequential or adaptive design applied in a more complex setting, for example, to determine which treatment or set of treatments amongst several candidates should be recommended. This article has been motivated by consideration of a trial in which four treatments for sepsis are to be compared, with interim analyses allowing the dropping of treatments or termination of the trial to declare a single winner or to conclude that there is little difference between the treatments that remain. The approach taken is based on the method of Rao-Blackwellization which enhances the accuracy of unbiased estimates available from the first interim analysis by taking their conditional expectations given final sufficient statistics. Analytic approaches to determine such expectations are difficult and specific to the details of the design: instead “reverse simulations” are conducted to construct replicate realizations of the first interim analysis from the final test statistics. The method also provides approximate confidence intervals for the differences between treatments.     

Interim analysis on survival data: its potential bias and how to repair it

We consider interim analyses in clinical trials or observational studies with a time-to-event outcome variable where the survival curves are compared using the hazard ratio resulting from a proportional hazards (PH) model or tested with the logrank test or another two-sample test. We show and illustrate with an example that if the PH assumption is violated, the results of interim analyses can be heavily biased. This is due to the fact that the censoring pattern in interim analyses can be completely different from the final analysis. We argue that, when the PH assumption is violated, interim analyses are only sensible if a fixed time horizon for the final analysis is specified, and at the time of the interim analysis sufficient information is available over the whole time interval up to the horizon. We show how the bias can then be remedied by introducing in the estimation and testing procedures an appropriate weighting that reflects the weights to be expected in the final analysis. The consequences for design and analysis are discussed and some practical recommendations are given.     

Personal use of hair dyes and the risk of bladder cancer: results of a meta-analysis

OBJECTIVE: This study examined the methodology of observational studies that explored an association between personal use of hair dye products and the risk of bladder cancer. METHODS: Data were pooled from epidemiological studies using a general variance-based meta-analytic method that employed confidence intervals. The outcome of interest was a summary relative risk (RRs) reflecting the risk of bladder cancer development associated with use of hair dye products vs. non-use. Sensitivity analyses were performed to explain any observed statistical heterogeneity and to explore the influence of specific study characteristics of the summary estimate of effect. RESULTS: Initially combining homogenous data from six case-control and one cohort study yielded a non-significant RR of 1.01 (0.92, 1.11), suggesting no association between hair dye use and bladder cancer development. Sensitivity analyses examining the influence of hair dye type, color, and study design on this suspected association showed that uncontrolled confounding and design limitations contributed to a spurious non-significant summary RR. The sensitivity analyses yielded statistically significant RRs ranging from 1.22 (1.11, 1.51) to 1.50 (1.30, 1.98), indicating that personal use of hair dye products increases bladder cancer risk by 22% to 50% vs. non-use. CONCLUSION: The available epidemiological data suggest an association between personal use of hair dye products and increased risk of bladder cancer.     

An analysis of two-period crossover designs with carry-over effects.

The crossover design is a type of longitudinal study with subjects receiving different treatments in different time periods. When carry-over effects are absent, the usual crossover design is structured so that all the information about treatment effects is contained in the within-subject contrasts; standard analyses are based on these within-subject contrasts and ignore any between-subject information. With carry-over effects present these standard analyses can be very inefficient, especially for suboptimal designs. We describe alternative approaches based on methods for the analysis of longitudinal data.     

Use of topical sunscreen and the risk of malignant melanoma. Results of a meta-analysis of 9,067 patients

PURPOSE: Prior epidemiological studies suggest that the use of sunscreen preparations is associated with increased risk of cutaneous malignant melanoma (CMM) although the data are conflicting. A meta-analysis was performed to evaluate this suspected association.METHODS: A protocol was developed for a meta-analysis examining the association between frequent sunscreen use versus non-use and the development of CMM. Data from observational studies were pooled using a general variance based meta-analytic method employing confidence intervals. The outcome of interest was a summary relative risk (RRs) reflecting the risk of melanoma associated with sunscreen use versus non-use. Sensitivity analyses were performed when necessary to explain any observed statistical heterogeneity.RESULTS: Eleven case-control studies enrolling 9,067 patients were combined in a meta-analysis. This yielded a RRs of 1.11 (CI = 0.37-3.32), a non-statistically significant result, (i.e. no association between sunscreen use and melanoma risk). Since the data were found to be heterogeneous, i.e. Q = 42.0 (p < 0.001), a series of sensitivity analyses were performed to explore possible sources of heterogeneity. Stratifying studies based on study design, i.e. hospital based versus population based, showed that hospital derived data were highly heterogeneous (Q = 36.9, p < 0.001) while the population registry data were not (Q = 4.9, p = 0.18). Combining those studies using population based data gave a RRs of 1.01 (95% CI = 0.46-2.28) indicating no association between sunscreen use and the development of CMM.CONCLUSIONS: The available epidemiological data do not support the existence of a relationship between topical sunscreen use and an increased risk of CMM.     

Assessing intervention effects in a school-based nutrition intervention trial: which analytic model is most powerful?

This article compares four mixed-model analyses valid for group-randomized trials (GRTs) involving a nested cohort design with a single pretest and posttest. This study makes estimates of intraclass correlations (ICCs) available to investigators planning GRTs addressing dietary outcomes. It also provides formulae demonstrating the potential benefits to the standard error of the intervention effect (sigma(delta)) from adjustments for both fixed and time-varying covariates and correlations over time. These estimates will allow other researchers using these variables to plan their studies by estimating a priori detectable differences and sample size requirements for any of the four analytic options. These methods are demonstrated using data from the Teens Eating for Energy and Nutrition at School study. Mixed-model analyses of covariance proved to be the most powerful analysis in that data set. The formulae may be applied to any dependent variable in any GRT given corresponding information for those variables on the parameters that define the formulae.     

Dietary supplement use and mortality in a cohort of Swedish men

The use of dietary supplements has increased substantially in most industrialized countries. The aim of this study was to prospectively examine the association between use of dietary supplements and all-cause mortality, cancer mortality and CVD mortality in men. We used the population-based prospective cohort of 38 994 men from central Sweden, 45-79 years of age, with no cancer or CVD at baseline and who completed a self-administered FFQ including questions on dietary supplement use and life-style factors in 1997. During average 7.7 years of follow-up, 3403 deaths were ascertained; among them, 771 due to cancer and 930 due to CVD (during 5.9 years of follow-up). In multivariate adjusted models including all men there was no association observed between use of any dietary supplement or of multivitamins, vitamin C, vitamin E or fish oil specifically and all-cause mortality, cancer or CVD mortality. Among current smokers, regular use of any supplement was associated with statistically significant increased risk of cancer mortality: relative risk (RR) 1.46 (95 % CI 1.06, 1.99). Among men reporting an inadequate diet at baseline (assessed by Recommended Food Score), there was a statistically significant inverse association between use of any dietary supplement and CVD mortality (RR 0.72; 95 % CI 0.57, 0.91), no associations were observed among men with adequate diets. In conclusion, we cannot exclude that the use of dietary supplements is harmful for smokers. On the other hand, among men with an insufficient diet, the use of supplements might be beneficial in reducing CVD mortality.     

Use of an interrupted time-series design to evaluate a cancer screening program

An alternative approach to intervention-control designs to evaluate community health education studies is to use a quasi-experimental design in which the outcomes of interest are examined over time in the intervention unit. The Forsyth County Cancer Screening Project (FoCaS) was a comprehensive clinic- and community-based education program to increase screening for cervical cancer and breast cancer among low-income women. This paper reports the use of piecewise regression accounting for potential effects of auto-correlation in the data to evaluate the effectiveness of the project in increasing mammography screening. Data for the evaluation of trends in screening consisted of all mammograms performed during the period of May 1992 through June 1995 at the Reynolds Health Center in Forsyth County, North Carolina. The results suggested that the FoCaS project was effective in increasing mammography screening among women age 40 or older in the study population. Analysis of the trends by age indicated that the program had differential effects on women age 40-49 and 50 or older. The results demonstrate that analyses of the type presented here can either complement or serve as an alternative to more traditional intervention-control analyses.     

Traits of persons who drink decaffeinated coffee

PURPOSE: Little is known about the traits of decaffeinated coffee drinkers, who are sometimes used to ascertain whether the health effects of coffee intake are due to caffeine or some other coffee ingredient. METHODS: We studied these traits in 12,467 persons who reported type of coffee consumed at health examinations; 36% drank caffeinated only, 13% drank decaffeinated only, 27% drank both types and 24% drank no coffee. RESULTS: Odds ratios estimated from logistic regression analyses revealed that compared with regular (caffeinated) coffee drinkers or abstainers, decaffeinated coffee drinkers were less likely to be heavy coffee drinkers, smokers, alcohol drinkers, users of caffeinated soft drinks and medication and to be free of illness. Increased decaffeinated coffee drinking was associated with older age, female sex, African American ethnicity, use of special diets and cardiovascular, gastrointestinal, or neuropsychiatric symptoms. Persons on special diets were more likely to drink decaffeinated coffee whether they had heart disease or were free of any illness. CONCLUSION: These data suggest that decaffeinated coffee use is related to illness in some persons but to a healthy lifestyle in others. These potential and possibly conflicting confounding factors need to be considered when studying the health effects of coffee or caffeine.     

Statistical heterogeneity in systematic reviews of clinical trials: a critical appraisal of guidelines and practice

OBJECTIVE: Heterogeneity between study results can be a problem in any systematic review or meta-analysis of clinical trials. Identifying its presence, investigating its cause and correctly accounting for it in analyses all involve difficult decisions for the researcher. Our objectives were: to collate recommendations on the subject of dealing with heterogeneity in systematic reviews of clinical trials; to investigate current practice in addressing heterogeneity in Cochrane reviews; and to compare current practice with recommendations. METHODS: We review guidelines for those undertaking systematic reviews and examine how heterogeneity is addressed in practice in a sample of systematic reviews, and their protocols, from the Cochrane Database of Systematic Reviews. RESULTS: Advice to reviewers is on the whole consistent and sensible. However, examination of a sample of Cochrane protocols and reviews demonstrates that the advice is difficult to follow given the small numbers of studies identified in many systematic reviews, the difficulty of pre-specifying important effect modifiers for subgroup analysis or meta-regression and the unresolved debate concerning fixed versus random effects meta-analyses. There was disagreement between protocols and reviews, often either regarding choice of important potential effect modifiers or due to the review identifying too few studies to perform planned analyses. CONCLUSION: Guidelines that address practical issues are required to reduce the risk of spurious findings from investigations of heterogeneity. This may involve discouraging statistical investigations such as subgroup analyses and meta-regression, rather than simply adopting a cautious approach to their interpretation, unless a large number of studies is available. The notion of a priori specification of potential effect modifiers for a retrospective review of studies is ill-defined, and the appropriateness of using a statistical test for heterogeneity to decide between analysis strategies is suspect.