Model Performance Evaluation (Validation and Calibration) in Model-based Studies of Therapeutic Interventions for Cardiovascular Diseases

Decision analytic models play an increasingly important role in the economic evaluation of health technologies. Given uncertainties around the assumptions used to develop such models, several guidelines have been published to identify and assess ‘best practice’ in the model development process, including general modelling approach (e.g., time horizon), model structure, input data and model performance evaluation. This paper focuses on model performance evaluation. In the absence of a sufficient level of detail around model performance evaluation, concerns regarding the accuracy of model outputs, and hence the credibility of such models, are frequently raised. Following presentation of its components, a review of the application and reporting of model performance evaluation is presented. Taking cardiovascular disease as an illustrative example, the review investigates the use of face validity, internal validity, external validity, and cross model validity. As a part of the performance evaluation process, model calibration is also discussed and its use in applied studies investigated. The review found that the application and reporting of model performance evaluation across 81 studies of treatment for cardiovascular disease was variable. Cross-model validation was reported in 55 % of the reviewed studies, though the level of detail provided varied considerably. We found that very few studies documented other types of validity, and only 6 % of the reviewed articles reported a calibration process. Considering the above findings, we propose a comprehensive model performance evaluation framework (checklist), informed by a review of best-practice guidelines. This framework provides a basis for more accurate and consistent documentation of model performance evaluation. This will improve the peer review process and the comparability of modelling studies. Recognising the fundamental role of decision analytic models in informing public funding decisions, the proposed framework should usefully inform guidelines for preparing submissions to reimbursement bodies.     

Low quality of dying and death in patients with septic shock as perceived by nurses and resident physicians

Septic shock is a disease with both high prevalence and mortality. Few studies have evaluated the quality of dying and death (QODD) in patients with septic shock. The authors compared the QODD of patients who died of septic shock versus other causes. They prospectively collected QODD surveys from nurses and residents caring for 196 patients who died in the medical intensive care unit (ICU) at an urban, university hospital. Patients were included in the analysis if either a nurse or resident returned a survey. Chart review established cause of death. The authors compared total QODD scores (on a scale of 0–100) and a single-item score (QODD-1; on a scale of 0–10) of patients who died of septic shock versus other causes. Survey response rates were 59% (n = 155) for residents and 49% (n = 129) for nurses. Nurses rated patients as having lower total QODD and QODD-1 scores for septic (Δ 7.5 points, p = 0.03, and 0.9 points, p = 0.05, respectively). Residents rated septic patients with lower QODD-1 scores than nonseptic patients (Δ 0.8 points, p = 0.03). This study shows that nurses rate patients with septic shock as having lower QODD than patients dying of other causes. These findings are important for clinicians who counsel families of patients dying of septic shock.     

Female Genital Cosmetic Surgery

ObjectiveTo provide Canadian gynaecologists with evidence-based direction for female genital cosmetic surgery in response to increasing requests for, and availability of, vaginal and vulvar surgeries that fall well outside the traditional realm of medically-indicated reconstructions.EvidencePublished literature was retrieved through searches of PubMed or MEDLINE, CINAHL, and The Cochrane Library in 2011 and 2012 using appropriate controlled vocabulary and key words (female genital cosmetic surgery). Results were restricted to systematic reviews, randomized control trials/controlled clinical trials, and observational studies. There were no date or language restrictions. Searches were updated on a regular basis and incorporated in the guideline to May 2012. Grey (unpublished) literature was identified through searching the websites of health technology assessment and health technology-related agencies, clinical practice guideline collections, clinical trial registries, and national and international medical specialty societies.ValuesThe quality of evidence in this document was rated using the criteria described in the Report of the Canadian Task Force on Preventive Health Care (Table).     

Predicting Risk-Taking With and Without Substance Use: The Effects of Parental Monitoring, School Bonding, and Sports Participation

Risk-taking is statistically normative during adolescence, yet is associated with adverse outcomes including substance use. The present study draws the distinction between protective factors (effective for those identified as high risk takers) and promotive factors (effective for all) against substance use, focusing on parental monitoring, school bonding, and sports participation. A total of 36,514 8th and 10th grade participants in the national Monitoring the Future study were included. Although parental monitoring was associated with lower alcohol and marijuana use among all adolescents (i.e., promotive effect), these effects were strongest among the highest risk takers (i.e., protective effect) and females. School bonding was associated with lower levels of both alcohol and marijuana use among all groups of adolescents, but these promotive effects were weak. Sports participation was associated with higher levels of alcohol use among all males and among 8th grade females who did not identify as high risk takers. Despite being a risk factor for alcohol use, sports participation did demonstrate a promotive effect against marijuana use among 10th grade females only, and especially so for high risk-taking females (i.e., protective effect). Overall, these findings suggest that of the three mechanisms studied, parental monitoring emerged as the most promising entry point for substance use prevention and intervention across groups, particularly for females and high risk-taking adolescents.