Cardiovascular effects of sodium-glucose co-transporter-2 inhibitors and glucagon-like peptide-1 receptor agonists: The P value and beyond

Despite growing awareness of the dangers of a dichotomous interpretation of trial results based on the ‘statistical significance’ of a treatment effect, the uptake of new approaches has been slow in diabetes medicine. We showcase a number of ways to interpret the evidence for a treatment effect applied to the cardiovascular outcome trials of glucagon-like peptide-1 receptor agonists (GLP-1RAs) and sodium-glucose co-transporter-2 inhibitors (SGLT-2is): the P value function (or confidence curves), which depicts the treatment effect across the whole spectrum of confidence levels; the counternull value, which is the hazard ratio (i.e. treatment effect size) supported by the same amount of evidence as the null value (i.e. no treatment effect); and the S value, which quantifies the strength of the evidence against the null hypothesis in terms of the number of coin tosses yielding the same side. We show how this approach identifies potential treatment effects, highlights similarities among trials straddling the threshold of statistical significance, and quantifies differences in the strength of the evidence from trials reporting statistically significant results. For example, while REWIND, CANVAS and CREDENCE failed to reach statistical significance at the .05 level for all-cause mortality, their counternull values indicate that reduced death rates by 19%, 24% and 31%, respectively, are supported by the same amount of evidence as that indicating no treatment effect. Moreover, similarities among results emerge in trials of GLP-1RAs (REWIND, EXSCEL and LEADER) lying closely around the threshold of ‘statistical significance’. Lastly, several S values, such as for the primary outcome in HARMONY Outcomes (S value 10.9) and all-cause death in EMPAREG-OUTCOME (S value 15.0), stand out compared with values for other outcomes and other trials, suggesting much larger differences in the evidence between these studies and several others that cluster around the .05 significance threshold. P value functions, counternull values and S values should complement the standard reporting of the treatment effect to help interpret clinical trials and make decisions among competing glucose-lowering medications.