Do different physicians' panels reach similar conclusions?: A case study on practice guidelines for limited surgery in breast cancer

This study was aimed at assessing the extent to which differentpanels of physicians produce similar recommendations and whethertheir agreement is greater when members of the panels practicein the same geographical area. A total of 34 physicians, organizedin 3 multispeciality and one monospeciality (all surgeons) panels,working at both specialized and community hospitals, participatedin the study. They were asked to rate the appropriateness ofan indication for limited surgery in breast cancer for 432 hypotheticalpatients described through clinical scenarios. The primary outcomeof the study was the reliability of appropriateness ratingsbetween the pairs of panels, measured as the percent absoluteagreement and kappa statistic. The agreement of the ratingsbetween pairs of panel of the same region was always quantitativelygood - panels I-II kappa = 0.58 (95% Cl: 0.50–0.67) andpanels III-IV kappa = 0.65 (95% Cl: 0.56–0.75) - and higherthan when panels from different regions were compared. We concludethat in our study an acceptable level of agreement was reachedwhen different panels of experts were asked to produce guidelinesusing a structured process which includes exposure to the relevantscientific literature. The fact that local (organizational,cultural, etc.) factors seemed to play a modulating role overscientific evidence has implications that should be consideredwhen deciding on the level (local versus central) where guidelinesare produced.