Multirater agreement for grading the femoral and tibial cartilage surface lesions at CT arthrography and analysis of causes of disagreement

ObjectiveTo assess the multirater agreement of the modified Outerbridge system for the grading of predefined areas of femorotibial cartilage at CT arthrography with multiple readers, with varying experience.DesignFive readers with varying experience (two junior radiologists, three musculoskeletal radiologists including two experts in cartilage imaging) separately analyzed 962 cartilage sectors from pre-divided knee CT arthrograms with femorotibial osteoarthritis (Kellgren/Lawrence = 3). Each cartilage area was graded twice by each reader, at a three-month interval, according to the modified 5-grade Outerbridge system. Interobserver and intraobserver agreement were assessed. After the second reading, 121 areas exhibiting the highest interobserver disagreement were reviewed in consensus to determine the sources of disagreement.ResultsThe global interobserver agreement was fair (k = 0.35), and increased with the grade (from k = 0.14 to k = 0.76 from grade 0–4). The intraobserver agreement varied with the readers’ experience from moderate (k = 0.59) to almost perfect (k = 0.92). The majority of cases of disagreement (44%) was due to difficulties in assessing the normal variations of cartilage thickness, including diffuse cartilage thinning (23%) and normal variants of cartilage thickness (22%). 32% of cases of disagreement were due to retrospectively avoidable interpretation errors.ConclusionsThe multirater agreement of the modified Outerbridge system is only fair when readers of different level of experience are taken into account, and interobserver agreement increases with readers’ experience. However, interobserver agreement is substantial for grade 4 lesions. We report normal variations of cartilage thickness that may improve observer agreement in reporting cartilage lesions.