Controlling for occasion-specific effects when assessing the test–retest reliability of self-report health questionnaires

OBJECTIVE: This study proposes a method for self-report health questionnaires to adjust test-retest reliability for changes during the test-retest interval based on an external measure, and to distinguish such changes from random response errors. METHODS: In our application, eighty participants completed the Symptoms of Illness Checklist (SIC) on two occasions, two weeks apart, immediately before interviews given on each occasion by one of two physicians in a crossover design. The physician interview scores served as external measures, and structural equation modeling was used to estimate the parameters of a model that corrected for the occasion-specific effect of participants' responses using information from the interviews. RESULTS: Correcting for changes in symptoms during the test-retest interval increased SIC test-retest reliability from .744 to .804 and significantly improved model fit (chi2(diff)(1) = 30.78, p < .001). CONCLUSIONS: The results suggest methods that can improve the evaluation of self-report health questionnaire test-retest reliability by identifying changes using an external measure, and distinguishing these from random response errors; these increased the estimated SIC test-retest reliability and indicated that the SIC was indeed able to measure changes over the studied time interval. This method can be applied across a broad range of questionnaires.