A longitudinal evaluation of an educational software program: a case study of Urinalysis-Tutor.

PURPOSE: To examine students' learning before and after revising an educational software program and to explore students' patterns of use of an interactive feature that compares images. METHOD: Study participants were 466 University of Washington School of Medicine students. Two cohorts of students (one in 1996 and one in 1997) used the original version of the software. Following analysis of the students' learning, the software program was modified based on instructional design principles pertaining to visual learning and concept acquisition. A 1998 cohort of students used the revised program and their performance was compared with that of the 1996 cohort. Analyses were based on pre- and post-test scores, data collected from the observation of students, and navigational pathways tracked by the program. RESULTS: There was very little difference in the overall performances of the students who used the original program and those who used the revised program. Error analysis focusing on 11 conceptual areas showed that reductions in errors occurred for six of 11 concepts, with statistically significant reductions of errors for two concepts. Additional navigational data collected in 1998 showed that students used an interactive feature for comparing images in different patterns. The data showed a positive association between performance and the anchored viewing mode of image display. CONCLUSIONS: While this study cannot point to specific design components that facilitated or hindered learning, it demonstrated a potential benefit of linking usage-pattern data and performance. Future studies should evaluate design factors that affect usage patterns and performances based on navigational data collected while students interact with software programs.