Development and Validation of Assessment Measures for a Newly Developed Physical Examination Simulator

Objective: Define, extract and evaluate potential performance indicators from computer-generated data collected during simulated clinical female pelvic examinations.Design: Qualitative and quantitative study analyzing computer generated simulator data and written clinical assessments collected from medical students who performed physical examinations on three clinically different pelvic simulators.Setting: Introduction to patient care course at a major United States medical school.Participants: Seventy-three pre-clinical medical students performed 219 simulated pelvic examinations and generated 219 written clinical assessments.Measurements: Cronbach’s alpha for the newly defined performance indicators, Pearson’s correlation of performance indicators with scored written clinical assessments of simulator findings.Results: Four novel performance indicators were defined: time to perform a complete examination, number of critical areas touched during the exam, the maximum pressure used, and the frequency at which these areas were touched. The reliability coefficients (alpha) were time = 0.7240, critical areas = 0.6329, maximum pressure = 0.7701, and frequency = 0.5011. Of the four indicators, three correlated positively and significantly with the written clinical assessment scores: critical areas, p < 0.01; frequency, p < 0.05; and maximum pressure, p < 0.05.Conclusion: This study demonstrates a novel method of analyzing raw numerical data generated from a newly developed patient simulator; deriving performance indicators from computer generated simulator data; and assessing validity of those indicators by comparing them with written assessment scores. Results show the new assessment measures provide an objective, reliable, and valid method of assessing students’ physical examination techniques on the pelvic exam simulator.Objective assessment of clinical and technical skills is now possible with simulation and virtual reality technologies.^1,2Virtual reality simulators such as the Minimally Invasive Surgery Trainer-Virtual Reality (MIST-VR), the Mentice Shoulder Arthroscopy Simulator, and the Endoscopic Sinus Surgery (ESS) Simulator have been developed to provide trainees with practice performing surgical procedures and immediate feedback on their performance.^3–6The simulator scoring systems or “internal metrics” capture user performance data and convert this information into scores, using variables such as “time” to complete the task, number of “collisions”, and “path length” to intended target. The emerging field of simulator development raises challenging research questions about the psychometric properties of simulators, including how much and what kind of data to collect as well as how to record and report the scores so they provide useful feedback for students and trainees. Performance data generated from these novel teaching and assessment tools have the potential to revolutionize skills assessments by providing more objective and reliable assessment measures than those most commonly used in medical training today.Development of the E-Pelvis, a novel physical examination simulator, has afforded the opportunity to define and validate assessment measures that have never been used before in evaluating clinicians’ technical skills. The purpose of this research project was to demonstrate concurrent validity of the simulator by comparing computer generated data collected during simulated pelvic exams with students’ written assessments of the clinical findings on three simulators.