Open-book Tests to Complement Assessment-programmes: Analysis of Open and Closed-book Tests

Today’s health sciences educational programmes have to deal with a growing and changing amount of knowledge. It is becoming increasingly important for students to be able to use and manage knowledge. We suggest incorporating open-book tests in assessment programmes to meet these changes. This view on the use of open-book tests is discussed and the influence on test quality is examined. To cope with the growing amount of medical knowledge, we have divided the body of knowledge into core knowledge, which students must know without need for references, and backup knowledge, which students need to understand and use properly with the help of references if so desired. As a result, all tests consist of a subtest for reproduction and understanding of core knowledge (a closed-book test) and a subtest for the ability to understand and manage backup knowledge (an open-book test). Statistical data from 14 such double-subtest exams for first and second-year students were analyzed for two cohorts (N = 435 and N = 449) with multilevel analysis, in accordance with generalizability theory. The reliability of the open and closed-book sections of the separate tests varied between 0.712 and 0.850. The open-book items reduce reliability somewhat. The estimated disattenuated correlation was 0.960 and 0.937 for cohorts 1 and 2 respectively. It is concluded that the use of open-book items with closed-book items slightly decreases test reliability but the overall index is acceptable. In addition, open and closed-book sections are strongly positively related. Therefore, open-book tests could be helpful in complementing today’s assessment programmes.     

Comparing the long-term retention of a physiology course for medical students with the traditional and problem-based learning

The rapid improvements in medical sciences and the ever-increasing related data, however, require novel methods of instruction. One such method, which has been given less than due attention in Iran, is problem-based learning (PBL). In this study, we aimed to evaluate the impact of study skills and the PBL methods on short and long-term retention of information provided for medical students in the course of respiratory physiology and compare it with traditional learning method. In this study, 39 medical students from Medical School of Kerman University of Medical Sciences, Kerman, Iran (2006–2010) were enrolled in the study and allocated randomly in three equal groups (13 in each group). All groups underwent a pre-test to be assessed for their basic information regarding respiratory physiology. Two groups were instructed using the traditional method, and one group used PBL. Among the two groups of the traditional method, one was instructed about study skills and the other was not. Once the PBL group took the study skill workshop, they were aided by tutors for their education. In the final term test, those students who had learned study skills and were instructed with the traditional method scored higher compared to other groups (p < 0.05). However, in the 1 year (p < 0.05) and 4 year (p < 0.01) interval examinations, the PBL group achieved significantly higher scores. Despite the fact that PBL had no positive effect on the final term exam of our students, it yielded a more profound and retained understanding of the subject course. Moreover, considering the positive effect of study skills on long-term student scores, we recommend students to receive instructions regarding the appropriate study skills when initiated into universities.     

Differential Effects of Two Types of Formative Assessment in Predicting Performance of First-year Medical Students

Formative assessments are systematically designed instructional interventions to assess and provide feedback on students’ strengths and weaknesses in the course of teaching and learning. Despite their known benefits to student attitudes and learning, medical school curricula have been slow to integrate such assessments into the curriculum. This study investigates how performance on two different modes of formative assessment relate to each other and to performance on summative assessments in an integrated, medical-school environment. Two types of formative assessment were administered to 146 first-year medical students each week over 8 weeks: a timed, closed-book component to assess factual recall and image recognition, and an un-timed, open-book component to assess higher order reasoning including the ability to identify and access appropriate resources and to integrate and apply knowledge. Analogous summative assessments were administered in the ninth week. Models relating formative and summative assessment performance were tested using Structural Equation Modeling. Two latent variables underlying achievement on formative and summative assessments could be identified; a “formative-assessment factor” and a “summative-assessment factor,” with the former predicting the latter. A latent variable underlying achievement on open-book formative assessments was highly predictive of achievement on both open- and closed-book summative assessments, whereas a latent variable underlying closed-book assessments only predicted performance on the closed-book summative assessment. Formative assessments can be used as effective predictive tools of summative performance in medical school. Open-book, un-timed assessments of higher order processes appeared to be better predictors of overall summative performance than closed-book, timed assessments of factual recall and image recognition. This research was presented at the 86th Annual Meeting of the American Educational Research Association (AERA) in Montreal, Canada, April 11–15, 2005.     

Validation of a performance assessment instrument in problem-based learning tutorials using two cohorts of medical students

Although problem-based learning (PBL) has been widely used in medical schools, few studies have attended to the assessment of PBL processes using validated instruments. This study examined reliability and validity for an instrument assessing PBL performance in four domains: Problem Solving, Use of Information, Group Process, and Professionalism. Two cohorts of medical students (N = 310) participated in the study, with 2 years of PBL evaluation data extracted from archive rated by a total of 158 faculty raters. Analyses based on generalizability theory were conducted for reliability examination. Validity was examined through following the Standards for Educational and Psychological Testing to evaluate content validity, response processes, construct validity, predictive validity, and the relationship to the variable of training. For construct validity, correlations of PBL scores with six other outcome measures were examined, including Medical College Admission Test, United States Medical Licensing Examination (USMLE) Step 1, National Board of Medical Examiners (NBME) Comprehensive Basic Science Examination, NBME Comprehensive Clinical Science Examination, Clinical Performance Examination, and USMLE Step 2 Clinical Knowledge. Predictive validity was examined by using PBL scores to predict five medical school outcomes. The highest percentage of PBL total score variance was associated with students (60 %), indicating students in the study differed in their PBL performance. The generalizability and dependability coefficients were moderately high (Ep² = .68, ? = .60), showing the instrument is reliable for ranking students and identifying competent PBL performers. The patterns of correlations between PBL domain scores and the outcome measures partially support construct validity. PBL performance ratings as a whole significantly (p < .01) predicted all the major medical school achievements. The second year PBL scores were significantly higher than those of the first year, indicating a training effect. Psychometric findings provided support for reliability and many aspects of validity of PBL performance assessment using the instrument.     

Introducing a problem-based unit into a lifespan nutrition class using a randomized design produces equivocal outcomes

Problem-based learning (PBL) is a tutorial, student-centered, problem-driven educational strategy adopted by medical and allied health educators to positively influence self-directed learning, critical thinking, and learning behavior. PBL was examined in dietetics education through random assignment of 32 undergraduate dietetics students for two weeks to either a problem-based or a lecture-based case format for the infant and elderly units of a 16-week lifespan nutrition course. Random assignment followed stratification for gender and earlier course examination performance. Expert validation of PBL curricular components and noted differences in discussion structure and information resources verified curriculum distinctiveness. Main outcome measures were pre- and post-Cognitive Behavior Survey scores for memorization, reflection, and positive learning experience scales, unit and course evaluations and unit examination scores. Students in problem-based modules demonstrated greater gains in reflective thinking with stable memorization, suggesting improved critical thinking skills. Tenets that problem-based learning promotes knowledge retention and provides a more positive learning experience were not upheld. Knowledge acquisition was not hindered by a problem-based approach. We conclude that gains in reflective thinking and evidence of increased self-directed learning argue for inclusion of PBL in dietetics curricula and that a problem-based education will help dietitians successfully respond to professional development needs.     

Implementation of problem-based learning in Asian medical schools and students' perceptions of their experience

BACKGROUND: Since the introduction of problem-based learning (PBL) at McMaster University in 1969, many medical schools in the USA, Canada and Europe have included PBL in their curricula. In the past decade, many medical schools in Asia have also done so. However, so far no one has questioned whether the outcomes expected of the learner in a PBL setting are applicable to students from different cultural upbringings. AIM: The aim of this study was to investigate the implementation of PBL in Asian medical schools, their students' perceptions of this new mode of teaching/learning and how the problems that have arisen may be overcome. METHOD: Published reports and conference presentations were gathered on the implementation of PBL in some Asian medical schools and comparisons of the experiences in PBL of Asian and students of other ethnic backgrounds. RESULTS: Most Asian medical schools and their students appear to be positive about adapting to PBL in their curriculum. The positive and negative observations appear to be similar to those experienced in non-Asian medical schools. The problems that arose for students in Asian medical schools in the early stages of implementing PBL appear to have been overcome after a period of adjustment with the help of supportive and open-minded tutors. The reports also suggested that trigger problems should be carefully designed to make them relevant and interesting for the students. CONCLUSION: Strong support from the academic administrators (dean and other staff responsible for implementation of the curriculum) in the introduction of PBL into the curriculum and careful training of both faculty and students appear to be key factors to ensure the successful implementation of PBL in Asian medical schools.     

Faculty opinions about a revised pre-clinical curriculum

INTRODUCTION: Medical schools having innovative curricula have been encouraged to ascertain the levels of satisfaction of faculty members with the curriculum. Faculty at schools that employ problem-based learning (PBL) have been shown to have positive perceptions, but not all schools are in a position to adopt PBL on a large scale. This study sought to determine faculty members' opinions about a new curriculum that is less ambitious than one utilizing true PBL. CONTEXT AND SETTING: Since 1997, the University of Otago Medical School (Dunedin, New Zealand) has had an integrated, modular pre-clinical curriculum that emphasizes clinical relevance. It has proved popular with students. This study focused on faculty members' impressions. METHODS: We surveyed faculty members' opinions with a questionnaire identical to one used in studies at PBL schools. Faculty compared the students and their own levels of satisfaction in the old and new curricula on 7 to 10 items. The overall response rate was 85.4% (152 of 178). RESULTS: Perceptions of the new curriculum were positive among teachers who taught during the pre-clinical years and those who taught the students only after they reached the clinical years. Results for individual questions were in the same direction and generally similar in magnitude to those reported on identical items for PBL. CONCLUSION: We conclude that a hybrid curriculum that is more acceptable to many traditional teachers and students than is PBL has almost as great a positive effect on faculty members' perceptions of students' abilities and of the curriculum as does PBL.     

Comparison of the Effects of Stable and Dynamic Furniture on Physical Activity and Learning in Children

We compared the effects of traditional (stable) and non-traditional (dynamic) school furniture on children’s physical activity (PA), energy expenditure (EE), information retention, and math skills. Participants were 12 students (8.3 years, 58 % boys) in grades 1–5. Participants wore an Actigraph GT3X+ accelerometer (to assess PA), and an Oxycon Mobile indirect calorimetry device (to assess EE) for 40 min (20 min for each session). Each session consisted of a nutrition lecture, multiple choice questions related to the lecture, and grade-appropriate math problems. We used paired t tests to examine differences between the stable and dynamic furniture conditions. Average activity counts were significantly greater in the dynamic than the stable furniture condition (40.82 vs. 9.81, p < 0.05). We found no significant differences between conditions for average oxygen uptake (p = 0.34), percentage of nutrition questions (p = 0.5), or math problems (p = 0.93) answered correctly. Movement was significantly greater in the dynamic than the stable furniture condition, and did not impede information acquisition or concentration. Future studies should compare the long-term effects of traditional and dynamic furniture on health and academic outcomes in schools and other settings.     

Knowing How and Knowing Why: testing the effect of instruction designed for cognitive integration on procedural skills transfer

Transfer is a desired outcome of simulation-based training, yet evidence for how instructional design features promote transfer is lacking. In clinical reasoning, transfer is improved when trainees experience instruction integrating basic science explanations with clinical signs and symptoms. To test whether integrated instruction has similar effects in procedural skills (i.e., psychomotor skills) training, we studied the impact of instruction that integrates conceptual (why) and procedural (how) knowledge on the retention and transfer of simulation-based lumbar puncture (LP) skill. Medical students (N = 30) were randomized into two groups that accessed different instructional videos during a 60-min self-regulated training session. An unintegrated video provided procedural How instruction via step-by-step demonstrations of LP, and an integrated video provided the same How instruction with integrated conceptual Why explanations (e.g., anatomy) for key steps. Two blinded raters scored post-test, retention, and transfer performances using a global rating scale. Participants also completed written procedural and conceptual knowledge tests. We used simple mediation regression analyses to assess the total and indirect effects (mediated by conceptual knowledge) of integrated instruction on retention and transfer. Integrated instruction was associated with improved conceptual (p < .001) but not procedural knowledge test scores (p = .11). We found no total effect of group (p > .05). We did find a positive indirect group effect on skill retention (B _ab  = .93, p < .05) and transfer (B _ab  = .59, p < .05), mediated through participants improved conceptual knowledge. Integrated instruction may improve trainees’ skill retention and transfer through gains in conceptual knowledge. Such integrated instruction may be an instructional design feature for simulation-based training aimed at improving transfer outcomes.     

Problem-based learning in medical education: Developing a research agenda

While the use of problem-based learning (PBL) methods continues to increase in medical education, three literature reviews of PBL have appeared in the past several years which come to different opinions about their merits. This analysis summarizes the research evidence regarding PBL by examining how well it has met its originators' goals, what we know about how PBL works, and how PBL fares in a goal-free comparison with conventional curricula. A research agenda is suggested to refine our understanding of well-documented effects of PBL, to probe for other possible longer term PBL outcomes, and to examine if and how PBL affects knowledge acquisition and retention. Consistency of evidence from a variety of PBL implementations can help decide whether the effects seen can be attributed to PBL or are the results of other curricular features unique to one setting.     

Influence of national culture on the adoption of integrated and problem-based curricula in Europe

Context  There is an evident misbalance in the frequency of medical schools with problem-based learning (PBL) curricula in northern versus southern Europe. This study explores the hypothesis that national culture influences the flexibility of (medical) schools in terms of their propensity to adopt integrated and PBL curricula.
Methods  National culture was defined by a country's scores on indexes for 4 dimensions of culture as described by Hofstede, defined as: power distance ; individualism/collectivism ; masculinity/femininity , and uncertainty avoidance . Non-integrated medical curricula were defined as those that included courses in 2 of the 3 basic sciences (anatomy, biochemistry and physiology) in the first 2 years; otherwise, by exclusion, curricula were assumed to be integrated. The medical curricula of 134 of the 263 schools in the 17 European countries included in Hofstede's study were examined.
Results  Correlations were calculated between the percentage of integrated medical curricula in a country and that country's scores on indexes for each of the 4 dimensions of culture. Significant negative correlations were found between the percentage of integrated curricula and scores on the power distance index (correlation coefficient [CC]: − 0.692; P  =   0.002) and the uncertainty avoidance index (CC: − 0.704; P  =   0.002). No significant correlations were found between the percentage of integrated curricula and scores on the indexes for individualism/collectivism and masculinity/femininity .
Conclusions  A (medical) school which is considering adopting an integrated or PBL curriculum and which is based in a country with a high score on Hofstede's power distance index and/or uncertainty avoidance index must a priori design strategies to reduce or overcome the obstructive effects of these dimensions of culture on the school's organisation.     

The effect of problem-based learning on the academic performance of students studying podiatric medicine in Ontario

OBJECTIVES: To investigate the effect of problem-based learning (PBL) on student achievement. DESIGN: Two cohorts of students studying podiatric medicine in Ontario were compared - one having undertaken the traditional lecture-based curriculum, and the other the problem-based learning programme. SETTING: The Michener Institute for Applied Health Sciences and The Toronto Hospital. SUBJECTS: Chiropody students. RESULTS: The performance of the students on the written Provincial Registration Examinations of Ontario was analysed utilizing independent t-tests and demonstrated that the PBL cohort of students achieved significantly higher overall examination scores (P < 0.005) than the traditional cohort. Further analysis revealed that no significant difference existed between student cohorts with respect to factual biomedical knowledge (P > 0.5), but that the PBL students performed significantly better in tests of deeper understanding and the cognitive skills related to patient management (P < 0.0005). Additionally, intragroup analysis using Spearman's rank order correlation indicated that there was no rank ordered association between performance on the multiple-choice and essay sections of the examination. CONCLUSIONS: The above results suggest that the PBL cohort of students was more knowledgeable than, and possessed superior cognitively related patient management skills, to their traditional counterparts.     

Evolutionary trends of problem-based learning practices throughout a two-year preclinical program: a comparison of students’ and teachers’ perceptions

Implementation of a pedagogical approach is a continuous and evolving process. As an institution with more than 15 years problem-based learning (PBL), we studied how the learning and teaching processes are currently practiced in a 2-year preclinical basic sciences program to assess whether they still match the intended objectives. Using both students’ and tutors’ evaluations, we analyzed and compared their perceptions on the program content and its organization, on tutors’ functioning and on the duration of PBL sessions throughout 11 instructional units of the second and third-year of a 6 years medical curriculum. Whereas both tutors and students indicated that the content and problems selected for the curriculum were well adapted to the PBL process, they felt the references as well as the self-study time as moderately appropriate. Over the course of the 2-year program, tutorial sessions got linearly shorter, whereas reporting sessions got longer. While tutors knew well the PBL approach and were suitably prepared to their tutorials, they however, became less regular in providing feedback and in evaluating group functioning over the 2 years. Our results suggest that the practice of the PBL process evolves within and throughout a 2-year preclinical program and moves away from the original intentions. Possible underlying reasons and their implications are discussed within the context of tutors’ and students’ concepts of teaching and learning, the medical schools’ learning environment and teaching practices and the difficulty of developing and maintaining in the long term a deep and self-directed learning approach.     

Problem-based learning: a case study in providing e-health education using the Internet

The current trend among many universities is to increase the number of courses available online. However, there are fundamental problems in transferring traditional education courses to virtual formats. Delivering current curricula in an online format does not assist in overcoming the negative effects on student motivation which are inherent in providing information passively. Using problem-based learning (PBL) online is a method by which computers can become a tool to encourage active learning among students. The delivery of curricula via goal-based scenarios allows students to learn at different rates and can successfully shift online learning from memorization to discovery. This paper reports on a Web-based e-health course that has been delivered via PBL for the past 12 months. Thirty distance-learning students undertook postgraduate courses in e-health delivered via the Internet (asynchronous communication). Data collected via online student surveys indicated that the PBL format was both flexible and interesting. PBL has the potential to increase the quality of the educational experience of students in online environments.     

Development of knowledge in basic sciences: a comparison of two medical curricula

Medical Education 2012: 46 : 1206–1214 Context Basic medical sciences education differs among medicine courses, especially as traditional and integrated problem‐based learning (PBL) curricula teach basic sciences in very different ways. The literature shows no clear differences in the performance of students of these different educational philosophies. The Charité Medical University of Berlin (Charité Universitätsmedizin Berlin) teaches both a traditional medical curriculum (TMC) and a PBL reformed medical curriculum (RMC). Both curricula conduct the Progress Test in Medicine (PTM), which examines competence in the basic and clinical sciences from the first to the last semester. Objectives The aim of this study was to compare the development and retention of knowledge in the basic medical sciences between students on the traditional and reformed undergraduate medical curricula, respectively. Methods For each student and single PTM, relative frequencies of correct answers were computed for basic sciences items only and for the whole curriculum. Frequencies were averaged and grouped by semester and curriculum. Analyses of variance (anova s) were performed at all measurement points with a Bonferroni‐corrected p‐value at the level of p < 0.005. Eta‐squared (η²) was used to classify effect size. Results In the first three semesters, RMC students slightly outperform TMC students in the basic sciences, although TMC students receive more systematic teaching. After this, TMC students develop a peak of knowledge in basic sciences and overtake RMC students. The knowledge of TMC students then decreases over time, but despite this, they perform better in the final semester. Students on the RMC show constant progress throughout their undergraduate studies. Overall, the development of medical knowledge is consistent in both curricula. There is no significant difference in this outcome between the traditional and PBL courses. Conclusions Progress testing as a longitudinal method allows us to better understand the development of knowledge during formal undergraduate education. The main difference between traditional and problem‐based medical education seems to be provoked by the high‐stakes national examination undertaken in the traditional course (the Physikum).