The effectiveness of integration of virtual patients in a collaborative learning activity

Abstract

Background: Virtual patients (VPs) have been recently integrated within different learning activities.

Aim: To compare between the effect of using VPs in a collaborative learning activity and using VPs in an independent learning activity on students’ knowledge acquisition, retention and transfer.

Methods: For two different topics, respectively 82 and 76 dental students participated in teaching, learning and assessment sessions with VPs. Students from a female campus and from a male campus have been randomly assigned to condition (collaborative and independent), yielding four experimental groups. Each group received a lecture followed by a learning session using two VPs per topic. Students were administrated immediate and delayed written tests as well as transfer tests using two VPs to assess their knowledge in diagnosis and treatment.

Results: For the treatment items of the immediate and delayed written tests, females outperformed males in the collaborative VP group but not in the independent VP group.

Conclusion: On the female campus, the use of VPs in a collaborative learning activity is more effective than its use as an independent learning activity in enhancing students’ knowledge acquisition and retention. However, the collaborative use of VPs by itself is not enough to produce consistent results across different groups of students and attention should be given to all the factors that would affect students’ interaction.     

Using language of entrustable professional activities to define learning objectives of radiology clerkship: A modified Delphi study

Abstract

Background: A standard undergraduate radiology education is essential to prepare graduates for multidisciplinary clinical practice yet the literature lacks clear guidelines or consensus about the learning objectives of an optimal radiology clerkship.

Aim: To define a competency-based framework for undergraduate radiology education by using language of Entrustable Professional Activities (EPAs).

Methods: A modified Delphi method with three iterative rounds was used as an expert consensus approach. An online questionnaire with Likert scale was formulated incorporating EPAs and their components (competencies, assessment strategies, and supervision level) and distributed to 45 consultant radiologists following pilot study. Items reaching consensus were accepted and rest were resent in round 2. In round 3, a dichotomous scale was used for final approval and to see response stability.

Results: A final set of six EPAs with 87 competencies and respective assessment strategies, all aiming for ‘level 3a’ of supervision was identified. These include recommending cost effective appropriate imaging tests for common pathologies, obtaining informed consent for diagnostic contrast studies, basic interpretation and communication of common pathologies/emergencies on radiographs (chest, abdominal, and skeletal) and on CT brain.

Conclusion: This EPA framework for radiology clerkship is a first step towards a competency-based approach to undergraduate radiology training and assessment.     

Theoretical considerations on programmatic assessment

Abstract

Introduction: Programmatic assessment (PA) is an approach to assessment aimed at optimizing learning which continues to gain educational momentum. However, the theoretical underpinnings of PA have not been clearly described. An explanation of the theoretical underpinnings of PA will allow educators to gain a better understanding of this approach and, perhaps, facilitate its use and effective implementation. The purpose of this article is twofold: first, to describe salient theoretical perspectives on PA; second to examine how theory may help educators to develop effective PA programs, helping to overcome challenges around PA.

Results: We outline a number of learning theories that underpin key educational principles of PA: constructivist and social constructivist theory supporting meaning making, and longitudinality; cognitivist and cognitive development orientation scaffolding the practice of a continuous feedback process; theory of instructional design underpinning assessment as learning; self-determination theory (SDT), self-regulation learning theory (SRL), and principles of deliberate practice providing theoretical tenets for student agency and accountability.

Conclusion: The construction of a plausible and coherent link between key educational principles of PA and learning theories should enable educators to pose new and important inquiries, reflect on their assessment practices and help overcome future challenges in the development and implementation of PA in their programs.     

Incorporation and evaluation of serial concept maps for vertical integration and clinical reasoning in case-based learning tutorials: Perspectives of students beginning clinical medicine

Abstract

Background: Concept maps and case-based learning (CBL) are recognized and useful strategies to enhance undergraduate medical learning. However, research on the use of a mixed approach is limited.

Aims: To incorporate serial concept mapping (CM) into CBL tutorials, to explore students’ perspectives on the worth of the method to better understand patients’ problems and elicit diagnoses, and to assess the student’s learning.

Methods: We designed a four-phase method of CBL that incorporated serial mapping to assist students in the process of knowledge construction regarding the underlying principles of the patients’ present complaints, the recognition of disease patterns and the eliciting of diagnostic hypotheses. Students worked both individually and collaboratively. We used a questionnaire to explore the students’ perspectives of the method and a score system to assess end-of-course performance.

Results: The students perceived that serial CM was useful to integrate previous knowledge into new clinical information for case analysis and to elicit diagnoses. They also reported an increase in content-related knowledge. The end-of-course scores were high for most students.

Conclusions: Novice medical students perceived serial CM in CBL tutorials as an effective strategy for learning. End-of-course examination scores indicated that they improved case analysis and clinical reasoning skills.     

The roles of basic psychological needs,self-compassion,and self-efficacy in the development of mastery goals among medical students

Abstract

Aim: Competency-based medical education aims to foster mastery goals in learners. We examined medical students’ mastery approach (beneficial) and mastery avoidance (maladaptive) goals and their associations with students’ basic psychological needs, self-compassion, and self-efficacy.

Methods: This was a cross-sectional study employing an online questionnaire. Two hundred medical students in all four years of the medical program completed the questionnaire, containing measures of mastery goals, basic psychological needs (autonomy, competence, relatedness), self-compassion, and self-efficacy. Regression analyses were performed.

Results: Of the three basic psychological needs, the need for competence was significant in explaining both types of mastery goals. Self-efficacy and self-compassion were significant in explaining mastery approach and mastery avoidance goals, respectively.

Conclusions: Creating learning environments that are supportive of students’ need for competence, raising students’ awareness of the value of learning from mistakes in competency acquisition, and providing opportunities for students to experience self-efficacy may foster beneficial mastery approach goals in medical students.     

Influences on the implementation of Mini-CEX and DOPS for postgraduate medical trainees’ learning: A grounded theory study

Abstract

Introduction: In order for Mini-Clinical Evaluation Exercise (Mini-CEX) and Direct Observation of Procedural Skills (DOPS) to actually have a positive effect on trainees’ learning, the way in which the tools are implemented is of key importance. However, there are many factors influencing their implementation. In this study, we aim to develop a comprehensive model of such factors.

Methods: Using a constructivist grounded theory approach, we performed eight focus groups. Participants were postgraduate trainees and supervisors from three different specialties; all were experienced with Mini-CEX and/or DOPS. Data were analyzed for recurring themes, underlying concepts and their interactions using constant comparison.

Results: We developed a model demonstrating how the implementation of Mini-CEX and DOPS for trainees’ learning is influenced by 13 factors relating to four categories: organizational culture (e.g. value of teaching and feedback), work structure (e.g. time for Mini-CEX and DOPS, faculty development), instruments (e.g. content of assessment), and users (e.g. relationship between trainees and supervisors), and their interaction.

Conclusions: We developed a complex model of influencing factors relating to four categories. Consideration of this model might support successful implementation and trainees’ learning with Mini-CEX and DOPS.     

Learning,assessment and professional identity development in public health training

Background: Professional identity formation is important for new recruits to training programmes. The integration of the accumulation of knowledge and assessment is a key aspect in its acquisition. This study assessed this interaction in Public Health Training in one English region.

Methods: Semi-structured interviews were held with 15 registrars from the West Midlands Public Health Training Programme. Pre-interview questionnaires gathered background information. A thematic content analysis approach was taken.

Results: There was a lack of integration between academic and workplace learning, the professional examination process and professional identity development. Registrars considered sitting the examination and their workplace learning as two parallel processes. Passing the examination was considered a key part in the early development of a professional identity but this was replaced by the opinions of others by the third year of training. Having a Masters’ in Public Health was less important but played a different role in their perceived acceptance by the wider Public Health workforce.

Conclusion: The lack of integration between assessment and learning seemed to have a detrimental effect on professional identity development. A review of how these two aspects might combine in a more positive manner is needed.     

Teaching to varied disciplines and educational levels simultaneously: An innovative approach in a neonatal follow-up clinic

Purpose: Across various health conditions and geographic regions, there remains a dearth of clinicians with the expertise and confidence to identify and manage children with disabilities. At the front line of this crisis are clinician-educators, who are tasked with caring for these unique patients and with training the future workforce. Balancing patient care and clinical instruction responsibilities is particularly challenging when trainees of varied educational levels and specialties report simultaneously. The lack of a standard curriculum further compounds the clinician-educator’s teaching demands and threatens the consistency of trainees’ learning. Recognizing these challenges in their work in a neonatal follow-up clinic, the authors sought a solution through an established curriculum development process.

Materials and methods: A needs assessment survey was conducted to gauge medical trainees’ knowledge, skills, and experiences. Applying needs assessment findings, the authors developed a curriculum, which was administered online to several trainee cohorts just prior to rotations in the neonatal follow-up clinic.

Results: After completing the curriculum, trainees scored significantly higher on neonatal follow-up knowledge tests.

Conclusions: Providing advance exposure helped to ensure that trainees arrived with comparable basal knowledge, which served as a foundation for more advanced instruction. This curricular approach may be useful across teaching venues, especially those with multi-level or multi-discipline learners.     

How do formative objective structured clinical examinations drive learning? Analysis of residents’ perceptions

Introduction: Although several studies have explored the relationship between learning and written tests, little is understood about how performance-based examinations influence learning. The purpose of this study was to explore how a formative objective structured clinical examination (OSCE) drives learning.

Methods: We administered surveys to residents (n?=?35) at three time points to determine if and how an OSCE influenced their learning: before and immediately following the OSCE, and after the distribution of their results. Differences in quantitative responses between high- and low-performing residents and across time were compared using repeated-measures ANOVA. Thematic analysis was used to analyze narrative comments.

Results: Participants’ goals for the OSCE related to performance, mastery and feedback. Almost all participants reported that they had learned something from the OSCE (94%) and most participants generated learning goals after the OSCE (71%). High performers appeared to recognize the importance of content-related knowledge for scoring well before and after the OSCE, whereas low performers may have under-estimated its importance until after the examination.

Discussion: Participants viewed a formative OSCE as both a hurdle to overcome (assessment of learning) and an opportunity to learn (assessment for learning). Understanding how OSCEs influence study behavior can help guide the development of assessments that promote learning.     

Providing a liminal space: Threshold concepts for learning in palliative medicine

Abstract

Introduction: Experience in palliative medicine provides a beneficial learning opportunity for doctors-in-training. There is, however, a gap in understanding which aspects of learning are most useful, which are problematic and how learning can best be facilitated. This study addresses that gap using the ‘threshold concepts’ framework. Threshold concepts are critical points of learning, often unique to a discipline. The learning occurs within a transitional or ‘liminal’ space and has specific characteristics including being ‘troublesome’ and ‘transformative.’

Methods: A qualitative, exploratory study was carried using the threshold concept framework. Semi-structured focus group interviews were held with doctors-in-training who had undertaken a 6-month palliative medicine attachment. Data were analysed using a content analysis approach with deductive and inductive phases, in order to identify threshold concepts.

Results: Five threshold concepts were identified. Two of these, ‘emotional engagement’ and ‘communication management,’ displayed all the typical characteristics of threshold concepts. This learning was highly valued by participants, had not occurred elsewhere in training and continued to influence practice.

Conclusions: Specific threshold concepts were identified for doctors undertaking a palliative medicine placement. These highlights where specific supports are required for learning and can be used to inform curriculum design.     

Legitimisation,personalisation and maturation: Using the experiences of a compulsory mobile curriculum to reconceptualise mobile learning

Background: Smartphone use is well established in society, with increasing acceptance in many professional workplaces. Despite the growth in mobile resources, how students and teachers benefit from these devices remains under-researched.

Aims: An exploration of student and educators views on the impact of mobile learning re-sources on placement learning experiences as part of the Leeds ‘MBChB Mobile’ programme.

Methods: Focus groups incorporating visual methodologies were undertaken with students from each year group; semi-structured interviews were undertaken with clinical teaching staff, including those who experienced the mobile programme as students themselves.

Results: Four key themes emerged. ‘Maturity of learning’, related to the way in which senior students use resources in a more nuanced way than junior colleagues. ‘Learning differently’, identified ‘personalisation’ and ‘just in time’ opportunities that mobile resources afforded. ‘Learning legitimately’ was identified by students as key to ensuring they adopted appropriate behaviours in relation to mobile learning. Using mobile resources at undergraduate level was found to ‘change learning patterns for life’.

Conclusions: There is a need to further develop the educational theory behind using mobile resources for learning. The results here suggest that mobile technologies are shaping learning behaviours, and are an indicator of learning maturity, reflecting the wider context of societal enculturation.     

Community of practice and student interaction at an acute medical ward: An ethnographic study

Background: A deeper understanding is needed of the acute medical care setting as a learning environment for students.

Aim: To explore workplace culture of an acute medical ward and students’ interactions within this community.

Method: An ethnographic design was applied. Medical and nurse students’ interactions were observed and informal questioning performed. Field notes were transcribed and analysed qualitatively, inspired by Wengers’ “Community of practice” theory.

Results: We identified four characteristics that regulated how students adapt and interact in the community of practice. Complex and stressful situations were stabilized by routines and carriers of culture. Variable composition and roles of community members were a part of the daily routine but did not seam obvious to students. Transitions through community boundaries were confusing especially for new students. Levels of importance and priority: Hierarchies and orders of priority were present as regulators of roles, routines and interactions, and of how staff approach different patient groups.

Conclusion: The culture shaped a pattern for, and created prerequisites that challenged students’ adaptation and created a space for learning. Students’ task on arrival was to enter the semipermeable membrane of the community of practice and to understand and adapt to its culture, and try to become accepted.     

Embedding of the progress test in an assessment program designed according to the principles of programmatic assessment

Background: Progress tests (PT) are used to assess students on topics from all medical disciplines. Progress testing is usually one of the assessment methods of the cognitive domain. There is limited knowledge on how positioning of the PT in a program of assessment (PoA) influences students’ PT scores, use of PT feedback and perceived learning value.

Methods: We compared PT total scores and use of a PT test feedback (ProF) system in two medical courses, where the PT is either used as a summative assessment or embedded in a comprehensive PoA and used formatively. In addition, an interview study was used to explore the students’ perception on use of PT feedback and perceived learning value.

Results: PT total scores were higher, with considerable effect sizes (ESs) and students made more use of ProF when the PT was embedded in a comprehensive PoA. Analysis of feedback in the portfolio stimulated students to look for patterns in PT results, link the PT to other assessment results, follow-up on learning objectives, and integrate the PT in their learning for the entire PoA.

Conclusions: Embedding the PT in an assessment program designed according to the principles of programmatic assessment positively affects PT total scores, use of PT feedback, and perceived learning value.     

Peer assessment in the objective structured clinical examination: A scoping review

Background: The objective structured clinical examination (OSCE), originally designed with experts assessing trainees’ competence, is more frequently employed with an element of peer assessment and feedback. Although peer assessment in higher education has been studied, its role in OSCEs has not reviewed.

Aims: The aim of this study is to conduct a scoping review and explore the role of peer assessment and feedback in the OSCE.

Methods: Electronic database and hand searching yielded 507 articles. Twenty-one full records were screened, of which 13 were included in the review. Two independent reviewers completed each step of the review.

Results: Peer-based OSCEs are used to assess students’ accuracy in assessing OSCE performance and to promote learning. Peer examiners (PE) tend to award better global ratings and variable checklist ratings compared to faculty and provide high-quality feedback. Participating in these OSCEs is perceived as beneficial for learning.

Conclusions: Peer assessment and feedback can be used to gauge PE reliability and promote learning. Teachers using these OSCEs must use methodology which fits their purpose. Competency-based education calls for diversification of assessment practices and asks how assessment impacts learning; the peer-based OSCE responds to these demands and will become an important practice in health professions education.     

Students’ perception of portfolio as a learning tool at King Abdulaziz University Medical School

Abstract

Background: Medical education has a longstanding tradition of using logbooks to record activities. The portfolio is an alternative tool to document competence and promote reflective practice. This study assessed the acceptance of portfolio use among Saudi undergraduate medical students.

Methods: Portfolios were introduced in the 2nd through 5th years at King Abdulaziz University over a two-year period (2013–2015). At the end of each academic year, students completed a mixed questionnaire that included a self-assessment of skills learned through the use of portfolio.

Results: The results showed a difference in focus between basic and clinical years: in basic years students’ focus was on acquiring practical skills, but in clinical years they focused more on acquiring complex skills, including identifying and managing problems. The questionnaire responses nonetheless revealed a positive trend in acceptance (belief in the educational value) of portfolios among students and their mentors, across the years of the program.

Conclusions: Using portfolios as a developmental learning and formative assessment tool in the early undergraduate years was found to contribute to students’ ability to create their own clinical skills guidelines in later years, as well as to engage in and appreciate reflective learning.     

Eight-year outcomes of a competency-based residency training program in orthopedic surgery

Abstract

Background: The Division of Orthopaedic Surgery at the University of Toronto implemented a pilot residency training program that used a competency-based framework in July of 2009. The competency-based curriculum (CBC) deployed an innovative, modularized approach that dramatically intensified both the structured learning elements and the assessment processes.

Methods: This paper discusses the initial curriculum design of the CBC pilot program; the refinement of the curriculum using curriculum mapping that allowed for efficiencies in educational delivery; details of evaluating resident competence; feedback from external reviews by accrediting bodies; and trainee and program outcomes for the first eight years of the program’s implementation.

Results: Feedback from the residents, the faculty, and the postgraduate residency training accreditation bodies on the CBC has been positive and suggests that the essential framework of the program may provide a valuable tool to other programs that are contemplating embarking on transition to competency-based education.

Conclusions: While the goal of the program was not to shorten training per se, efficiencies gained through a modular, competency-based program have resulted in shortened time to completion of residency training for some learners.     

Differences between medical school and PGY1 learning outcomes: An explanation for new graduates not being “work ready”?

Abstract

Background: Widespread concerns about new medical graduates’ ‘work readiness’ may reflect, in part, differences in mandatory learning outcomes for medical students and new medical graduates.

Purpose: To examine differences between required medical student and PGY1 (first year resident) training program outcomes, and the nature and magnitude of these differences.

Method: Comparison, systematic identification and thematic analysis of differences between the graduate outcomes in the Australian Medical Council Standards for the Assessment and Accreditation of Primary Medical Programs and those in the New Zealand Curriculum Framework for Prevocational Training.

Results: The relationship between these outcome statements were categorized as: essentially similar; continuity; partial discontinuity; and complete discontinuity of learning trajectory. Areas requiring substantial new learning may reflect medical schools’ focus on individual student performance, and on learning and assessments based on single episodes of often uncomplicated illness. This contrasted with a post-graduate focus on integrated health care delivery by teams and management of complex illnesses over the whole patient care journey.

Conclusions: Characterizing these marked differences between pre-graduate and postgraduate standards, within a trajectory of learning, explains some of the difficulties in students’ preparation for work readiness. These could inform learning interventions to support new graduates’ professional development to ensure patient safety. Development and revision of accreditation standards should include formal review against the expectations of the preceding and succeeding phases of learning.     

The effectiveness of sequencing virtual patients with lectures in a deductive or inductive learning approach

Background: Virtual patients (VPs) can be sequenced with other instructional methods in different ways.

Aim: To investigate the effect of sequencing VPs with lectures in a deductive approach, in comparison with an inductive approach, on students’ knowledge acquisition, retention, and transfer.

Methods: For two different topics, 84 out of 87 students have participated in the lecture and VP sessions. Students from female and male campuses have been randomly assigned to one of the two learning approaches (deductive and inductive), yielding four experimental groups. Each group received a lecture session and an independent VP learning activity, which either followed the lecture session in the deductive group or preceded it in the inductive group. Students were administrated knowledge acquisition and retention written tests as well as transfer tests using two new VPs.

Results: There was no significant effect for the learning approach on knowledge acquisition or retention, while for knowledge transfer, males have benefited from the inductive approach in topic 1 while in the more complex topic 2, they have benefited from the deductive approach. On the other hand, females seem to be largely unaffected by learning approach.

Conclusions: Sequencing VPs in inductive and deductive learning approaches leads to no significant differences on students’ performance when full guidance is offered in the inductive approach.     

An international study on teachers’ conceptions of learning and teaching and corresponding teacher profiles

Abstract

Introduction: Teachers’ conceptions of learning and teaching (COLT) affect their teaching behaviour. The 18 item COLT instrument has been developed in the Netherlands and comprises three scales, ‘teacher centredness’, ‘appreciation of active learning’ and ‘orientation to professional practice’. Previously we found five teacher profiles. The aim of this study was to find out if the COLT instrument can be used in an international setting.

Methods: Data were collected with the web-based COLT. Cronbach’s alphas of the three COLT scales were calculated. Subsequently a cluster analysis was conducted to identify different teacher profiles, followed by a split half validation procedure.

Results: Respondents (n?=?708) worked in 28 countries. Cronbach’s alphas were 0.67, 0.54, and 0.66. A six-cluster solution fitted best, based on meaning and explained variance. The sixth teacher profile scored high on ‘teacher centredness’, average on ‘appreciation of active learning’ and low on ‘orientation to professional practice’. The split half validation resulted in a Cohen’s kappa of 0.744.

Discussion: Cronbach’s alphas indicated acceptable reliablities for all three subscales. The new, sixth profile was labelled ‘neo-transmitter’.

Conclusion: We found evidence supporting the validity of the use of COLT in an international context and identified a new, sixth teacher profile.     

Bolstering diagnostic reasoning skills with adaptive learning

Abstract

Purpose: Adaptive learning emerges when precise assessment informs delivery of educational materials. This study will demonstrate how data from Human Dx, a case-based e-learning platform, can characterize an individual’s diagnostic reasoning skills, and deliver tailored content to improve accuracy.

Methods: Pearson Chi-square analysis was used to assess variability in accuracy across three groups of participants (attendings, residents, and medical students) and three categories of cases (core medical, surgical, and other). Logistic regression analyses were conducted to explore the relationship between solve duration and accuracy. Mean accuracy and duration were calculated for 370 individuals. Repeated measures analysis of variance (ANOVA) were used to assess variability for an individual solver across the three categories.

Results: There were significant differences in accuracy across the three groups and the three categories (p?<?0.001). Individual solvers have significant variance in accuracy across the three categories. Shorter solve duration predicted higher accuracy. Patterns of performance were identified; four profiles are highlighted to demonstrate potential adaptive learning interventions.

Conclusions: Human Dx can assess diagnostic reasoning skills. When weaknesses are identified, adaptive learning strategies can push content to promote skill development. This has implications for customizing curricular elements to improve the diagnostic skills of healthcare professionals.