Programmatic assessment: From assessment of learning to assessment for learning

In assessment a considerable shift in thinking has occurred from assessment of learning to assessment for learning. This has important implications for the conceptual framework from which to approach the issue of assessment, but also with respect to the research agenda. The main conceptual changes pertain to programmes of assessment. This has led to a broadened perspective on the types of construct assessment tries to capture, the way information from various sources is collected and collated, the role of human judgement and the variety of psychometric methods to determine the quality of the assessment. Research into the quality of assessment programmes, how assessment influences learning and teaching, new psychometric models and the role of human judgement is much needed.     

Enhancing reliability in portfolio assessment: 'shaping' the portfolio

This paper reports a follow-on project that assessed a series of portfolios assembled by a cohort of participants attending a course for prospective general practice trainers. In an attempt to enhance reliability, a framework for defining and addressing problems using a reflective practice model was offered to participants. The reliability of the judgements made by a panel of assessors about individual 'components', together with an overall global judgement about performance were studied. The reliability of individual assessors' judgements (i.e. their consistency) was moderate, but inter-rater reliability did not reach a level that could support making a safe summative judgement. Despite offering a possible structure for demonstrating reflective processes, the levels of reliability reached were similar to the earlier work and other subjective assessments generally, and perhaps reflected individuality of personal agendas of both the assessed and the assessors, and variations in portfolio structure and content; even agreement among the assessors about evidence of the framework being used was poor. Suggestions for approaches in the future are made. The conclusion remains that while portfolios might be valuable as resources for learning, as assessment tools they should be treated as problematic.     

Embracing the new paradigm of assessment in residency training: an assessment programme for first-year residency training in anaesthesiology

Assessment of clinical competence is facing a paradigm shift in more than one sense. The shift relates to test content, which increasingly covers a broader spectrum of competences than mere medical expertise, and to test methods, with an increasing focus on testing performance in realistic settings. Also there is a shift in the concept of assessment in that instruction and assessment are no longer seen as being separate in time and purpose, but as integral parts of the learning process. The nature of the new paradigm for assessment is well described but the challenge to programme directors is to specify the evaluation situations and develop appropriate methods. This paper describes the intrinsic rational validation process in outlining an assessment programme for first-year anaesthesiology residency training according to the new paradigm. The applicability to other residency programmes and higher-level training in anaesthesiology is discussed.     

Peer assessment in the objective structured clinical examination: A scoping review

Background: The objective structured clinical examination (OSCE), originally designed with experts assessing trainees’ competence, is more frequently employed with an element of peer assessment and feedback. Although peer assessment in higher education has been studied, its role in OSCEs has not reviewed.

Aims: The aim of this study is to conduct a scoping review and explore the role of peer assessment and feedback in the OSCE.

Methods: Electronic database and hand searching yielded 507 articles. Twenty-one full records were screened, of which 13 were included in the review. Two independent reviewers completed each step of the review.

Results: Peer-based OSCEs are used to assess students’ accuracy in assessing OSCE performance and to promote learning. Peer examiners (PE) tend to award better global ratings and variable checklist ratings compared to faculty and provide high-quality feedback. Participating in these OSCEs is perceived as beneficial for learning.

Conclusions: Peer assessment and feedback can be used to gauge PE reliability and promote learning. Teachers using these OSCEs must use methodology which fits their purpose. Competency-based education calls for diversification of assessment practices and asks how assessment impacts learning; the peer-based OSCE responds to these demands and will become an important practice in health professions education.     

Moving IPE from being ‘worthy’ to ‘required’ in health professional curriculum: Is good governance the missing part?

Abstract

Introduction: There is an accelerating trend towards interdisciplinary learning and teaching activities in higher education. However, traditional discipline-based approaches to making and implementing decisions (academic governance) can be out of step. Within health professional education, there is a particular need to embrace interdisciplinary approaches to learning in the form of interprofessional education (IPE). The aim of this study was to identify academic governance models that successfully, or otherwise, supported the maintenance of quality standards of IPE programs and learning activities.

Method: A 10-year literature search yielded 11 articles that addressed the IPE governance of academic standards.

Results: Three models were identified: centralized, decentralized, and stand-alone. Key features of each are described with discussion on strengths and weaknesses for curriculum development, academic leadership and student learning, and the challenges of enabling interprofessional governance within traditional university academic governance structures.

Conclusion: As with interdisciplinary education more broadly, there is emerging literature regarding effective governance systems to enable quality IPE within individual institutions. Educators should give careful consideration to the optimal governance model for their particular institution and context.     

Communication,learning and assessment: Exploring the dimensions of the digital learning environment

Abstract

Advances in technology make it possible to supplement in-person teaching activities with digital learning, use electronic records in patient care, and communicate through social media. This relatively new “digital learning environment” has changed how medical trainees learn, participate in patient care, are assessed, and provide feedback. Communication has changed with the use of digital health records, the evolution of interdisciplinary and interprofessional communication, and the emergence of social media. Learning has evolved with the proliferation of online tools such as apps, blogs, podcasts, and wikis, and the formation of virtual communities. Assessment of learners has progressed due to the increasing amounts of data being collected and analyzed. Digital technologies have also enhanced learning in resource-poor environments by making resources and expertise more accessible. While digital technology offers benefits to learners, the teachers, and health care systems, there are concerns regarding the ownership, privacy, safety, and management of patient and learner data. We highlight selected themes in the domains of digital communication, digital learning resources, and digital assessment and close by providing practical recommendations for the integration of digital technology into education, with the aim of maximizing its benefits while reducing risks.     

Designing programmes of assessment: A participatory approach

Programmatic approaches to assessment provide purposeful and meaningful assessment yet few examples of their development exist. The aim of this study was to describe the development of a programme of assessment using a participatory action research (PAR) approach. Nine work-based assessors together with three academics met on six occasions to explore the current approach to competency-based assessment in the placement component of a dietetics university course, the findings of which were used to design a programme of assessment. Findings revealed disconnect between current assessment approaches and best practice. The PAR methodology fostered a shared vision for the design of a programmatic approach to assessment and strong leadership was essential. Participants experienced a philosophical shift in their views towards assessment, supporting the implementation of a new assessment programme. This paper is the first to describe a PAR approach as a feasible and effective way forward in the design of programmatic assessment. The approach engaged stakeholders to strengthen their abilities as work-based assessors and produced champions for best practice assessment.     

Entrustment decisions: Implications for curriculum development and assessment

Abstract

With increased interest in the use of entrustable professional activities (EPAs) in undergraduate and postgraduate medical education, comes questions about their implications for curriculum development and assessment. This paper addresses some of those questions, discussed at a symposium of the 2017 conference of AMEE, by presenting the components of an EPA, describing their importance and application, identifying their implications for assessment, and pinpointing some of challenges they pose in undergraduate and postgraduate settings. It defines entrustment, describes the three levels of trust, and presents trainee and supervisor factors that influence it as well as perceived benefits, and risks. Two aspects of EPAs have implications for assessment: units of professional practice and decisions based on entrustment, which impact an assessment’s blueprint, test methods, scores, and standards. In an undergraduate setting EPAs have great appeal, but work is needed to identify and develop a robust assessment system for core EPAs. At the postgraduate level, there is tension between the granularity of the competencies and the integrated nature of the EPAs. Even though work remains, EPAs offer an important step in the evolution of competency-based education.     

Enhancing reliability in portfolio assessment: discussions between assessors

This paper reports the reliability in assessments of a series of portfolios assembled by a cohort of participants attending a course for prospective general practice trainers. Initial individual assessments are compared with open discussion between random pairs of assessors to produce paired composite scores, and analysed using kappa statistics. Overall reliability of a global pass/refer judgement improved from a kappa of 0.26 (fair) using individual assessment, to 0.5 (moderate) with paired discussants.