A Scoping Review of the Pharmacy Curriculum Outcomes Assessment Literature

Objective. To identify themes, gaps, and sources of evidence from the literature regarding the Pharmacy Curriculum Outcomes Assessment (PCOA) to inform practice and additional areas for research in pharmacy education.Findings. Nineteen articles describing the administration and use of PCOA were identified. Since PCOA was made a curricular requirement by the Accreditation Council of Pharmacy Education in 2016, the focus of literature related to the PCOA has shifted from administration practices (four articles published before 2016 vs two articles published since) to determining models that may predict student performance on the assessment (two vs five articles) or how the examination might be used to predict future performance (one vs seven articles), especially on the North American Pharmacist Licensing Examination. While there is a growing body of literature focused on the PCOA’s utility for measuring performance, few variables have been consistently used.Summary. This review found no studies with objectives that aligned with the initial intended use of the PCOA as defined by the National Association of Boards of Pharmacy, which included tracking individual student performance throughout the curriculum, benchmarking programs against other programs, and evaluating whether a program was meeting their desired outcomes. Additionally, no consensus across the Academy was found as to the proper use of the PCOA, and a paucity of literature exists regarding how the PCOA informs schools and colleges about the effectiveness of their curriculum. There is a need for the Academy to establish a uniform application for the PCOA in pharmacy schools, assess the resources that programs need to administer this required assessment, and determine the utility of the PCOA to measure curricular effectiveness and/or student performance.     

Pharmacy curriculum outcomes assessment for individual student assessment and curricular evaluation

The Pharmacy Curriculum Outcomes Assessment (PCOA) is a standardized examination for assessing academic progress of pharmacy students. Although no other national benchmarking tool is available on a national level, the PCOA has not been adopted by all colleges and schools of pharmacy. Palm Beach Atlantic University (PBAU) compared 2008-2010 PCOA results of its P1, P2, and P3 students to their current grade point average (GPA) and to results of a national cohort. The reliability coefficient of PCOA was 0.91, 0.90, and 0.93 for the 3 years, respectively. PBAU results showed a positive correlation between GPA and PCOA scale score. A comparison of subtopic results helped to identify areas of strengths and weaknesses of the curriculum. PCOA provides useful comparative data that can facilitate individual student assessment as well as programmatic evaluation. There are no other standardized assessment tools available. Despite limitations, PCOA warrants consideration by colleges and schools of pharmacy. Expanded participation could enhance its utility as a meaningful benchmark.     

Pharmacy students' test-taking motivation-effort on a low-stakes standardized test

Objective

To measure third-year pharmacy students'' level of motivation while completing the Pharmacy Curriculum Outcomes Assessment (PCOA) administered as a low-stakes test to better understand use of the PCOA as a measure of student content knowledge.

Methods

Student motivation was manipulated through an incentive (ie, personal letter from the dean) and a process of statistical motivation filtering. Data were analyzed to determine any differences between the experimental and control groups in PCOA test performance, motivation to perform well, and test performance after filtering for low motivation-effort.

Results

Incentivizing students diminished the need for filtering PCOA scores for low effort. Where filtering was used, performance scores improved, providing a more realistic measure of aggregate student performance.

Conclusions

To ensure that PCOA scores are an accurate reflection of student knowledge, incentivizing and/or filtering for low motivation-effort among pharmacy students should be considered fundamental best practice when the PCOA is administered as a low-stakes testKey words: Pharmacy Curriculum Outcomes Assessment (PCOA), testing, motivation, examination     

A case-based toxicology elective course to enhance student learning in pharmacotherapy

Objective. To assess the impact of a case-based toxicology elective course on student learning in related required courses and student performance on the Pharmacy Curriculum Outcomes Assessment (PCOA) examination.Design. A case-based clinical toxicology elective course that contained topics from 2 required courses, Pharmacology III and Pharmacotherapy II, was offered in the spring 2009 to second- and third-year pharmacy students.Assessment. Scores on the Toxicology subsection of the PCOA of students enrolled in the elective were higher than those of students not enrolled (91.3% ± 4.1 vs. 67.2% ± 5.7). Enrollment in the elective was related to increased examination scores among Pharmacotherapy II students (89.5% ± 2.0 vs. 83.9% ± 1.8). Students indicated on course survey instruments that they were satisfied with the new elective offering.Conclusions. A toxicology elective provided a clinically relevant, active-learning experience for pharmacy students that addressed a curricular need within the college and increased examination scores.     

Development of a Summative Examination with Subject Matter Expert Validation

Objective. To describe the development, implementation and impact of a summative examination on student learning and programmatic curricular outcomes.Methods. The summative examination was developed using a systematic approach. Item reliability was evaluated using standard psychometric analyses. Content validity was assessed using necessity scoring as determined by subject matter experts.Results. Almost 700 items written by 37 faculty members were evaluated. Passing standards increased annually (45% in 2009 to 67% in 2014) as the result of targeting item difficulty and necessity scores. The percentage of items exhibiting discrimination above 0.1 increased to 100% over the four years. Necessity scores above 2.75 out of 4 increased from 65% to 100% of items over six years of examination administration.Conclusion. This examination successfully assessed student and curricular outcomes. Faculty member engagement observed in this process supports a culture of assessment. This type of examination could be beneficial to other programs.     

COVID-19 impact on pharmacy education in Saudi Arabia: Challenges and opportunities

The first case of COVID-19 was announced at the end of year 2019, and later many cases were identified worldwide. In Saudi Arabia, the first case was announced on 2 March 2020. To prevent the spread of this pandemic disease, many precautionary actions were taken by Saudi government. One of these actions was closing public and private schools and universities and transfer the educational activities to virtual platforms. All colleges of Pharmacy in Saudi Arabia, whether the 21 public or the eight private ones, were affected by those sudden transitions and their responses varied according to their preparedness levels. Here we shared our experience in king Saud University in the curricular components of pharmacy school that includes classroom teaching, laboratory teaching, experiential training, assessment, and extracurricular activity and student support during COVID-19 compulsory lockdown. Lastly, we presented the lesson learned toward pharmacy education from COVID-19 pandemic.     

Assessment of Students’ Critical-Thinking and Problem-Solving Abilities Across a 6-Year Doctor of Pharmacy Program

Objective. To determine the feasibility of using a validated set of assessment rubrics to assess students’ critical-thinking and problem-solving abilities across a doctor of pharmacy (PharmD) curriculum.Methods. Trained faculty assessors used validated rubrics to assess student work samples for critical-thinking and problem-solving abilities. Assessment scores were collected and analyzed to determine student achievement of these 2 ability outcomes across the curriculum. Feasibility of the process was evaluated in terms of time and resources used.Results. One hundred sixty-one samples were assessed for critical thinking, and 159 samples were assessed for problem-solving. Rubric scoring allowed assessors to evaluate four 5- to 7-page work samples per hour. The analysis indicated that overall critical-thinking scores improved over the curriculum. Although low yield for problem-solving samples precluded meaningful data analysis, it was informative for identifying potentially needed curricular improvements.Conclusions. Use of assessment rubrics for program ability outcomes was deemed authentic and feasible. Problem-solving was identified as a curricular area that may need improving. This assessment method has great potential to inform continuous quality improvement of a PharmD program.     

Development of a Reliable,Valid Annual Skills Mastery Assessment Examination

Objective

To develop a methodology for a reliable, valid annual skills mastery assessment examination to provide formative student feedback, inform curricular review, and comply with the Accreditation Council for Pharmacy Education (ACPE) Standards 2007.

Design

A sample of program-level ability-based outcomes skills were chosen for the examination. Test items were written, underwent quality control, and were scored for level of difficulty. Versions of the examination for first-, second-, third-, and fourth-year pharmacy students were developed and administered, the results were analyzed, reliability and validity were evaluated, and reports were generated. Item-writing guidelines, quality control procedures, and examination production steps were codified to create a criterion-referenced examination. Students and faculty advisors received detailed score reports and results were used to guide student performance and stimulate a review of curricular outcomes.

Assessment

Content, criterion, and construct validity were analyzed as defined in the literature for the intended use of this assessment tool. Data suggest the Annual Skills Mastery Assessment (ASMA) examination is both reliable and valid. Students and faculty members were surveyed regarding the usefulness of the examination. Results indicate general satisfaction with the assessment program.

Conclusion

A reasonably reliable, reasonably valid multiple-choice annual skills mastery assessment for selected outcomes statements providing formative feedback and informed curricular review was developed.     

Team-Based Learning in US Colleges and Schools of Pharmacy

Objective. To characterize the use of team-based learning (TBL) in US colleges and schools of pharmacy, including factors that may affect implementation and perceptions of faculty members regarding the impact of TBL on educational outcomes.Methods. Respondents identified factors that inhibit or enable TBL use and its impact on student learning. Results were stratified by type of institution (public/private), class size, and TBL experience.Results. Sixty-nine of 100 faculty members (69%) representing 43 (86%) institutions responded. Major factors considered to enable TBL implementation included a single campus and student and administration buy-in. Inhibiting factors included distant campuses, faculty resistance, and lack of training. Compared with traditional lectures, TBL is perceived to enhance student engagement, improve students’ preparation for class, and promote achievement of course outcomes. In addition, TBL is perceived to be more effective than lectures at fostering learning in all 6 domains of Bloom’s Taxonomy.Conclusions. Despite potential implementation challenges, faculty members perceive that TBL improves student engagement and learning.     

Enhancing an Introductory Pharmacy Practice Experience at free medical clinics

Objectives The aim of the study was to assess and improve first‐year student pharmacists' satisfaction and learning experience in a Student‐Run Free Medical Clinic Project (SFMCP) providing medical care to an underserved population. Methods Two consecutive classes of first‐year student pharmacists at the University of California San Diego (UCSD) Skaggs School of Pharmacy and Pharmaceutical Sciences participated in an Introductory Pharmacy Practice Experience (IPPE) at the UCSD SFMCP. This IPPE involved two inter‐professional evening free clinics which provide medical care to an underserved population and opportunities for healthcare professional training and service. Year 1 students completed a self‐assessment survey instrument and year 2 students completed the survey instrument plus a new competency checklist tool. Average scores from the self‐assessment survey instrument were compared between years 1 and 2. Key findings Initial survey results showed that students felt the SFMCP was worthwhile; however, they did not experience enough involvement in the patient assistance programme or non‐pharmacy‐related clinic activities. After the competency checklist tool implementation, overall student pharmacist satisfaction of the SFMCP IPPE remained high (88%), participation in identified weak areas improved and students agreed that the tool helped focus their clinic experience. Conclusions Areas of improvement were identified with the survey instrument and the competency checklist tool increased achievement of learning objectives. Overall, student pharmacists felt the SFMCP IPPE was a good learning experience. Practising pharmacists can employ these or similar tools in specific practice settings, to evaluate and help ensure that student pharmacists or interns are achieving applicable learning objectives.     

Towards an Operational Definition of Clinical Competency in Pharmacy

Objective. To estimate the inter-rater reliability and accuracy of ratings of competence in student pharmacist/patient clinical interactions as depicted in videotaped simulations and to compare expert panelist and typical preceptor ratings of those interactions.Methods. This study used a multifactorial experimental design to estimate inter-rater reliability and accuracy of preceptors’ assessment of student performance in clinical simulations. The study protocol used nine 5-10 minute video vignettes portraying different levels of competency in student performance in simulated clinical interactions. Intra-Class Correlation (ICC) was used to calculate inter-rater reliability and Fisher exact test was used to compare differences in distribution of scores between expert and nonexpert assessments.Results. Preceptors (n=42) across 5 states assessed the simulated performances. Intra-Class Correlation estimates were higher for 3 nonrandomized video simulations compared to the 6 randomized simulations. Preceptors more readily identified high and low student performances compared to satisfactory performances. In nearly two-thirds of the rating opportunities, a higher proportion of expert panelists than preceptors rated the student performance correctly (18 of 27 scenarios).Conclusion. Valid and reliable assessments are critically important because they affect student grades and formative student feedback. Study results indicate the need for pharmacy preceptor training in performance assessment. The process demonstrated in this study can be used to establish minimum preceptor benchmarks for future national training programs.     

A Simulated Interprofessional Rounding Experience in a Clinical Assessment Course

Objective

To implement a simulated interprofessional rounding experience using human patient simulators as a required activity for third-year pharmacy students in a clinical assessment course.

Design

Interprofessional student teams consisting of pharmacy, medical, and physician assistant students participated in a simulated interprofessional rounding experience in which they provided comprehensive medical care for a simulated patient in an inpatient setting.

Assessment

Students completed a survey instrument to assess interprofessional attitudes and satisfaction before and after participation in the simulated interprofessional rounding experience. Overall student attitudes regarding interprofessional teamwork and communication significantly improved; student satisfaction with the experience was high and students’ self-perceived clinical confidence improved after participation. The mean team clinical performance scores were 65% and 75% for each simulated interprofessional rounding experience.

Conclusion

Incorporating a simulated interprofessional rounding experience into a required clinical assessment course improved student attitudes regarding interprofessional teamwork and was associated with high student satisfaction.     

Pharmacy Students’ Interpretation of Academic Integrity

Objective. To explore pharmacy students’ recognition and interpretation of situations constituting breaches of academic integrity.Methods. A survey instrument comprising 10 hypothetical student(s) scenarios was completed by 852 students in the bachelor of pharmacy program at an Australian university. The scenarios were relevant to current modes of assessment and presented degrees of ambiguity around academic integrity.Results. Identification of the hypothetical student(s) at fault, particularly in the deliberately ambiguous scenarios, was not related to the respondents’ year of study or sex. Students with fewer years of postsecondary education were more definitive in their interpretation of contentious cases. Respondents from all 4 years of study reported witnessing many of these behaviors among their peers.Conclusion. This study provided novel insight into the ambiguity surrounding academic integrity and students’ perceptions relating to the deliberate or inadvertent involvement of other parties.     

Effectiveness of Journal Club Activities Versus Clinical Debate Activities in Pharmacy Experiential Education

Objective. Within pharmacy experiential education, practicing literature evaluation skills usually occurs via journal clubs. Clinical debates have gained traction as an engaging alternative to journal club meetings while completing advanced pharmacy practice experiences (APPEs). The purpose of this study was to compare clinical knowledge and literature evaluation application between journal clubs and clinical debates during APPEs.Methods. This mixed-methods prospective study was conducted in fourth year pharmacy students completing inpatient general medicine APPEs at four institutions. Students participated in a journal club and clinical debate during their experience. Students completed a 10-item knowledge assessment after each activity. Differences in journal club and clinical debate assessment scores were analyzed. Following completion of both activities, a perception survey was administered to gauge preferences and opinions. Differences in perception survey scores for journal clubs compared to clinical debates were evaluated quantitatively, and a thematic analysis was completed for qualitative responses.Results. Fifty students participated in both activities. There were no differences between journal club and clinical debate assessment scores (57.4%±21.0% and 62.9%±20.7%, respectively). Forty students completed the post-perceptions survey and globally agreed or strongly agreed that both journal clubs and clinical debates improved confidence in literature evaluation and clinical skills. Common themes identified included applicability to pharmacists’ roles and need for clear instructions and examples.Conclusion. There was no significant difference between student performance on knowledge assessments of journal clubs and clinical debates, and students found both activities to be beneficial. Clinical debates are a reasonable alternative to journal clubs to improve pharmacy students’ knowledge and literature evaluation skills.     

Pharmacy Students’ Application of Knowledge From the Classroom to Introductory Pharmacy Practice Experiences

Objective. To determine whether and how pharmacy students used knowledge learned in the classroom during their introductory pharmacy practice experiences (IPPEs) in community and hospital settings.Design. To reinforce course concepts and make connections between coursework and practice, students documented examples of how knowledge from first-year courses was used in IPPEs.Assessment. Data submitted were categorized by classroom-based pharmacy course, including the frequency with which each course was cited. For community practice experiences, most student examples of knowledge application related to the self-care therapeutics course, pharmacy practice laboratory course, and dose form/compounding laboratory courses. Hospital IPPE examples were most frequently based on the pharmaceutical calculations course, physiology/pathophysiology course, medicinal chemistry course, and pharmacy practice laboratory course.Conclusion. All prior classroom-based pharmacy courses were cited by students as being useful during IPPEs, although some were more frequently cited than others. This activity provided useful programmatic assessment data.     

Correlation Between Active-Learning Coursework and Student Retention of Core Content During Advanced Pharmacy Practice Experiences

Objective. To implement an active-learning approach in a pharmacotherapy course sequence in the second year (P2) and third (P3) year of a doctor of pharmacy (PharmD) program and determine whether the pedagogical changes correlated with retention of core content in the fourth year (P4).Design. Class sessions were transitioned from slides-based lectures to discussion-based active-learning pedagogy.Assessment. A comprehensive examination was created and administered to assess student retention of therapeutic topics taught. Students demonstrated significantly improved overall scores on questions derived from the active-learning pedagogy used in Pharmacotherapy II and III compared to those derived from Pharmacotherapy I in which content was delivered by lecture.Conclusion. The use of active-learning strategies over lecture-based methods in pharmacotherapy courses resulted in higher retention of core content. Students’ performance in areas taught using the discussion-based methodology was superior to that which was taught using lecture-based slide presentations.