The ACGME Outcome Project in Ophthalmology: Practical Recommendations for Overcoming the Barriers to Local Implementation of the National Mandate

The Accreditation Council for Graduate Medical Education (ACGME) Outcome project is a well-defined, 10-year, ambitious national mandate to improve resident education through the teaching and assessing of six general competencies (i.e., patient care, medical knowledge, professionalism, interpersonal and communication skills, practice-based learning and improvement, and systems-based practice). Over the past 5 years a Task Force at the University of Iowa has deployed 10 major competency assessment tools: 1) the University of Iowa residency curriculum; 2) direct observation using the Ophthalmic Clinical Exercise (OCEX); 3) journal club; 4) multi-source evaluation; 5) a modified Dreyfus scale faculty global evaluation form; 6) the Iowa Cataract surgical curriculum; 7) the Iowa Ophthalmic Laser curriculum; 8) traditional written and oral examinations; 9) self reflection exercises and projects; and 10) learner portfolio. We report our 5-year experience with an implementation matrix for translating the national ACGME mandate into local compliance. We identify the barriers encountered by our Task Force in local implementation and propose practice solutions based upon our experience for overcoming the cultural, institutional, financial, and other barriers to success. We hope that our institutional work and experience will stimulate other programs to participate more fully in the ACGME Outcomes Project.     

Teaching and assessing competency in retinal lasers in ophthalmology residency

BACKGROUND AND OBJECTIVE: The Accreditation Council for Graduate Medical Education has mandated that residency programs teach and assess six specific competencies (ie, medical knowledge, patient care, communication and interpersonal skills, professionalism, practice-based learning, and systems-based learning). To the authors' knowledge, there is no standardized and widely used curriculum for teaching and assessing resident competencies in retinal lasers. METHODS: The pertinent literature on resident education in retinal lasers is reviewed and specific "good practices" for teaching and assessing laser competency are presented. RESULTS: Development and deployment of educational tools that teach and assess laser competency simultaneously; are reliable, reproducible, and valid; have low faculty time burden; and are affordable, generalizable, and fair are recommended. CONCLUSION: Retinal laser competency can be taught and assessed in ophthalmology residency training programs. Future research will be needed to provide evidence that these teaching and assessment tools produce improved educational and patient outcomes and provide verifiable, reliable, and valid evidence of resident competence in retinal lasers.     

Implementing a structured oral examination into a residency program: getting started.

BACKGROUND AND OBJECTIVE: To describe the steps involved in implementing a structured oral examination into a residency program PATIENTS AND METHODS: Existing oral examination format was evaluated and deficiencies addressed by creating a structured examination and increasing the frequency of administration to ophthalmology residents at the University of Arkansas for Medical Sciences. RESULTS: A structured oral examination was implemented. It provided both a realistic simulation of the board certification examination and a process that was a useful teaching method and assessment tool. It was valuable in addressing the Accreditation Council for Graduate Medical Education (ACGME) competencies. CONCLUSIONS: Oral examinations can be a versatile means of teaching and assessing resident physicians. Although the implementation process can be onerous, once in place, it can become an efficient tool for assessing all categories of the ACGME competencies and can provide valuable training in oral examination performance.     

Re-engineering the resident applicant selection process in ophthalmology: a literature review and recommendations for improvement

The current resident selection process for ophthalmology has undergone little change over the last several years and remains highly dependent on the traditional selection factors (i.e., grades, honors, letters of recommendation, and an interview). Unfortunately, these selection factors have not been shown to be consistently predictive of future resident performance. In addition, the Accreditation Council for Graduate Medical Education (ACGME) has mandated implementation of six new competencies in resident training in the USA and the current selection process does not directly recruit for these competencies. We propose an implementation strategy to re-engineer and improve the resident selection process in ophthalmology and potentially develop assessments that would be predictive of actual downstream resident performance that would encompass the ACGME related competencies. An intra-departmental Task Force for the ACGME Competencies reviewed a PubMed literature search regarding resident selection. A content expert (AGL) gleaned selected “good practices” from the literature review and summarized the results. Specific recommendations were reviewed for topicality to ophthalmology and where possible for feasibility, reliability, and validity. We summarize several good practices identified from the literature review and propose an implementation matrix for aligning the resident application process with the ACGME competencies that might include: using a standardized and consolidated academic score for the cognitive domains; converting the letter of recommendation format into a letter of evaluation; standardizing the letters of evaluation, including the “Dean's letter”; using behavior specific interview techniques with standardized questions; and developing a specialty based consensus for the selection of traits specific to ophthalmology that might predict success. The resident selection process for ophthalmology might be improved by implementation of specific good practices from the literature. Ophthalmology should strive to develop applicant selection tools that might be useful for predicting residency performance and that would align with the ACGME competency mandate for tools to predict future performance as a physician.     

Teaching and assessing surgical competency in ophthalmology training programs.

BACKGROUND AND OBJECTIVE: The Accreditation Council for Graduate Medical Education (ACGME) has mandated implementation of six new competencies in resident training in the United States. An implementation strategy is proposed to teach and assess cataract surgical competence. PATIENTS AND METHODS: An intradepartmental Task Force for the ACGME competencies reviewed the literature for assessment tools to develop an implementation matrix for assessing surgical competence. RESULTS: "Good practices" (gleaned from the literature) were adapted for the institution's needs and tested, including (1) written and explicit goals or objectives for each stage of training; (2) substitution of a criterion-referenced (Dreyfus model) scoring rubric for a norm-referenced, peer-benchmarked global evaluation; (3) use of formative rather than summative feedback; (4) incorporation of deliberate practice (Ericsson model); and (5) portfolio-based documentation of sentinel event markers and remediation. CONCLUSION: An implementation matrix for teaching and assessing surgical competence might be useful for local compliance with the ACGME mandate.     

Teaching and assessing systems-based competency in ophthalmology residency training programs

The Accreditation Council for Graduate Medical Education (ACGME) has mandated that residency programs, including ophthalmology, teach and assess specific competencies, including systems-based learning. We review the pertinent literature on systems-based learning for ophthalmology and recommend specific “good practices” to manage the ACGME mandate. Tools are required that both teach and assess systems based learning competency simultaneously, that are reliable and valid, that have low faculty burden, and that are affordable, practical, and fair. Future research should provide evidence that these interventions produce improved educational and patient outcomes and show proof of competence in systems based learning among residents and clinicians in practice.     

Attitudes toward parental leave and breastfeeding during ophthalmology residency

IntroductionAs mandated by the Accreditation Council for Graduate Medical Education (ACGME), residency programs are required to have parental leave policies. A lack of standardized requirements leads to a lack of uniformity among programs. We discuss resident and program director attitudes toward parental leave and examine the range of policies on parental leave and breastfeeding within ophthalmology residency programs.MethodsTwo electronic surveys assessing perceptions toward parental leave during residency and breastfeeding on return to clinical duties were created individually for completion by ophthalmology residents or residency program directors, respectively, with responses collected over 4 weeks.ResultsOf residents who took parental leave, 23 (87%) denied taking time off without pay. The most commonly reported effects on training by residents were missed surgical training and impact on research. Nearly 60% of residents (N = 26) reported receiving negative feedback or actions prior to or after the leave. The majority of residents felt program directors and coresidents were supportive (53.8%, 48.1%, respectively), but parental leave negatively affected their coresidents (46.2%). Twenty-five program directors reported that there are written parental leave policies in place at their institution. Sex disparities were noted, with program directors reporting more negative impacts on surgical training in female residents (p = 0.035). There was no statistically significant difference between program director attitudes on clinical training, well-being, or burnout by resident sex. All program directors were supportive of breastfeeding; half reported an institutional breastfeeding policy.ConclusionA national discussion on standardizing parental leave and breastfeeding policies over all ophthalmology residency programs is warranted.     

Microsurgical lab testing is a reliable method for assessing ophthalmology residents' surgical skills

Background

Formal assessment of clinical competencies is necessary to ensure that all residents are acquiring important skills and, in the United States, will soon become a requirement for residency programme accreditation by the Accreditation Council for Graduate Medical Education (ACGME). The Eye Surgical Skills Assessment Test (ESSAT), a laboratory‐based surgical skills obstacle course, was developed in response to the need for improved tools for the assessment of surgical skills during residency. The ESSAT has previously been shown to have face and content validity, and in this study we sought to determine its inter‐rater reliability and, to some extent, its construct validity.

Methods

Twenty‐seven content experts (residency programme directors and faculty members involved with resident surgical training) watched videos of a junior resident and senior resident completing the three ESSAT stations (skin suturing, muscle recession, and phacoemulsification: wound construction & suturing technique) and completed assessment forms, both task‐specific checklists and a global rating scale of performance.

Results

The ESSAT showed strong inter‐rater reliability for determining whether a resident “passed” a threshold of competency at each station for both the checklists and global rating scale. In addition, for each station, the senior resident was consistently rated above a “passing” threshold using either assessment form, whereas the junior resident was more often rated below (94% vs 30% passing on completed forms).

Conclusion

These results, along with the findings of our face and content validity analysis, support the reliability and validity of the ESSAT, and indicate that it could be a useful tool for improving the assessment of surgical skill during residency. The ESSAT is a tool that all residency programmes could implement as a part of their ophthalmic surgical curriculum and competency assessment, and may be useful to set a threshold of competence that all residents would need to achieve prior to entering the operating room.New assessment tools are needed to improve the process of teaching and evaluating residents in core competencies. In the United States, this need has become a mandate, as the Accreditation Council for Graduate Medical Education (ACGME) has set forth a timeline by which all residency programmes, in order to maintain their accreditation, must develop and integrate new tools for teaching and evaluating residents in six core competencies.¹ Surgical skills was added as a seventh competency by the American Board of Ophthalmology.² The Eye Surgical Skills Assessment Test (ESSAT) was developed both as a response to these mandates and to the need for more objective and structured methods of assessing residents'' surgical skills. In ophthalmology, several new surgical skills assessment tools have recently been developed: in addition to the wet lab‐based ESSAT, programmes will have in their armamentarium procedure specific evaluation forms (which many programmes have already been using), assessment of OR performance with videos as well as specific forms^3,4 and simulation technology.^5,6Modelled after the Objective Structured Assessment of Technical Skills (OSATS), a laboratory‐based surgical skills‐assessment test developed⁷ and validated⁸ by researchers in the field of general surgery, the ESSAT is made up of three simulated surgical tasks that the resident is required to complete in the microsurgical laboratory. These tasks are (1) skin suturing, (2) muscle recession and (3) phacoemulsification: a wound construction & suturing technique. The resident''s performance may be observed live or on videotape by a surgical educator who completes a task‐specific checklist as well as a global rating scale of performance for each task.The ESSAT offers the controlled setting of the microsurgical laboratory for residents to learn and be assessed in a standardised fashion. In addition, the ESSAT takes skills assessment and basic competency determination out of the operating room, where patient risks become involved. We previously established that the ESSAT has face and content validity by surveying experts in the field and incorporating their suggestions for improving the ESSAT.⁹ To ensure that the ESSAT has the test characteristics needed of a good assessment tool (ie, validity, reliability), we set forth in this study to also establish the inter‐rater reliability and, to a limited degree, the construct validity of the ESSAT, particularly for the purpose of establishing a threshold of basic skills competency that all residents must achieve in order to enter the operating room.     

An update on the novel approaches towards skills assessment of ophthalmology residents in the Indian scenario

An essential part of the teaching-learning paradigm is assessment. It is one of the ways to achieve feedback for the various methods that have been used to impart a particular skill. This is true of ophthalmology training, where various clinical and surgical skills are learned as part of the residency program. In preparation for residents to become proficient ophthalmologists, both formative and summative assessments are of paramount importance. At present, assessment is primarily summative in the form of a university examination, including theory and practical examinations that are conducted at the end of the three years of residency. A formative assessment can make course corrections early on, allowing for an improved understanding of the subject and the acquisition of clinical and surgical skills. Formative assessments also allow us to customize the teaching methodology considering individual residents’ learning capabilities. In addition, formative assessments have the advantage of alleviating the stress of a “final” examination, which could sometimes result in a less-than-optimum performance by the residents. The COVID-19 pandemic has forced us to adopt new teaching methods, which has led to the adoption of changes in assessment. In this regard, we discuss the different assessment tools available, their pros and cons, and how best these tools can be made applicable in the setting of an ophthalmology residency program.     

The new competencies and their impact on resident training in ophthalmology

Changes in the health care environment and the evolving needs of external stakeholders have created a demand for a new set of competencies in residency training. This article reviews the forces that have shaped the development of the new competencies; defines the specific Accreditation Council for Graduate Medical Education competencies; describes the tools that might be used to measure these competencies; introduces current concepts and terms in the field of post-graduate medical education; and summarizes an implementation plan for ophthalmology. Meeting the new Accreditation Council for Graduate Medical Education competencies will require the development of a toolbox for measurement outcomes and a reinvention of the residency training process in ophthalmology. A coordinated, thoughtful, and standardized effort will be required to meet the challenge.     

Defining the core competency of professionalism based on the patient's perception

BACKGROUND: To better define professionalism using a 10-question multiple-choice survey of patient preferences. METHODS: One hundred and ninety-one adult patients (response rate: 52% +/- 5%) completed a survey over a 3-week period in resident and faculty ophthalmology clinics in a university setting in a rural portion of the southern USA. Most patients voluntarily provided information on gender, race and age. Data are reported at proportions +/- 95% confidence intervals. RESULTS: Patients desire a degree of formality from their physicians in the form of a handshake (61% +/- 7%), greeting of family members (69% +/- 7%) and in addressing oneself as doctor. They also prefer note taking by the physician while speaking with them. However, patients do not think that the wearing of a white coat is necessary. Most patients assume (84% +/- 5%) that the physician washes his/her hands. Surprisingly, patients (60% +/- 7%) are willing to maintain a relationship with a physician despite the use of medical jargon. We found few differences related to gender and none related to race. Women (64% +/- 9%) preferred a closed door during the exam. Men (81% +/- 8%) either did not want the physician to wear a white coat or said that it made no difference. Those younger than 46 years (67% +/- 10%) preferred the door closed compared with those who were older (45% +/- 10%). CONCLUSIONS: Our study helps to define professionalism by providing concrete examples of the expectations of patients in the southern USA during physician interaction. Minor adjustments to the patient encounter based on these findings may increase patients' perception of professionalism, creating a higher level of trust. These are teachable precepts that can be incorporated into residency training.     

The Ophthalmic Clinical Evaluation Exercise (OCEX)

PURPOSE: New concise tools must be developed to assess reliably and validly the core residency competencies identified by the Accreditation Council for Graduate Medical Education. PARTICIPANTS: Eighteen content experts (residency program directors). METHODS: A 1-page Ophthalmic Clinical Exercise Examination (OCEX) checklist, for use during observed resident-patient interactions, was developed by an American Board of Ophthalmology taskforce. The OCEX checklist was sent to 18 content experts for their review and constructive comments. RESULTS: Experts' comments were incorporated, establishing face and content validity. CONCLUSIONS: The OCEX has face and content validity. It can be used to assess a resident's patient care skills, medical knowledge, and interpersonal skills. Reliability and predictive validity still need to be determined.     

The ACGME Milestone Project in Ophthalmology

The ACGME is moving towards the next generation of accreditation in the USA called the Next Accreditation System (NAS). The NAS is anticipated to reduce the burden on programs to comply with accreditation requirements; to produce meaningful, innovative, and continuous benchmark outcomes data; to use ongoing individual and programmatic milestones to judge performance; and ultimately to produce better trained residents, to improve the quality of care, to reduce health care costs and health care disparities, and to provide objective evidence to the public and other external stakeholders of the quality of graduate medical education across the specialties of medicine. We describe the ACGME milestone development process for ophthalmology. If successful, the NAS will benefit all programs by reducing the programmatic burden and paperwork; increasing accreditation cycle length; and improving all programs through formative and summative feedback.     

Evaluation of motivating factors for selecting optometric residency education.

There are now more than 50 accredited optometric residency programs. We sought to learn the primary motivating factors for selecting residency education beyond the traditional 4-year professional program. Questionnaires addressing this issue were sent in May 1990 to directors of all Council on Optometric Education (COE)-accredited residency programs for distribution to currently participating residents. Of the 81 existing residency positions, 61 questionnaires (75 percent) were returned. Three residents (5 percent) accepted positions in residency programs because they had no definite practice plans following graduation. One resident (2 percent) could not take state board examinations and elected to enter a residency program until licensure could be obtained. Fifty-three (85 percent) selected residency education to enhance their clinical skills, and five residents (8 percent) cited other reasons, such as to prepare for a career in optometric education, to enhance long-term career objectives, and to mature. For 50 (83 percent) of the respondents, residency education was their first choice following graduation from optometry school. We conclude that optometric residency education is highly valued as a source of enrichment of clinical skills.     

Mapping the landscape of cataract surgery teaching assessment in Canadian residency programs

Objective

The Royal College of Physicians and Surgeons of Canada has mandated a shift in post-graduate residency education in Canada towards a competency-based model. Within this context, it is unclear how resident's competence in cataract surgery is currently being assessed for both formative and summative purposes. Therefore, we conducted a national survey to evaluate the current landscape of cataract surgery teaching in Canadian Ophthalmology programs.

Involving ophthalmology departmental stakeholders in developing workplace-based assessment tools

ObjectiveAs competency-based medical education (CBME) rolls out across Canada, the assessment process is changing. Our purpose was to involve stakeholders in the selection and modification of workplace-based assessment (WBA) tools for use in an ophthalmology residency program.DesignThis is a qualitative case study conducted in one ophthalmology department at a midsized teaching hospital in eastern Ontario.ParticipantsOphthalmology faculty and residents within the Emergency Eye Clinic participated in the study.MethodsPhase 1 consisted of faculty piloting the tools and providing written feedback. Phase 2 consisted of 2 focus groups, 1 for residents (n = 9) and 1 for faculty (n = 6), to explore their experiences with the 4 piloted adapted WBA tools.ResultsResidents and faculty discussed ongoing issues with buy-in and formalizing feedback through the new assessment process. Residents also reiterated the need for more constructive feedback delivered in a timely and sensitive manner. Generally, residents did not value numerical scales on the tools and preferred written comments. Both residents and faculty valued oral over written feedback given the interactive nature of oral feedback.ConclusionsThis study provides insight into faculty and resident perspectives about WBA tools within one ophthalmology department. These results informed the development of WBA tools within the department and highlighted the importance of shifting the assessment culture to accommodate programmatic approaches to assessment in CBME. Involving key stakeholders in the change process has been a valuable strategy. Future research should examine whether or not these perspectives change after CBME implementation.     

Vision-related research priorities and how to finance them

A number of organizations have employed a consultative process with the vision community to engage relevant parties in identifying needs and opportunities for vision research. The National Eye Institute in the US and the European Commission are currently undergoing consultation to develop priorities for vision research. Once these priorities have been established, the challenge will be to identify the resources to advance these research agendas. Success rates for Federal funding for research have decreased recently in the USA, UK, and Australia. Researchers should consider various potential funding sources for their research. The universal consideration for funding is that the reason for funding should align with the mission of the funding organization. In addition to Federal research organizations that fund investigator-initiated research, other potential funding sources include nongovernmental organizations, for-profit companies, individual philanthropy, and service organizations. In addition to aligning with organizational funding priorities, researchers need to consider turn-around time and total funds available including whether an organization will cover institutional indirect costs. Websites are useful tools to find information about organizations that fund research, including grant deadlines. Collaboration is encouraged.     

A review of teaching methods and outcomes of resident phacoemulsification

Cataract surgery with phacoemulsification is a challenging procedure for surgeons in training to learn to perform safely, efficiently, and effectively. We review the auxiliary learning tools outside the operating room that residency programs have incorporated into their curriculum to improve surgical skills, including wet laboratory and surgical simulators. We then discuss different methods of teaching cataract surgery in the operating room. Our goal is to define a learning curve for cataract surgery. We demonstrate that complication rates decline significantly after a resident performs an average of 70 cases. We summarize the reported incidence and risk factors for complications in resident-performed cataract surgery to help identify cases that require a higher level of skill to improve visual outcomes. We suggest that future studies include details on preoperative comorbidities, risk stratification, resident skill level, and frequency of takeover by attending.     

A current analysis of optometric residency programs

Optometric residency programs were introduced into the profession in the 1970's and have proliferated markedly since that time. The philosophy of residency programs in general is included as well as a summary of currently offered residencies in optometry. Additional information including areas of program emphasis, the "typical" residency program and compensation levels are provided. An assessment of the advantages for the continuance of current programs and future expansion of optometric residency opportunities is discussed.