Implementing the logic model for measuring the value of faculty affairs activities.

In today's environment of increasing accountability in higher education and health care, it is critical that administrative units of a medical school demonstrate the added value of their activities to the school's mission and that these units discriminate those activities that demonstrate the most return on investment. This is particularly important for administrative units whose activities may not be considered essential to the basic functioning of the medical school. For example, admissions would likely be considered an essential administrative unit that the medical school cannot do without, while faculty development might be considered nonessential. Effective measurement systems serve two purposes. They guide decision making throughout the organization and they serve as a basis for evaluating performance. This article describes use of the program logic model to measure the contribution of faculty affairs and development offices to the recruitment, retention, and development of a medical school's teaching faculty, an outcome central to the mission of the medical school. The process of developing and rewarding faculty for teaching is used to illustrate the application of this method in linking activities of faculty affairs and development offices to outcomes that are of importance to the medical school.     

The Academy at Harvard Medical School: nurturing teaching and stimulating innovation.

The Academy at Harvard Medical School, established in 2001, was formed at a critical moment for medical schools in this country. Several decades of enormous growth in the biomedical research and clinical care activities of medical school faculty have resulted in great societal benefit. The unintended consequence has been a decline in faculty time and reward for the educational mission that is unique to a medical school. The impact of this decline is particularly felt now because the explosive growth in the science and technology relevant to medical practice, coupled with dramatic changes in the health care delivery system, calls for new models for the education of the next generation of physicians. The mission of the academy is to renew and reinvigorate the educational mission of Harvard Medical School (HMS). By bringing together a select group of some of the school's most talented and dedicated faculty and providing direct support for their work related to education, the academy has created a unique mechanism for increasing the recognition of teaching contributions of both academy members and the teaching faculty at large, fostering educational innovation, and providing a forum for the exchange of ideas related to medical education that cross departmental and institutional lines. The authors describe the academy's membership criteria, structure, governance, activities, institutional impact, and plans for long-term evaluation, and indicate challenges the academy will face in the future.     

The metrics process: Baylor's development of a "report card" for faculty and departments.

In 1996, Baylor College of Medicine began the first year of its "metrics process," collecting, analyzing, and reporting data on the performance of each individual faculty member and each department in achieving the school's missions of education, patient care, research, service, and finance. This article is a report of the first two years of the process, with updates about the 1999 process, future plans, and lessons learned. The primary goal of the metrics process is to provide meaningful data to assess and improve the performance of faculty and departments across all missions. The authors (1) indicate the categories chosen, within each mission of the school, for measuring faculty time and effort (e.g., patient care, with or without learners) and state the measures chosen (e.g., percentage of time); (2) describe the development of questionnaires in 1996 and 1997 to acquire data from faculty, in the chosen categories and measures, about the time and effort they spent; and (3) report highlights of the resulting departmental data that were gathered in 1997. Among the key categories and units of measure chosen for measuring faculty (and departmental) time and effort are research grant dollars (total and per research full-time equivalent, or FTE); basic research grant dollars per square foot of laboratory space; percentage of faculty who spend at least 50% of their time in research who are National Institutes of Health principal investigators; numbers of inpatient and outpatient visits per evaluation and management FTE; total relative value units (RVUs) per patient-care FTE; patient-care income/RVU and expense/RVU for total faculty and support staff; percentage of faculty with at least one leadership position in a state or national organization; and income in excess of expense, by mission (e.g., patient care). Results of comparing data from the first two years of the metrics process demonstrate marked improvements in performance for most research measures (i.e., items of measurement agreed upon for the metrics process). The process is continually being redeveloped; the ultimate challenge is to place the objective measurements in a context where less objective qualities (e.g., innovation) also figure importantly in the evaluation and fostering of excellence. The metrics process is providing important management data, encouraging significant discussions among faculty and chairs about performance and accountability, and aiding greatly in departmental goal-setting and ultimately in determining the overall performance of the school.     

Measuring contributions to the research mission of medical schools.

The authors of this article, who were the members and staff of a research panel formed by the AAMC as part of its mission-based management initiative, reflect on the growing interest in quantitative information in the management of the research mission of medical schools. They note the serious limitations of any such system of measures for research, particularly its inability to represent directly the quality of the research effort. Despite these concerns, the authors acknowledge that leaders in academic medicine have always used quantitative measures in one form or another to compare performance or assess progress. Two factors appear to be driving increases in this practice: (1) the need to demonstrate to institutional stakeholders that resources are being used wisely and that the school's performance justifies continued investment in the research mission; and (2) the need to fashion an economic strategy to manage precious institutional resources, particularly research space. Given these realities, the authors offer guidelines for the proper development and use of measures to assess contributions by faculty, departments, and institutions to the research mission. They also comment on the measures most commonly used in four areas: grants and other revenue-generating activities; publications; faculty members' research reputation and contributions to the national research enterprise; and support to the general research mission of the school. The authors conclude that quantitative information can help institutional leaders in important management decisions. However, the potential for misuse is great. The key is always to regard this information as an aid to judgment, not a substitute for it.     

How do medical schools use measurement systems to track faculty activity and productivity in teaching?

The authors describe their findings from a study that (1) identified 41 medical schools or medical school departments that used metric systems to quantify faculty activity and productivity in teaching and (2) analyzed the purposes and progress of those systems. Among the reasons articulated for developing these systems, the most common was to identify a "rational" method for distributing funds to departments. More generally, institutions wanted to emphasize the importance of the school's educational mission. The schools varied in the types of information they tracked, ranging from a selective focus on medical school education to a comprehensive assessment of teaching activity and educational administration, committee work, and advising. Schools were almost evenly split between those that used a relative-value-unit method of tracking activity and those that used a contact-hour method. This study also identified six challenges that the institutions encountered with these metric systems: (1) the lack of a culture of data in management; (2) skepticism of faculty and chairs; (3) the misguided search for one perfect metric; (4) the expectation that a metric system will erase ambiguity regarding faculty teaching contributions; (5) the lack of, and difficulty with developing, measures of quality; and (6) the tendency to become overly complex. Because of the concern about the teaching mission at medical schools, the number of institutions developing educational metric systems will likely increase in the coming years. By documenting and accounting financially for teaching, medical schools can ensure that the educational mission is valued and appropriately supported.     

Implementing a simpler approach to mission-based planning in a medical school.

Changes in the education, research, and health care environments have had a major impact on the way in which medical schools fulfill their missions, and mission-based management approaches have been suggested to link the financial information of mission costs and revenues with measures of mission activity and productivity. The authors describe a simpler system, termed Mission-Aligned Planning (MAP), and its development and implementation, during fiscal years 2002 and 2003, at the School of Medicine at the University of Texas Health Science Center at San Antonio, Texas. The MAP system merges financial measures and activity measures to allow a broad understanding of the mission activities, to facilitate strategic planning at the school and departmental levels. During the two fiscal years mentioned above, faculty of the school of medicine reported their annual hours spent in the four missions of teaching, research, clinical care, and administration and service in a survey designed by the faculty. A financial profit or loss in each mission was determined for each department by allocation of all departmental expenses and revenues to each mission. Faculty expenses (and related expenses) were allocated to the missions based on the percentage of faculty effort in each mission. This information was correlated with objective measures of mission activities. The assessment of activity allowed a better understanding of the real costs of mission activities by linking salary costs, assumed to be related to faculty time, to the missions. This was a basis for strategic planning and for allocation of institutional resources.     

A mission-based reporting system applied to an academic pathology department

We report how data from the University of California (UC) Davis mission-based reporting system (MBR) can be used to define contributions for each division within a Department of Pathology based on faculty rank and series, and to evaluate whether these contributions are in alignment with the missions of the department and the goals of the school's leadership. MBR summary reports were generated for each division within the Department of Pathology; these reports illustrated the average contribution for each faculty rank and series in each of the following missions: investigative/creative work (research), teaching, clinical service, and administrative/community service. All divisions contributed equally to the teaching mission, averaging approximately 1/3 of a faculty member's time. Research was the primary mission for faculty in both the Research and the Clinical Pathology divisions, whereas clinical service was the primary mission for Anatomic Pathology. Both Anatomic Pathology and Clinical Pathology also played a large role in the administration/community service mission. These roles were appropriate based on the division's distribution of faculty in each of the faculty series. The average contribution to both the research and administrative/community service missions were larger for the Department of Pathology than it was for the school as a whole. The Department of Pathology's average contribution to both the teaching and clinical service missions was less than the school's average. We conclude that MBR data creates unique profiles for divisions and the department and enables interdepartmental comparisons that would not be possible by other means. Within the context of our school, the present analysis illustrates that the Department of Pathology is fulfilling the expectations of the school's leadership. In a more general sense, these profiles allow appropriate monitoring of the workforce, funds flow analysis, allocation of resources, and strategic planning in an academic medical center.     

Measuring contributions to the clinical mission of medical schools and teaching hospitals.

This is the final report of a panel convened as part of the Association of American Medical College's (AAMC's) Mission-based Management Program to examine the use of metrics (i.e., measures) in assessing faculty and departmental contributions to the clinical mission. The authors begin by focusing on methods employed to estimate clinical effort and calculate a "clinical full-time equivalent," a prerequisite to comparing productivity among faculty members and departments. They then identify commonly used metrics, including relative-value units, total patient-care gross charges, total net patient fee-for-service revenue, total volume per CPT (current procedural terminologies) code by service category and number of patients per physician, discussing their advantages and disadvantages. These measures reflect the "twin pillars" of measurement criteria, those based on financial or revenue information, and those based on measured activity. In addition, the authors urge that the assessment of quality of care become more highly developed and integrated into an institution's measurement criteria. The authors acknowledge the various ways users of clinical metrics can develop standards against which to benchmark performance. They identify organizations that are sources of information about external national standards, acknowledge various factors that confound the interpretation of productivity data, and urge schools to identify and measure secondary service indicators to assist with interpretation and provide a fuller picture of performance. Finally, they discuss other, non-patient-care, activities that contribute to the clinical mission, information about which should be incorporated into the overall assessment. In summary, the authors encourage the use of clinical productivity metrics as an integral part of a comprehensive evaluation process based upon clearly articulated and agreed-upon goals and objectives. When carefully designed, these measurement systems can provide critical information that will enable institutional leaders to recognize and reward faculty and departmental performance in fulfillment of the clinical mission.     

Creating a mission-based reporting system at an academic health center.

The authors developed a Web-based mission-based reporting (MBR) system for their university's (UC Davis's) health system to report faculty members' activities in research and creative work, clinical service, education, and community/university service. They developed the system over several years (1998-2001) in response to a perceived need to better define faculty members' productivity for faculty development, financial management, and program assessment. The goal was to create a measurement tool that could be used by department chairs to counsel faculty on their performances. The MBR system provides measures of effort for each of the university's four missions. Departments or the school can use the output to better define expenditures and allocations of resources. The system provides both a quantitative metric of times spent on various activities within each mission, and a qualitative metric for the effort expended. The authors report the process of developing the MBR system and making it applicable for both clinical and basic science departments, and the mixed success experienced in its implementation. The system appears to depict the activities of most faculty fairly accurately, and chairs of test departments have been generally enthusiastic. However, resistance to general implementation remains, chiefly due to concerns about reliability, validity, and time required for completing the report. The authors conclude that MBR can be useful but will require some streamlining and the elimination of other redundant reporting instruments. A well-defined purpose is required to motivate its use.     

Expanding the view of scholarship: introduction.

The Association of American Medical Colleges' Council of Academic Societies (CAS) has a long-standing interest in scholarship as it relates to research, education, and service, the traditional definition of the activities of medical school. The work of Ernest Boyer and Charles Glassick is highly respected for redefining scholarship and conceiving how scholarship as thus defined can be assessed. Because their ideas have been applied in other areas of the academy but not widely in medical faculties, the CAS Task Force on Scholarship collected a special set of papers on Boyer's four areas of scholarship as applied to medical school, including case studies and the perspective from the university. The four areas of scholarship defined by Boyer and Glassick are the scholarship of discovery, the scholarship of integration, the scholarship of application, and the scholarship of teaching. The scholarship of discovery-research-has for decades been the primary focus for promotion and tenure for medical school faculty, even though the faculty also had major and critical activities in the other areas of scholarship. The CAS hopes that the ideas put forth in this special theme issue will produce a continuing dialogue as faculty and administrators at medical schools reflect on the value of these different forms of scholarship, their application by medical school faculty, and their contributions to the individual missions of each medical school and teaching hospital. In addition, these articles will stimulate continuing discussions that will definite equitable methods for the continued assessment of the scholarly accomplishments of medical school faculty.     

Score reporting for the 1991 Medical College Admission Test

In December 1988, a 16-member advisory panel, made up of medical school deans, basic science faculty, clinical scientists, medical education researchers, medical students, premedical advisers, and practicing physicians, completed a full-scale review of the format, content, and score-reporting system of the Medical College Admission Test (MCAT). The advisory panel's recommendations will shape the MCAT to be introduced in 1991. Presented here are data the panel used in deciding to keep the score scale for the upcoming MCAT. Survey data on student selection practices in U.S. and Canadian medical schools and acceptance information for 1987 applicants showed that medical schools used the scores differently within each school and between the schools in deciding whom to admit. The data demonstrated the value of the current reporting system and led the advisory panel to recommend retention of the 15-point MCAT score scale in 1991.     

A communication assessment and skill-building exercise (CASE) for first-year residents.

OBJECTIVE: Good communication skills are essential for residents entering postgraduate education programs. However, these skills vary widely among medical school graduates. This pilot program was designed to create opportunities for (1) teaching essential interviewing and communication skills to trainees at the beginning of residency, (2) assessing resident skills and confidence with specific types of interview situations, (3) developing faculty teaching and assessment skills, (4) encouraging collegial interaction between faculty and new trainees, and (5) guiding residency curricular development. DESCRIPTION: During residency orientation, all first-year internal medicine residents (n = 26) at the University of Minnesota participated in the communication assessment and skill-building exercise (CASE). CASE consisted of four ten-minute stations in which residents demonstrated their communication skills in encounters with standardized patients (SPs) while faculty members observed for specific skills. Faculty and SPs were oriented to the educational purposes and goals of their stations, and received instructions on methods of providing feedback to residents. With each station, residents were provided one and a half minutes of direct feedback by the faculty observer and the SP. The residents were asked to deal with an angry family member, to counsel for smoking cessation, to set a patient-encounter agenda, and to deliver bad news. A resident's performance was analyzed for each station, and individual profiles were created. All residents and faculty completed evaluations of the exercise, assessing the benefits and areas for improvement. DISCUSSION: Evaluations and feedback from residents and faculty showed that most of our objectives were accomplished. Residents reported learning important skills, receiving valuable feedback, and increasing their confidence in dealing with certain types of stressful communication situations in residency. The activity was also perceived as an excellent way to meet and interact with faculty. Evaluators found the experience rewarding, an effective method for assessing and teaching clinical skills, a faculty development experience for themselves in learning about structured practical skills exercises, and a good way to meet new interns. The residency program director found individual resident performance profiles valuable for identifying learning issues and for guiding curricular development. Time constraints were the most frequently cited area for improvement. The exercise became feasible by collaborating with the medical school Office of Education-Educational Development and Research, whose mission is to collaborate with faculty across the continuum of medical education to improve the quality of instruction and evaluation. The residency program saved considerable time, effort, and expense by using portions of the medical school's existing student skills-assessment programs and by using chief residents and faculty as evaluators. We plan to use CASE next year with a wider variety of physician-patient scenarios for interns, and to expand the program to include beginning second- and third-year residents. Also, since this type of exercise creates powerful feedback and assessment opportunities for instructors and course directors, and because feedback was so favorable from evaluators, we will encourage participation in CASE as part of our faculty educational development program.     

The University of Missouri-Kansas City School of Medicine: thirty-five years of experience with a nontraditional approach to medical education.

The University of Missouri-Kansas City (UMKC) School of Medicine is a public medical school that opened in 1971 in response to a need to train more physicians in Missouri. As a six-year, integrated, combined-degree program leading to the baccalaureate and medical degrees, the school offers an innovative, nontraditional approach to medical education. In the past 35 years, UMKC has graduated over 2,400 physicians who are successful according to outcomes measures used at other medical schools. With recent interest in reforming medical education to prepare physicians for a changing world, a review of alternative models may be especially instructive.UMKC's academic plan offers a blueprint for the curriculum plan and governance of the school. The plan is built on four hallmarks: (1) a combined baccalaureate/MD program, (2) early exposure to clinical medicine, (3) small-group learning through the docent system, and (4) a continuing ambulatory care clinic experience for four years.This article catalogs the results of this plan including student, faculty, and graduates' perceptions of and satisfaction with the school's educational approach, students' achievement on licensing examinations and in the residency match, graduates' performance in residency programs, and their subsequent career patterns. The authors also discuss lessons learned and adjustments made in response to local needs in the context of a changing environment in education, health care, and health care delivery while continually improving the school's nontraditional approach to medical education. These include changes in basic and clinical science instruction, student assessment, faculty development, and funding and governance.     

Mission-based budgeting: removing a graveyard.

Many activities in today's medical schools no longer have medical students' education as their central reason for existence. Faculty are hired primarily to provide clinical service or to make discoveries, with the role of educator of secondary importance. Budgeting in medical schools has not evolved in concert with these changing roles of faculty. The cost of medical students' education is still calculated as if all faculty were hired primarily to teach medical students and their other activities were to support this "central" mission. Most medical schools still mix revenues without regard to intent and cannot accurately determine costs because they confuse expenses with costs. At the University of Florida College of Medicine, a group of administrators, chairpersons, and faculty developed a budgeting process now called mission-based budgeting. This is a three-step process: (1) revenues are prospectively identified for each mission and then aligned with intended purposes; (2) faculty productivity, i.e., faculty effort and its quality, is measured for each of the missions; and (3) productivity is linked to the prospective budget for each mission. This process allows the institution to understand the intent of its revenues, to measure how productive its faculty are, to learn the true costs of its missions, to make wise investment decisions (subsidies), and to justify to various constituents its use of revenues. The authors describe this process, focusing particularly on methods used to develop a comprehensive database for assessment of faculty productivity in education.     

Scholarship in teaching: an imperative for the 21st century.

At some medical schools broader definitions of scholarship have emerged along with corresponding changes in their academic reward systems. Such situations are not common, however. The definition of scholarship generally applied by medical schools is unnecessarily narrow and excludes areas of legitimate academic activity and productivity that are vital to the fulfillment of the school's educational mission. The authors maintain that creative teaching with effectiveness that is rigorously substantiated, educational leadership with results that are demonstrable and broadly felt, and educational methods that advance learners' knowledge are consistent with the traditional definition of scholarship. Faculty whose educational activities fulfill the criteria above are scholars and must be recognized by promotion. The authors specifically address scholarship in education, focusing on teaching and other learning-related activities rather than on educational research, which may be assessed and rewarded using the same forms of evidence as basic science or clinical research. They build on Boyer's work, which provides a vocabulary for discussing the assumptions and values that underlie the roles of faculty as academicians. Next, they apply Glassick et al.'s criteria for judging scholarly work to faculty members' educational activities to establish a basis for recognition and reward consistent with those given for other forms of scholarship. Finally, the authors outline the organizational infrastructure needed to support scholars in education.     

Scholarship in the medical faculty from the university perspective: retaining academic values.

Academic medicine and research universities have enjoyed a close relationship that has strengthened both, spawning an era of discovery and scholarship in medicine that has earned the U.S. academic medical enterprise a high level of public trust and a deserved leadership position in the world. However, changes in the financing of medical care and in the organization of health care delivery have dramatically affected the medical school-university partnership. The growing emphasis on delivery of clinical services and the concomitant decrease in time for tenured and clinician-educator faculty to teach and do scholarly work jeopardizes both the potential for continued discovery and the education of the next generation of medical scholars. The background of the medical school-university relationship and the factors leading to the development of clinician-educator faculty tracks are reviewed, and recent trends that impact faculty scholarship are discussed. Both tenure-track and clinician-educator medical faculty, as members of the broader university community, should expect from their university colleagues a continued demand for scholarship and educational activity that reflects the underlying philosophy of the parent university. As a corollary, the university, through its medical school, must provide these faculty the time and the financial support necessary to fulfill their academic mission. The size of the clinician-educator faculty should be determined by the academic needs of the medical school rather than by the service demands of its associated health care delivery system. To accomplish this, academic medical centers will have to develop cadres of associated or clinical faculty whose primary focus is on the practice of medicine.     

Factors influencing the development of area health education centers with Texas-Mexico border populations

Area Health Education Centers (AHECs) have been viewed as an appropriate vehicle for implementing new initiatives for training health professionals who will work along the U.S.-Mexico border. Perceptions about this program in Texas were evaluated from July 1988 to June 1989 to identify problems and formulate suggestions that might be of use to academic health science centers (HSCs)--and in particular medical schools--working with Hispanic populations. Interviews were conducted with 116 people: the presidents and/or deans of all eight Texas HSCs and/or medical schools, other deans and faculty, community leaders in five border counties, and state officials. The school and community perspectives about past and present AHEC activities were compared. Some of the barriers were: insufficient components of the health care delivery system to support medical education in severely underserved areas; differing school and community priorities; cultural differences between the school faculty and the community; and feeling among community physicians and dentists that AHECs were a source of competition. The school and community respondents agreed that the AHEC program needs more cooperative planning and training that emphasizes public health education for future AHEC-like activities with border populations.     

Collaboration and peer review in medical schools' strategic planning.

The management of medical schools has never been so difficult, but can be facilitated by using a well-developed, broadly accepted strategic plan. While the concept of strategic planning has been reasonably well accepted by both faculty and leaders at most medical schools, using the strategic plan to allocate resources has proved to be a challenge. Achieving "buy-in" by all parties involved can help meet this challenge and can be critical to the success of strategic planning and management. The authors describe the collaborative planning process that the University of Wisconsin Medical School used to develop its 1998-2000 strategic plan. This unique effort culminated in using a peer-review process--similar to that used by the National Institutes of Health (NIH)--and developing criteria to select a limited number of program priorities. This selection occurred after a school-wide process to solicit strategic program proposals had taken place. Over 130 faculty from most departments throughout the school helped develop and revise the strategic plan. There was frequent communication and engagement with faculty at all levels, which was important in gaining the faculty's acceptance and, indeed, endorsement of the process and its outcomes. Because the process was effective in achieving consensus about the school's strategic priorities, it enabled the school to reach a firmer end-point and implementation plan than had been possible with the previous strategic plan. It also identified important weaknesses in some areas of the medical school; the resulting attention to those areas will help strengthen the school. Finally, the process moved much more swiftly than the previous effort. The authors recommend that such an approach be used by other medical schools, and be carried out before a school implements mission-aligned budgeting and management of its fiscal resources.     

Survival and recovery: maintaining the educational mission of the Louisiana state university school of medicine in the aftermath of hurricane Katrina.

Hurricane Katrina devastated New Orleans and the coastlines of Louisiana, Mississippi, and Alabama on August 29, 2005. The flooding in New Orleans left hundreds of thousands of people homeless and threatened to close businesses and institutions, including Louisiana State University (LSU) School of Medicine and its two principle training sites in New Orleans, Charity Hospital and University Hospital. In the weeks immediately after the storm, LSU School of Medicine resumed undergraduate and graduate medical education in Baton Rouge, Louisiana and elsewhere. The authors discuss the specific challenges they faced in relocating administrative operations, maintaining the mission of medical education, and dealing with the displacement of faculty, staff, residents, students, and patients, and the processes used to overcome these challenges. They focus on the school's educational missions, but challenges faced by the offices of student affairs, faculty affairs, and admissions are also discussed. LSU School of Medicine's experience provides lessons about organizational preparedness for a mass disaster that may be of interest to other medical schools.     

An assessment of the clinical skills of fourth-year students at four New England medical schools

This paper describes a collaborative effort among five New England medical schools to assess important clinical skills of fourth-year medical students graduating in the class of 1988; results are presented from the four schools that provided sufficient data. Faculty from each school developed 36 patient cases representing a variety of common ambulatory-care problems. Over the course of a day, each student, on average, interacted with 16 different standardized patients, who were nonphysicians trained to accurately and consistently portray a patient in a simulated clinical setting. The students obtained focused histories, performed relevant physical examinations, and provided the patients with education and counseling. At each school, the performance of a small number of the students fell below standards set by the faculty. These deficiencies were not detected with the evaluation strategies currently being used. Although the use of standardized patients should not substitute for the process of faculty observing students as they interact with real patients, it appears that standardized patients can provide faculty with important information, not readily available from other sources, about students' performances of essential clinical activities and the levels of their clinical skills.