Conditions for successful reflective use of portfolios in undergraduate medical education

Aim Portfolios are often used as an instrument with which to stimulate students to reflect on their experiences. Research has shown that working with portfolios does not automatically stimulate reflection. In this study we addressed the question: What are the conditions for successful reflective use of portfolios in undergraduate medical education? Methodology/research design We designed a portfolio that was aimed at stimulating reflection in early undergraduate medical education, using experiences described in the medical education literature and elsewhere. Conditions for reflective portfolio use were identified through interviews with 13 teachers (mentors), who were experienced in mentoring students in the process of developing their portfolios. The interviews were analysed according to the principles of grounded theory. Results The conditions for successful reflective use of portfolios that emerged from the interviews fell into 4 categories: coaching; portfolio structure and guidelines; relevant experiences and materials, and summative assessment. According to the mentors, working with a portfolio designed to meet these conditions will stimulate students' reflective abilities. Conclusion This study shows that portfolios are a potentially valuable method of assessing and developing students' reflective skills in undergraduate medical training, provided certain conditions for effective portfolios are recognised and met. Portfolios have a strong potential for enhancing learning and assessment but they are very vulnerable and may easily lead to disappointment. Before implementing portfolios in education, one should first consider whether the necessary conditions can be fulfilled, including an appropriate portfolio structure, an appropriate assessment procedure, the provision of enough new experiences and materials, and sufficient teacher capacity for adequate coaching and assessment.     

A Retrospective Examination of the Relationship Between Implementation Quality of the Coordinated School Health Program Model and School-Level Academic Indicators Over Time*

BACKGROUND: Although models such as the coordinated school health program (CSHP) are widely available to address student health needs, school professionals have been unconvinced that scarce resources should be allocated to improving student health. Concern that attention may be diverted from meeting academic accountability goals is often seen as a reason not to attend to student health. Despite continuing calls for the study of multicomponent health programs in relation to educational achievement, the understanding of the extent to which adherence to the characteristics of CSHP contributes to or compromises academic outcomes over time remains incomplete. METHODS: A retrospective study was conducted of CSHP implementation across 158 public schools in Delaware, serving grades K-12. Using a doubly multivariate design, this study examined 3 levels of CSHP implementation across 5 school-level academic indicators for 3 years. Indicators included school performance, school progress, and aggregated student performance in 3 content areas—reading, mathematics, and writing. Data for the years prior to, during, and following implementation of CSHP were analyzed. RESULTS: Multivariate main effects of year by implementation level were detected. CSHP schools with high levels of implementation had better school-level performance and progress ratings. CSHP implementation did not have an effect on reading, math, and writing indicators, though all groups showed significant improvements over time in these areas. CONCLUSIONS: Results of this study suggest that quality implementation of CSHP does not adversely impact school-level academic indicators over time. Moreover, findings suggest a better fit with school-wide accountability indicators than with specific content-based achievement indicators.     

Portfolio use in general practice vocational training: a survey of GP registrars

BACKGROUND: Portfolios are increasingly advocated in medical education. Qualitative studies have suggested their value in stimulating experiential learning, promoting deep learning and encouraging reflection. This study explored the use of educational portfolios in reflective learning by general practice (GP) registrars in Yorkshire, England. METHOD: A postal questionnaire was sent to the 92 registrars of a deanery in November 2001, after a pilot study with registrars in a single district had been carried out. The questionnaire explored the registrars' use of the portfolio to collect information and for reflection, as well as registrars' attitudes towards the portfolio. It was returned by 71 registrars, representing a 77% response rate. Structured in-depth interviews were used to support the results obtained. RESULTS: Of the registrars who responded, 65% recorded information on a regular basis and 42% used the portfolio in reflective learning. Experienced registrars used the portfolio least. Those with supportive trainers used the portfolio more in reflection. CONCLUSIONS: The study suggests that the role of the trainer/supervisor is vital in portfolio-based learning. It raises questions about the acceptability of portfolio learning. It adds weight to the suggestion that careful introduction of portfolios and training of supervisors is vital. Further work to establish the role of portfolios in reflective learning is recommended.     

The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: a case study

AIM: Because it deals with qualitative information, portfolio assessment inevitably involves some degree of subjectivity. The use of stricter assessment criteria or more structured and prescribed content would improve interrater reliability, but would obliterate the essence of portfolio assessment in terms of flexibility, personal orientation and authenticity. We resolved this dilemma by using qualitative research criteria as opposed to reliability in the evaluation of portfolio assessment. METHODOLOGY/RESEARCH DESIGN: Five qualitative research strategies were used to achieve credibility and dependability of assessment: triangulation, prolonged engagement, member checking, audit trail and dependability audit. Mentors read portfolios at least twice during the year, providing feedback and guidance (prolonged engagement). Their recommendation for the end-of-year grade was discussed with the student (member checking) and submitted to a member of the portfolio committee. Information from different sources was combined (triangulation). Portfolios causing persistent disagreement were submitted to the full portfolio assessment committee. Quality assurance procedures with external auditors were used (dependability audit) and the assessment process was thoroughly documented (audit trail). RESULTS: A total of 233 portfolios were assessed. Students and mentors disagreed on 7 (3%) portfolios and 9 portfolios were submitted to the full committee. The final decision on 29 (12%) portfolios differed from the mentor's recommendation. CONCLUSION: We think we have devised an assessment procedure that safeguards the characteristics of portfolio assessment, with credibility and dependability of assessment built into the judgement procedure. Further support for credibility and dependability might be sought by means of a study involving different assessment committees.     

Implementation of a coordinated school health program in a rural, low-income community

Background: Coordinated school health programs (CSHPs) bring together educational and community resources in the school environment. This method is particularly important in rural areas like Kansas, where resources and trained health professionals are in short supply. Rural Stafford County, Kansas, struggles with health professional shortages and a low-income, high-need population. Methods: In 2001, Stafford County’s Unified School District 349 began a multiyear CSHP development process, which required adaptations for implementation in a rural area. First, a CSHP team was formed of community and administrative stakeholders as well as school system representatives. Next, the CSHP team assessed school district demographics so the program framework could be targeted to health needs. During a yearlong planning phase, the CSHP team determined 4 priority areas for program development, as limited staff and funds precluded developing programs in all 8 traditional CSHP areas. Program activities were tailored to the population demographics and available resources. Results: Program outcomes were supported by School Health Index (SHI) data. Of the 8 CSHP focus areas, the SHI found high scores in 3 of the Stafford CSHP’s priority areas: Health Services; Psychological, Counseling, and Social Services; and Physical Education. The fourth Stafford CSHP priority area, Nutrition Services, scored similarly to the less prioritized areas. Conclusions: The process by which the Stafford school district modified and implemented CSHP methods can serve as a model for CSHPs in other rural, high-need areas.     

Demonstration of Portfolios to Assess Competency of Residents

Residency educators are identifying approaches to measure resident competence. Portfolios are well suited since they require work already completed as part of patient care where competency must be demonstrated. This paper describes assessment of the reliability and validity of portfolios in a psychiatry residency program. This was a cross-sectional study across 4years of residency education. Using guidelines, 18 residents assembled portfolios containing five entries chosen from 13 skills. Trained raters scored the portfolios. Residents and faculty were interviewed about their perceptions. Generalizability results indicated five entries and two raters were sufficient for relative decisions. Six entries or a third rater would be sufficient for absolute decisions. Portfolio scores tended to improve with years of training and correlated with psychiatric knowledge but not clinical performance. Residents and faculty identified benefits to assembling a portfolio. Portfolios incorporate tasks embedded in the residency to provide evidence of resident competency. The results support that the score is reliable and valid.     

Demonstration of Portfolios to Assess Competency of Residents

Residency educators are identifying approaches to measure resident competence. Portfolios are well suited since they require work already completed as part of patient care where competency must be demonstrated. This paper describes assessment of the reliability and validity of portfolios in a psychiatry residency program. This was a cross-sectional study across 4years of residency education. Using guidelines, 18 residents assembled portfolios containing five entries chosen from 13 skills. Trained raters scored the portfolios. Residents and faculty were interviewed about their perceptions. Generalizability results indicated five entries and two raters were sufficient for relative decisions. Six entries or a third rater would be sufficient for absolute decisions. Portfolio scores tended to improve with years of training and correlated with psychiatric knowledge but not clinical performance. Residents and faculty identified benefits to assembling a portfolio. Portfolios incorporate tasks embedded in the residency to provide evidence of resident competency. The results support that the score is reliable and valid.     

Portfolio of health advice for young people with disabilities transferring to adult care

BACKGROUND: The transition from paediatric to adult services for young people with complex disabilities is fraught with anxieties for families. To facilitate the transition process a portfolio comprising reports from parents, paediatrician and therapists was collected and given to the young person and family on leaving school. AIM: To evaluate the young people and their parents' views of the usefulness of portfolios in providing comprehensive picture of their health needs. METHODS: A prospective study was performed. Eight families were interviewed and semi-quantitative feedback obtained using standardized questionnaires. RESULTS: Seven out of eight young people and their families were satisfied with the portfolios they received. All of them found the portfolios worthwhile and useful. A quarter of the families would like to have been consulted more about the content of the individual reports. CONCLUSIONS: Portfolios of health reports facilitate transfer of information when young people with disabilities move from paediatric to adult services. Ideally portfolios should be interagency, and include reports from education and social services.     

Conducting a Comprehensive School Health Program

ABSTRACT: Since the late 1980s, the eight-component model of the comprehensive school health program (CSHP), has been adopted widely in the United States and internationally. While it is acknowledged that the eight program elements should be delivered in a coordinated, interactive manner, numerous issues regarding how this intergration best can be achieved, including who at the school level should have this responsibility and how the eight components relate conceptually and logistically, have not been addressed adequately. In essence, a CSHP transforms several solo performers into an orchestra. This article proposes the school health coordinator as an essential element in the eight-component model of the CSHP - the maestro, without whom there can be no symphony. The coordinator's principal responsibilities include administration, integration of personnel and programs, evaluation, and direct intervention. Three program elements - staff wellness, healthy environment, and community/family involvement - are subsumed within the coordinator's role, effectively reducing the number of program elements from eight to five. Potential benefits in addition to issues regarding feasibility of the SHC, are discussed and studies examining the effectiveness of the SHC model are recommended. (J Sch Health. 1996;66(2):59–63)     

Evaluation of the Florida coordinated school health program pilot schools project

ABSTRACT: The Florida Department of Education, with CDC funding, designed the Florida Coordinated School Health Program Pilot Schools Project (PSP) to encourage innovative approaches to promote coordinated school health programs (CSHP) in Florida schools. Each of eight pilot schools received $15,000 in project funding, three years of technical assistance including on-site and off-site assistance, a project office resource center, mailings of resource materials, needs assessment and evaluation assistance, and three PSP Summer Institutes. Project evaluators created a context evaluation, approaching each school independently as a “case study” to measure the school's progress in meeting goals established at baseline. Data were collected using the How Healthy is Your School? needs assessment instrument, a School Health Portfolio constructed by each school team, a Pilot Schools Project Team Member Survey instrument, midcourse team interviews, final team interviews, and performance indicator data obtained from pilot and control schools. The PSP posed two fundamental questions: “Can financial resources, professional training, and technical assistance enable individual schools to create and sustain a coordinated school health program?” and “What outcomes reasonably can one expect from a coordinated school health program, assuming programs receive adequate support over time?” First, activities at the eight schools confirmed that a coordinated school health programs can be established and sustained. Program strength and sustainability depend on long-term resources, qualified personnel, and administrative support. Second, though coordinated school health programs may improve school performance indicators, the PSP yielded insufficient evidence to support that belief. Future projects should include robust measurement and evaluation designs, thereby producing conclusive evidence about the influence of a coordinated school health program on such outcomes.     

The School Health Portfolio System: a new tool for planning and evaluating coordinated school health programs

The School Health Portfolio System (SHPS), developed originally to evaluate the Florida Coordinated School Health Program Pilot Schools Project, offers a new and innovative system for planning and evaluating a coordinated school health program at the individual school level. The SHPS provides practitioners a detailed but easy-to-use system that enables schools to create new programs or modify existing programs across all eight components of the CSHP model, as well as administrative support critical to sustainability. The System comes packaged as a self-contained, notebook-style manual divided into 15 sections. It includes electronic templates of key documents to guide school teams in creating a customized portfolio, and a list of sample goals and artifacts that confirm achievement of a goal related to the school's coordinated school health program. An evaluation rubric provides a structured method to assess a program portfolio's contents, and the extent to which the contents document achievement of program goals. The rubric produces both a qualitative assessment, such as a narrative summary of program strengths and areas for improvement, and a quantitative assessment, such as a numerical score (0-100), letter grade (A-F), or 5-star system (*-*****). The physical structure, function, and scoring of the rubric depend on the method of assessment. The SHPS enables schools to set goals based on individual school needs, and incorporate CSHP goals into school improvement plans--a critical factor in sustainability and accountability. The System also offers teams the option of coordinating their efforts with CDC's School Health Index as a companion assessment measure. This article outlines the process a team would follow in developing a portfolio, and includes a sample assessment for the area of School Health Education.     

Maintenance of a health promotion program in elementary schools: results from the CATCH-ON study key informant interviews.

To better understand the institutionalization process in Child and Adolescent Trial for Cardiovascular Health (CATCH) intervention and control schools, 199 key informant interviews were conducted with school food service staff, physical education teachers, classroom teachers, and administrators at the four CATCH-ON field centers. School personnel were asked to talk about the degree of CATCH program implementation, who at the school or school district was instrumental in promoting CATCH, and the conditions that facilitated or impeded the institutionalization of CATCH activities and philosophies. The CATCH Physical Education (PE) component appeared to have the highest level of institutionalization, and the CATCH classroom curriculum and family components appeared to have the lowest levels of institutionalization. The primary barriers expressed included the low priority for health promotion activities and time constraints of schools: lack of mechanisms for training of school staff; and lack of sufficient funds for materials, equipment, and lower fat vendor products.     

Web‐ or paper‐based portfolios: is there a difference?

OBJECTIVE: To determine the differential effects of a paper-based versus a web-based portfolio in terms of portfolio quality, user-friendliness and student motivation. METHODS: An experimental design was used to compare Year 1 medical students' reflective portfolios. The portfolios differed in presentation medium only (i.e. web-based versus paper-based). Content analysis, a student questionnaire and mentor interviews were used to evaluate portfolio quality, user-friendliness and student motivation. A total of 92 portfolios were scored independently by 2 raters using a portfolio quality-rating instrument. RESULTS: Portfolio structure, quality of reflection and quality of evidence showed no significant effects of presentation medium. Multi-level analysis showed a significant effect for student motivation: web-based portfolios scored 0.39 more than paper-based portfolios (P < 0.05; effect size 0.76). The mentors reported no differences in portfolio quality, except that there were more visuals in web-based portfolios. Students spent significantly more time preparing the web-based than the paper-based portfolios (15.4 hours versus 12.2 hours; t = 2.1, P < 0.05; effect size 0.46). The 2 student groups did not differ significantly in terms of their satisfaction with the portfolio. The mentors perceived the web-based portfolios as more user-friendly. CONCLUSIONS: The web-based portfolios were found to enhance students' motivation, were more user-friendly for mentors, and delivered the same content quality compared with paper-based portfolios. This suggests that web-based presentation may promote acceptance of portfolios by students and teachers alike.     

A Complementary Ecological Model of the Coordinated School Health Program

BACKGROUND: A complementary ecological model of the coordinated school health program (CSHP) reflecting 20 years of evolved changes is proposed. Ecology refers to the complex interrelationship between intrapersonal factors, interpersonal processes and primary groups, institutional factors, community factors, and public policy. METHODS: Public health and child development theories that incorporate the influence of personal and social environments on health behavior, along with models that incorporate the influence of ecology, were consulted. RESULTS: Concepts from several models were combined with the 8 components of CSHP to formulate an ecological model involving 6 program and services components in an inner circle surrounded by 4 concentric rings representing the healthy school environment, essential structures of CSHP, local school district governance, and family and community involvement. CONCLUSION: This complementary ecological model is intended to serve as an additional conceptual approach to CSHP practice, evaluation, and research, and should prove especially useful to practitioners and researchers who already have a fundamental understanding of CSHP.     

School Personnel Perspectives on their School’s Implementation of a School-Based Suicide Prevention Program

Youth suicide is a national public health priority, with policymakers highlighting schools as an ideal setting in which to deliver suicide prevention programs. Over the past decade, the number of schools implementing such programs has grown substantially, yet little is known about how successfully such programs are being implemented. This study examines the implementation of a district-wide suicide prevention program through key informant interviews with school personnel. Schools with higher rates of implementing district protocols for at-risk students had an organized system to respond to at-risk students, a process for effectively responding to students who were at-risk for suicide, and strong administrative support. In contrast, schools that had lower rates of implementing district protocols relied on a handful of individuals for suicide prevention activities and had limited administrative support. Attention to organizational factors leading to successful implementation of school-based suicide prevention programs may enhance the role of schools in national adolescent suicide prevention efforts.     

Portfolio learning in general practice vocational training – does it work?

Reflective learning has been widely addressed as an important learning mechanism in the educational literature. The creation of portfolios has been seen as a mechanism to promote this, though there has been little exploration of the place of a portfolio in general practice training. This study examined the introduction of a model of a portfolio learning strategy into one training region. The model had been developed by previous pilot work. The study explored the model in terms of its usefulness in general practice training and its relationship to reflective learning. An educational facilitator was used to support this introduction. Workshops and written material were developed to disseminate and refine ideas generated in the pilot study. This was followed by visits to trainer/general practice registrar (GPR) pairs over a 2-year period. These visits included semistructured interviews, which were tape-recorded and analysed using qualitative methods. Additional written resources, video and audio material as well as new workshops were designed with the researched participants in order to promote the development of the concepts of portfolio learning. Sixty interviews were carried out over a 2-year period with 44 pairs of trainers and GPRs. This included a total of 27 trainers and 44 registrars. Eighteen pairs were interviewed twice. Two focus groups were used at the end of the project. Portfolios have a place to play in general practice vocational training. They act as a bridge between hospital and general practice. They can be used to develop a learner-centred curriculum, explore difficult emotive concerns and facilitate feedback. They do not suit all learning styles. Their use is determined by a cost–benefit analysis described in this study.     

How Prospective School Health Educators Can Build a Portfolio to Communicate Professional Expertise

ABSTRACT: Portfolios — purposeful and selective collections of authentic evidence regarding teaching and learning — occur increasingly in K-12 classroom, preservice teacher education, and university initiatives to enhance teaching effectiveness of faculty. In addition to being used to improve instruction, portfolios can help teacher candidates with initial employment. This article describes an organized set of materials prospective school health educators should develop to accurately represent their newly acquired skills and competencies. (J Sch Health. 1997;67(6):228–231)     

Advocacy to action: addressing coordinated school health program issues with school boards

As the need for Coordinated School Health Programs (CSHP) increases, so does recognition of the importance for advocating with local school boards for their support. Identifying the diversified make up of school board members and implementing effective strategies to advocate for coordinated school health can help facilitate the successful inclusion of such a program. With increasing emphasis placed on standardized testing and the "basic" curriculum, school board members need to become aware of specific benefits a CSHP can provide their district. With the relationship between health status and academic achievement confirmed in scientific research, school boards may begin paying more attention to providing high-quality health services and health instruction for students. This article presents items to consider and steps to take before, during, and after addressing a local school board for their support in implementing a CSHP.     

A Collaborative Approach to Family Literacy Evaluation Strategies

McCosh Even Start, a federally funded project located at McCosh School in inner-city Chicago neighborhood, is a partnership between Northeastern Illinois University and The Chicago Public School. The parent#shchild components of the program are: home visiting, an after school family literacy program, parent#shchild field trips, computer classes, and the use of videotaping and photography to document learning. The adult literacy component includes computer, photography and video classes, a parent club for literacy career development skills and parent field trips. Computer, video technology and photography were used as tools to develop literacy, but they were also used to document the literacy progress of parents and children. To evaluate the results of the program, staff, parents, children and a video documenter collaborated to use a wide range of evaluation. These strategies used to show literacy improvement included video documentation and interviews, parent interest questionnaires, observations and book logs, photo collections with personalized captions, portfolios, information reading inventories and parent self-evaluations.