Factors affecting the utility of the multiple mini-interview in selecting candidates for graduate-entry medical school

Context  We wished to determine which factors are important in ensuring interviewers are able to make reliable and valid decisions about the non-cognitive characteristics of candidates when selecting candidates for entry into a graduate-entry medical programme using the multiple mini-interview (MMI).
Methods  Data came from a high-stakes admissions procedure. Content validity was assured by using a framework based on international criteria for sampling the behaviours expected of entry-level students. A variance components analysis was used to estimate the reliability and sources of measurement error. Further modelling was used to estimate the optimal configurations for future MMI iterations.
Results  This study refers to 485 candidates, 155 interviewers and 21 questions taken from a pre- prepared bank. For a single MMI question and 1 assessor, 22% of the variance between scores reflected candidate-to-candidate variation. The reliability for an 8-question MMI was 0.7; to achieve 0.8 would require 14 questions. Typical inter-question correlations ranged from 0.08 to 0.38. A disattenuated correlation with the Graduate Australian Medical School Admissions Test (GAMSAT) subsection 'Reasoning in Humanities and Social Sciences' was 0.26.
Conclusions  The MMI is a moderately reliable method of assessment. The largest source of error relates to aspects of interviewer subjectivity, suggesting interviewer training would be beneficial. Candidate performance on 1 question does not correlate strongly with performance on another question, demonstrating the importance of context specificity. The MMI needs to be sufficiently long for precise comparison for ranking purposes. We supported the validity of the MMI by showing a small positive correlation with GAMSAT section scores.     

Discerning quality: using the multiple mini-interview in student selection for the Australian National University Medical School

OBJECTIVE: To describe the development and pilot testing of a set of admissions instruments based on the McMaster University multiple mini-interview (MMI) and designed to assess desirable, non-cognitive characteristics in order to inform final decisions on candidate selection for entry to medical school. METHODS: Community and faculty consultation on desirable, non-cognitive characteristics of medical students informed the development of a 10-station interview. Two stations occurred as part of a group problem-based learning scenario and 8 occurred as individual observations. All interviewers were trained. Interviews were offered to 115 candidates on an academic merit list. Interview performance was used to exclude candidates considered unsuitable, but not to re-order the academic merit list. Admissions decisions were examined in terms of individual interview station performance. RESULTS: This method proved to be an efficient process by which to interview candidates and to determine suitability. Retained and rejected candidates had significantly different total scores and mean scores for each station. Ten independent observations contributed to each decision, without significant interviewer or logistic burden. Candidates reported high levels of satisfaction with the interview process. CONCLUSIONS: Admissions interviews can be streamlined and efficient, yet remain informative. A longitudinal study is in progress to evaluate the value of the admissions processes in predicting successful graduation to medical practice.     

Assessment of non-cognitive traits through the admissions multiple mini-interview

CONTEXT: Contemporary studies have shown that traditional medical school admissions interviews have strong face validity but provide evidence for only low reliability and validity. As a result, they do not provide a standardised, defensible and fair process for all applicants. METHODS: In 2006, applicants to the University of Calgary Medical School were interviewed using the multiple mini-interview (MMI). This interview process consisted of 9, 8-minute stations where applicants were presented with scenarios they were then asked to discuss. This was followed by a single 8-minute station that allowed the applicant to discuss why he or she should be admitted to our medical school. Sociodemographic and station assessment data provided for each applicant were analysed to determine whether the MMI was a valid and reliable assessment of the non-cognitive attributes, distinguished between the non-cognitive attributes, and discriminated between those accepted and those placed on the waitlist (waiting list). We also assessed whether applicant sociodemographic characteristics were associated with acceptance or waitlist status. RESULTS: Cronbach's alpha for each station ranged from 0.97-0.98. Low correlations between stations and the factor analysis suggest each station assessed different attributes. There were significant differences in scores between those accepted and those on the waitlist. Sociodemographic differences were not associated with status on acceptance or waiting lists. DISCUSSION: The MMI is able to assess different non-cognitive attributes and our study provides additional evidence for its reliability and validity. The MMI offers a fairer and more defensible assessment of applicants to medical school than the traditional interview.     

Contribution of medical students to admission interviews

Objective  Medical students were included in 3-person interview panels selecting applicants for a 4-year medical programme for graduates and health professionals. We analysed student contributions to the interview panels.
Methods  A total of 188 semi-structured interviews were carried out over 2 years. The scores of 17 student interviewers were compared with those of the other 2 members of the panels.
Results  Students were positive about the interview process and their involvement. Overall interview scores did not differ between students and other interviewers. However, students gave lower scores overall and in a number of individual areas to those who had the highest consensus scores and were offered places on the programme. Students were more likely than other interviewers to be indecisive between 2 scores (25.2% versus 17.5%; P  <   0.01).
Conclusions  Students can be integrated effectively into the interview process. However, they tend to be more reluctant to give high scores and less decisive about committing to a single score than more experienced interviewers. Students value the experience of participating in the selection process, especially when combined with an exploration of the literature related to selection procedures.     

Should candidate scores be adjusted for interviewer stringency or leniency in the multiple mini‐interview?

Medical Education 2010: 44 : 690–698 Context There are significant levels of variation in candidate multiple mini‐interview (MMI) scores caused by interviewer‐related factors. Multi‐facet Rasch modelling (MFRM) has the capability to both identify these sources of error and partially adjust for them within a measurement model that may be fairer to the candidate. Methods Using facets software, a variance components analysis estimated sources of measurement error that were comparable with those produced by generalisability theory. Fair average scores for the effects of the stringency/leniency of interviewers and question difficulty were calculated and adjusted rankings of candidates were modelled. Results The decisions of 207 interviewers had an acceptable fit to the MFRM model. For one candidate assessed by one interviewer on one MMI question, 19.1% of the variance reflected candidate ability, 8.9% reflected interviewer stringency/leniency, 5.1% reflected interviewer question‐specific stringency/leniency and 2.6% reflected question difficulty. If adjustments were made to candidates’ raw scores for interviewer stringency/leniency and question difficulty, 11.5% of candidates would see a significant change in their ranking for selection into the programme. Greater interviewer leniency was associated with the number of candidates interviewed. Conclusions Interviewers differ in their degree of stringency/leniency and this appears to be a stable characteristic. The MFRM provides a recommendable way of giving a candidate score which adjusts for the stringency/leniency of whichever interviewers the candidate sees and the difficulty of the questions the candidate is asked.     

The multiple mini‐interview: how long is long enough?

Objectives  The multiple mini-interview (MMI) overcomes the limitations of the traditional panel interview by multiple sampling to provide improved objectivity and reliability. Reliability of the MMI is affected by number of stations; however, there are few data reporting the influence of interview duration on MMI outcome and reliability. We aimed to determine whether MMI stations can be shortened without affecting applicant rankings or compromising test reliability.
Methods  A total of 175 applicants were interviewed and assessed at 10 8-minute stations. Applicants were scored once after 8 minutes at five control stations and twice after 5 minutes and 8 minutes at five experimental stations. Scores at 5 and 8 minutes were compared using t -tests and correlation coefficients. Rankings of applicants based on 5- and 8-minute scores were compared using Spearman's rank order coefficient. The reliability of the MMI was examined for 5- and 8-minute scores using generalisability theory.
Results  Mean scores at 5 minutes were lower than mean scores at 8 minutes. Cumulative scores at 5 minutes were also lower. There were highly significant correlations between 5- and 8-minute scores at all experimental stations (0.82–0.91; P  < 0.01) and between the cumulative scores at 5 and 8 minutes (0.92; P  < 0.01). There was a strong correlation between applicant rankings based on cumulative 5- and 8-minute scores (Spearman's rank order coefficient 0.92). Reliability was not affected.
Conclusions  Reducing the duration of MMI stations from 8 to 5 minutes conserves resources with minimal effect on applicant ranking and test reliability.     

An admissions OSCE: the multiple mini-interview

CONTEXT: Although health sciences programmes continue to value non-cognitive variables such as interpersonal skills and professionalism, it is not clear that current admissions tools like the personal interview are capable of assessing ability in these domains. Hypothesising that many of the problems with the personal interview might be explained, at least in part, by it being yet another measurement tool that is plagued by context specificity, we have attempted to develop a multiple sample approach to the personal interview. METHODS: A group of 117 applicants to the undergraduate MD programme at McMaster University participated in a multiple mini-interview (MMI), consisting of 10 short objective structured clinical examination (OSCE)-style stations, in which they were presented with scenarios that required them to discuss a health-related issue (e.g. the use of placebos) with an interviewer, interact with a standardised confederate while an examiner observed the interpersonal skills displayed, or answer traditional interview questions. RESULTS: The reliability of the MMI was observed to be 0.65. Furthermore, the hypothesis that context specificity might reduce the validity of traditional interviews was supported by the finding that the variance component attributable to candidate-station interaction was greater than that attributable to candidate. Both applicants and examiners were positive about the experience and the potential for this protocol. DISCUSSION: The principles used in developing this new admissions instrument, the flexibility inherent in the multiple mini-interview, and its feasibility and cost-effectiveness are discussed.     

A structured panel interview and group exercise in the selection of medical students

The selection of entrants for medical school, as now widely agreed, should include measurements of non-academic as well as academic attributes. The authors have developed a process to assess the personal attributes of applicants. This included a structured panel interview carried out twice on each applicant by independent interviewers and a group exercise in which several applicants were observed whilst discussing a problem. Training for interviewers took the form of a half-day interactive workshop.
One hundred and forty-one school-leaving applicants completed the new assessment. No relationship was found between academic achievement as reflected by marks in a national examination and scores in the panel interview, the group exercise, or the school principal's report. However, significant intercorrelations were found between the panel interview, group exercise and school report. The results of this experience have encouraged the Auckland School to continue to explore methods to measure these attributes in a carefully controlled study.     

Instructional methods and cognitive and learning styles in web-based learning: report of two randomised trials

CONTEXT: Adapting web-based (WB) instruction to learners' individual differences may enhance learning. Objectives This study aimed to investigate aptitude-treatment interactions between learning and cognitive styles and WB instructional methods. METHODS: We carried out a factorial, randomised, controlled, crossover, post-test-only trial involving 89 internal medicine residents, family practice residents and medical students at 2 US medical schools. Parallel versions of a WB course in complementary medicine used either active or reflective questions and different end-of-module review activities ('create and study a summary table' or 'study an instructor-created table'). Participants were matched or mismatched to question type based on active or reflective learning style. Participants used each review activity for 1 course module (crossover design). Outcome measurements included the Index of Learning Styles, the Cognitive Styles Analysis test, knowledge post-test, course rating and preference. RESULTS: Post-test scores were similar for matched (mean +/- standard error of the mean 77.4 +/- 1.7) and mismatched (76.9 +/- 1.7) learners (95% confidence interval [CI] for difference - 4.3 to 5.2l, P = 0.84), as were course ratings (P = 0.16). Post-test scores did not differ between active-type questions (77.1 +/- 2.1) and reflective-type questions (77.2 +/- 1.4; P = 0.97). Post-test scores correlated with course ratings (r = 0.45). There was no difference in post-test subscores for modules completed using the 'construct table' format (78.1 +/- 1.4) or the 'table provided' format (76.1 +/- 1.4; CI - 1.1 to 5.0, P = 0.21), and wholist and analytic styles had no interaction (P = 0.75) or main effect (P = 0.18). There was no association between activity preference and wholist or analytic scores (P = 0.37). CONCLUSIONS: Cognitive and learning styles had no apparent influence on learning outcomes. There were no differences in outcome between these instructional methods.     

Can a district hospital assess its doctors for re‐licensure?

Context The Chief Medical Officer’s recommendations on medical regulation in the UK suggest that National Health Service (NHS) trusts should assess their doctors and confirm whether they remain fit to practise medicine. Objective We set out to evaluate the utility of hospital trust‐based assessment in a ‘best‐case scenario’ within existing resources. Methods We carried out a generalisability analysis, and feasibility and validity evaluation, based on an assessment process for 137 career‐grade doctors at Chesterfield Royal Hospital, Chesterfield, UK, using validated multi‐source feedback (MSF) and patient rating (PR) instruments. Results Uptake and response rates were good for MSF (91% and 85%, respectively). However, only 6% of non‐clinical doctors and anaesthetists, and 48% of clinical doctors, obtained sufficient PR ratings. Aggregate scores were acceptably reliable. Nine combined MSF ratings and 15 PR ratings produce standard errors of measurement of 0.19 on a 6‐point scale and 0.15 on a 5‐point scale, respectively. Overall aggregate scores did not identify any doctor as unsatisfactory, but 6 doctors were scored as unsatisfactory by 2 or more colleagues or patients. These performance concerns appear to merit further investigation. Patients rated female doctors better than male doctors (4.61 versus 4.46; P < 0.05). Colleagues rated UK graduates better than non‐UK graduates (5.31 versus 5.15; P < 0.05). Conclusions This study shows that the commissioning of professional services makes the implementation of an assessment process linked to appraisal feasible. However, trust‐based assessment requires significant development: developmental appraisal needs protection; new instruments are needed for non‐clinical specialties; PR requires specific administrative support, and guidance is required over concern thresholds and demographic effects. Disaggregated assessment data may help identify doctors with potential performance problems.     

Views of doctors in training on the importance and availability of career advice in UK medicine

OBJECTIVES: To determine whether doctors in their first year after qualification wanted career advice, and, if so, whether they thought they had been able to obtain useful advice, and whether older doctors thought that adequate career advice had been available to them. METHODS: We carried out a postal questionnaire survey of all UK medical graduates of 1988, 1993, 1996, 1999 and 2002, and a 25% random sample of the graduates of 2000. RESULTS: The response rate was 67.4% (24 261/35 976 mailed questionnaires). Of doctors in the first postgraduate year, 95% agreed that: 'It is important to be given career advice at this stage of training.' A total of 38% disagreed with the statement: 'I have been able to obtain useful career advice since graduation.' Of more experienced doctors surveyed between 3 and 11 years after graduation, 34% agreed that: 'Making career choices has been made more difficult by inadequate career advice.' CONCLUSIONS: The great majority of junior doctors want career advice after qualification. It cannot be assumed that they are able to seek it out for themselves satisfactorily. Career advice needs to be planned into postgraduate work and training.     

Assessing students' personal and professional development using portfolios and interviews

BACKGROUND AND PURPOSE: Medical schools are placing more emphasis on students' personal and professional development (PPD) and are seeking ways of assessing student progress towards meeting outcome goals in relation to professionalism. The Faculty of Medicine at the University of Sydney sought an assessment method that would demonstrate the value of reflection in attaining PPD, provide feedback and encourage students to take responsibility for setting and achieving high standards of performance. METHODS: The instruments used to assess Year 1 students in PPD are a portfolio and interview. This assessment format encourages students to explore ideas and values that are important to them and relevant to the PPD theme. A confidential interview, based on the PPD goals, is held with a faculty member who has read the student's portfolio. RESULTS: In 1997/98, 96% of students agreed that they had engaged in useful reflection on their approach to the course and 91% agreed that the experience was worthwhile. A further 76% of students agreed that they could see opportunities to modify their approach in some ways as result of this exercise. CONCLUSION: Sustained PPD is essential in equipping doctors for the varied stresses of careers in medicine. Despite, or perhaps because of, the latitude in the Year 1 assessment, both students and faculty members found the process of value. This form of assessment acknowledges that the most valid assessment formats cannot always be made reliable and that in some parts of the curriculum it is more important to demonstrate trust in students' own motivation to become competent and mindful practitioners. The fact that the portfolio and interview are the only summative assessments in the first year emphasises the importance that the Faculty places on PPD.     

Videotaped vs traditional lectures for medical students

Closed-circuit television (CCTV) provides medical departments with alternatives in instructional formats. Concern, however, has been voiced about teaching via TV because the medium itself might cause inattention. This study investigated whether TV will lower the test scores of medical students. Sixty-one students were randomly divided into two groups. The lecture (control) group received the information via traditional lectures, including use of 2”× 2” transparencies. The video group received concurrently the same information via CCTV. Multiple-choice examinations were given after each of the six sessions. The cumulative mean scores were similar: lecture group = 87.56%, video group = 87.99%, i.e., no significant difference (P=0.77). To detect attitudinal differences toward the two formats, the students were surveyed at the end of the series and intragroup agreement on specific questions was calculated. The students rated the lecture format more highly. In response to the question ‘Did the audiovisual materials used by the instructor aid your learning?’, the mean rating from the lecture group was 7.37 (scale of 0–9) compared to a mean rating of 5.93 from the video group (P< 0.0003).     

Performance evaluation of graduates from a community-based curriculum: the housemanship period at Gezira

The housemanship performance of the first two classes of the University of Gezira Medical School was assessed on 107 occasions by using a questionnaire administered to the senior doctors (consultants) in the units in which they worked and the data, which covered 32% of all housemanship rotations taken by these graduates, were analysed. On a 5-point scale (poor to excellent) the rating of performance for 26 questions was average and above (S = 3: + S = 4: + S = 5) in 95% of instances. The good and excellent rating (S = 4 + S = 5) for the same questions was about 80%. The grand mean for rating of questions in the cognitive, psychomotor and attitudes domain was 4.1, 3.9 and 4.2 out of 5 respectively. The graduates were found to be better off (45%), comparable to (50%) and less than (5%) when compared to other graduates who worked with the same consultants in the past. Results of three general questions inquiring about graduates' practice safety, overall standing and consultants' preference for Gezira graduates had a 95% positive rating. It is concluded that these results provide significant positive answers for some basic questions concerning credibility which are often raised about Gezira and similar schools with innovative medical curricula.     

Paediatric option in MRCP examination: a survey into the results of candidates electing to take the paediatrics option in MRCP (UK) Part II

Five surveys on the performance of candidates in the paediatric option at MRCP(UK) Part II examination show the paediatric candidates to be at a disadvantage in the written part. The paediatric candidate is later in his career to obtain his MRCP (UK), as the mean interval from qualification to success at Part II is 41.7 months compared with 34 months for non-paediatric candidates (   P = 0.0026  ).
Lower marks were scored by paediatric candidates in the written section on the whole (   P = 0.055  ), slide identification (   P = 0.015  ), data interpretation (   P = 0.065  ) and non-paediatric case histories (   P = 0.023  ) as seen in those candidates who passed each part at first attempt, the discrepancy being wider in those having more attempts to pass. In the oral examination the paediatric candidate scored higher (   P = 0.07  ) but no significant difference in the clinical marks.
Although it appears that few paediatricians have been severely penalized by the present system, it does require the paediatric candidate to orientate his preparation for Part II towards his books rather than towards his clinical practice, which is contrary to the aim of the Part II examination of the MRCP(UK). Plans have been announced by the Presidents of the three Royal Colleges of Physicians of the United Kingdom for an entirely paediatric Part II.     

Multiple mini-interviews predict clerkship and licensing examination performance

OBJECTIVE: The Multiple Mini-Interview (MMI) has previously been shown to have a positive correlation with early medical school performance. Data have matured to allow comparison with clerkship evaluations and national licensing examinations. METHODS: Of 117 applicants to the Michael G DeGroote School of Medicine at McMaster University who had scores on the MMI, traditional non-cognitive measures, and undergraduate grade point average (uGPA), 45 were admitted and followed through clerkship evaluations and Part I of the Medical Council of Canada Qualifying Examination (MCCQE). Clerkship evaluations consisted of clerkship summary ratings, a clerkship objective structured clinical examination (OSCE), and progress test score (a 180-item, multiple-choice test). The MCCQE includes subsections relevant to medical specialties and relevant to broader legal and ethical issues (Population Health and the Considerations of the Legal, Ethical and Organisational Aspects of Medicine[CLEO/PHELO]). RESULTS: In-programme, MMI was the best predictor of OSCE performance, clerkship encounter cards, and clerkship performance ratings. On the MCCQE Part I, MMI significantly predicted CLEO/PHELO scores and clinical decision-making (CDM) scores. None of these assessments were predicted by other non-cognitive admissions measures or uGPA. Only uGPA predicted progress test scores and the MCQ-based specialty-specific subsections of the MCCQE Part I. DISCUSSION: The MMI complements pre-admission cognitive measures to predict performance outcomes during clerkship and on the Canadian national licensing examination.     

The West Midlands Matching Scheme 2000: a survey of participant satisfaction and outcomes

CONTEXT: The West Midlands Matching Scheme has been in operation since 1999 and is one of a number of schemes used for the allocation of pre-registration house officer positions in the UK. METHOD AND RESULTS: A questionnaire was distributed amongst candidates applying to the scheme in 2000. A response rate of 73.7% was obtained (160 replies from a sample population of 217). Half the responders reported to be dissatisfied with the explanation of the matching scheme given. A total of 66.6% were satisfied with the scheme being run via the Internet and E-mail. For 6-month positions in medicine, 85% of the candidates were happy with the job they had received. The figures for surgical jobs were similar and for rotations even higher. However, 23.8% of the sample was missing one or two jobs at the end of the matching process. This figure was still high when non-responders were taken into account. This group was not happy with the way in which the remaining jobs were distributed to them (71% dissatisfied). CONCLUSION: Although the matching scheme matched a high proportion of responders to jobs they were satisfied with, there was a significant group without jobs who were unhappy with the scheme. Some of those with jobs were also dissatisfied with the jobs they had received. Both problems should be addressed in future schemes. Further work in this area should examine the characteristics of the group without jobs and look at the success of other schemes in order to determine the optimal way in which to allocate PRHO jobs.     

Analysis of verbal interactions in tutorial groups: a process study

INTRODUCTION: Collaborative learning, including problem-based learning (PBL), is a powerful learning method. Group interaction plays a crucial role in stimulating student learning. However, few studies on learning processes in medical education have examined group interactions. Most studies on collaboration within PBL used self-reported data rather than observational data. We investigated the following types of interactions in PBL tutorial groups: learning-oriented interactions (exploratory questioning, cumulative reasoning and handling conflicts about knowledge); procedural interactions, and irrelevant/off-task interactions. AIM: The central question concerned how much time is spent on the different types of interaction during group sessions and how the types of interaction are distributed over the meeting. METHOD: Four tutorial group sessions in Year 2 of the PBL undergraduate curriculum of Maastricht Medical School were videotaped and analysed. The sessions concerned the reporting phase of the PBL process. We analysed the interactions using a coding scheme distinguishing several verbal interaction types, such as questions, arguments and evaluations. RESULTS: Learning-orientated interactions accounted for 80% of the interactions, with cumulative reasoning, exploratory questioning and handling conflicts about knowledge accounting for about 63%, 10% and 7% of the interactions, respectively. Exploratory questioning often preceded cumulative reasoning. Both types occurred throughout the meeting. Handling conflicts mainly occurred after the first 20 minutes. CONCLUSIONS: Task involvement in the tutorial groups was high. All types of learning-orientated interactions were observed. Relatively little time was spent on exploratory questions and handling conflicts about knowledge. Problem-based learning training should pay special attention to stimulating discussion about contradictory information.