Interrater reliability: completing the methods description in medical records review studies

In medical records review studies, information on the interrater reliability (IRR) of the data is seldom reported. This study assesses the IRR of data collected for a complex medical records review study. Elements selected for determining IRR included "demographic" data that require copying explicit information (e.g., gender, birth date), "free-text" data that require identifying and copying (e.g., chief complaints and diagnoses), and data that require abstractor judgment in determining what to record (e.g., whether heart disease was considered). Rates of agreement were assessed by the greatest number of answers (one to all n) that were the same. The IRR scores improved over time. At 1 month, the reliability for demographic data elements was very good, for free-text data elements was good, but for data elements requiring abstractor judgment was unacceptable (only 3.4 of six answers agreed, on average). All assessments after 6 months showed very good to excellent IRR. This study demonstrates that IRR can be evaluated and summarized, providing important information to the study investigators and to the consumer for assessing the reliability of the data and therefore the validity of the study results and conclusions. IRR information should be required for all large medical records studies.     

临床医师病历质量管理认知调查与分析

目的了解临床医师的病历质量管理需求,以便针对性改进。方法在文献研究基础上,制定调查问卷,对医院150名临床医师进行调查,通过比率分析,了解临床医师对病历质量管理的需求。结果临床医师对病历书写规范和病历质量检查标准的培训需求较大;对病历基本规则培训需求较高;认为病历质量重点培训对象为进修生,重点反馈对象为经治医师,重点检查病历为死亡病历;希望多接受“走动式”培训;希望将纳入千分制考核等作为主要奖惩方式。结论临床医师病历质量管理可从丰富培训形式,拓宽反馈途径,建立三级质控屏障,完善考核标准等4方面着手,从而更好地保障病历书写质量,确保医疗质量与安全。     

计算机监控7961份病案质量分析

目的通过计算机对出院病案的监控,并分析其结果及存在问题的原因,找出相应的解决方法,提高病案质量。方法运用计算机质控程序对7961份出院病案的质量进行监控,统计分析病案中存在的问题。结果病案的各部份均存在不同程度的问题,其中首页缺陷最多,占28.9%;医嘱缺陷最少,占4.2%。反映医生对病案重视不够。结论加强医务人员的法制观念,加强病历书写规范的培训,重视病案书写,上级医生做好环节质控,健全病案质控制度是提高病案质量的关键。     

开展精评活动提升病历内涵质量

目的 持续提升病历内涵质量。方法 设定病案首页填写、运行病历质控、终末病案质控、输血病历审核和培训整改5个指标,组织院级评价指导专家小组分批下科室进行病历质量精评专项检查,首次点评后的3个月、6个月、9个月再“回头看”。结果 经过初评和“回头看”抽取病历评价并进行前后比较,甲级病案率提升了26.67%,第2轮和第3轮“回头看”甲级病案率较第1轮均有统计学差异（P<0.05）。 结论 组织院内专家开展“精评病历”活动是提升病历质量的有效方法,操作性强,管理成本低,应继续坚持。     

Interrater reliability of measurements of comorbid illness should be reported

OBJECTIVE: Comorbidity indices are commonly used to stratify patients to control for treatment selection bias. The objectives here were to review the reporting of interrater reliability when studies use comorbidity indices in clinical research publications and to report the interrater reliability of four common indices in a particular research setting. STUDY DESIGN AND SETTING: Four trained abstractors reviewed the same 40 charts of patients with squamous cell carcinoma of the head and neck from a regional cancer center. Scores for the Charlson Index, the Index of Co-existent Disease, the Cumulative Illness Rating Scale, and the Kaplan-Feinstein Classification were calculated, and the intraclass correlation coefficient was used to assess interrater reliability. RESULTS: The details on the training of abstractors and the results of interrater reliability tests are not commonly reported. In our study setting, the Charlson Index had excellent reliability and the others had acceptable reliability. CONCLUSION: If the quality of a study using an index or scale is to be assessed, the reliability and interrater reliability of the score assignment process should be reported.     

Assessing Reliability of Medical Record Reviews for the Detection of Hospital Adverse Events

Objectives:

The purpose of this study was to assess the inter-rater reliability and intra-rater reliability of medical record review for the detection of hospital adverse events.

Methods:

We conducted two stages retrospective medical records review of a random sample of 96 patients from one acute-care general hospital. The first stage was an explicit patient record review by two nurses to detect the presence of 41 screening criteria (SC). The second stage was an implicit structured review by two physicians to identify the occurrence of adverse events from the positive cases on the SC. The inter-rater reliability of two nurses and that of two physicians were assessed. The intra-rater reliability was also evaluated by using test-retest method at approximately two weeks later.

Results:

In 84.2% of the patient medical records, the nurses agreed as to the necessity for the second stage review (kappa, 0.68; 95% confidence interval [CI], 0.54 to 0.83). In 93.0% of the patient medical records screened by nurses, the physicians agreed about the absence or presence of adverse events (kappa, 0.71; 95% CI, 0.44 to 0.97). When assessing intra-rater reliability, the kappa indices of two nurses were 0.54 (95% CI, 0.31 to 0.77) and 0.67 (95% CI, 0.47 to 0.87), whereas those of two physicians were 0.87 (95% CI, 0.62 to 1.00) and 0.37 (95% CI, -0.16 to 0.89).

Conclusions:

In this study, the medical record review for detecting adverse events showed intermediate to good level of inter-rater and intra-rater reliability. Well organized training program for reviewers and clearly defining SC are required to get more reliable results in the hospital adverse event study.     

HIPAA Transition: Challenges of a Multisite Medical Records Validation Study of Maternally Linked Birth Records

Numerous researchers have expressed concern over the impacts on medical records availability of the newly effective Medical Information Privacy rule, as authorized by the Health Insurance Portability and Accountability Act (HIPAA). The increased costs associated with compliance with the rule, and the increased potential for financial liability, raises the possibility that hospitals may be less likely to participate in such research, resulting in a decrease of the validity of multisite studies designed to represent an entire population. Our multisite medical record validation study, designed to assess the accuracy of maternally linked birth records, provides an overview of a number of HIPAA implementation challenges. We found that the new HIPAA rule presents new challenges for those who rely on the release of medical record information for epidemiologic research. At the very minimum, increased compliance costs associated with human subjects protection and increased administrative burden for researchers would seem to be inevitable as medical institutions address the requirements of the new HIPAA rule by instituting more complex and thus more cumbersome procedures. Researchers should anticipate increased costs and plan accordingly when budgeting for human subjects review processes.     

Methods to achieve high interrater reliability in data collection from primary care medical records

PURPOSE We assessed interrater reliability (IRR) of chart abstractors within a randomized trial of cardiovascular care in primary care. We report our findings, and outline issues and provide recommendations related to determining sample size, frequency of verification, and minimum thresholds for 2 measures of IRR: the κ statistic and percent agreement.METHODS We designed a data quality monitoring procedure having 4 parts: use of standardized protocols and forms, extensive training, continuous monitoring of IRR, and a quality improvement feedback mechanism. Four abstractors checked a 5% sample of charts at 3 time points for a predefined set of indicators of the quality of care. We set our quality threshold for IRR at a κ of 0.75, a percent agreement of 95%, or both.RESULTS Abstractors reabstracted a sample of charts in 16 of 27 primary care practices, checking a total of 132 charts with 38 indicators per chart. The overall κ across all items was 0.91 (95% confidence interval, 0.90–0.92) and the overall percent agreement was 94.3%, signifying excellent agreement between abstractors. We gave feedback to the abstractors to highlight items that had a κ of less than 0.70 or a percent agreement less than 95%. No practice had to have its charts abstracted again because of poor quality.CONCLUSIONS A 5% sampling of charts for quality control using IRR analysis yielded κ and agreement levels that met or exceeded our quality thresholds. Using 3 time points during the chart audit phase allows for early quality control as well as ongoing quality monitoring. Our results can be used as a guide and benchmark for other medical chart review studies in primary care.     

Improving efficiency of stroke research: the Brain Attack Surveillance in Corpus Christi study

We studied whether a computer algorithm or abstractor could diagnose stroke as well as a fellowship-trained stroke neurologist. As part of an ongoing prospective, community-based stroke surveillance project, a diagnostic algorithm was developed, and patients' neurologic signs and symptoms were collected in a computerized database. The abstractors were blinded to the results of this algorithm and were asked to verify whether the patient had a stroke. The separate results of the computer and abstractor were compared with the final diagnosis given by the blinded neurologist. From 1 January through 31 July 2000, 3418 cases were screened. The abstractors yielded sensitivity 91%, specificity 97%, positive predictive value (PPV) 85%, and negative predictive value (NPV) 99%. Three computer algorithms were evaluated. The sensitivities ranged from 83% to 96%, specificity ranged from 88% to 97%, PPV ranged from 54% to 81%, and NPV ranged from 97% to 99%. The use of computer verification or abstractors may obviate the need for physician stroke verification and may greatly improve study efficiency.     

手术过程质控对围手术期病历质量改进效果分析

目的 评价手术过程质控对围手术期病历质量改进的效果.方法 抽查2019年10％的围手术期病历进行质控,对比分析实施手术过程质控干预前后的病历质量缺陷.结果 干预组的授权委托书、知情告知书、术前小结、术前讨论、术者查房记录、麻醉访视记录、手术安全核查、手术记录、术后首程、麻醉记录、手术审批11项病历质量缺陷率均低于对照组...     

Methodological Innovations in Data Gathering: Newborn Screening Linkage with Live Births Records, Michigan, 1/2007–3/2008

Objective To match Michigan birth and newborn screening records to identify and follow-up potentially unscreened infants, assess data quality, and demonstrate the utility of Link Plus linkage software for matching MCH related administrative datasets. Methods Birth and newborn screening records maintained by the Michigan Department of Community Health from January 2007 through March 2008 were used in this study. Link Plus, a freely-available probabilistic record linkage software program developed at the Centers for Disease Control and Prevention, was used to match records. Linkage performance was assessed by the linkage success rate (percentage of valid matches). Follow-up of un-matched records was conducted by the Michigan Newborn Screening Follow-up Program. Results Nearly all (99.2%) of the 142,178 birth records included in this study were successfully matched to newborn screening records. Following a transition to a web-based electronic birth certificate system and inclusion of a newborn screening card identification number on the birth record in 2008, the linkage success rate increased to 99.6% based on analysis of approximately 18,000 records. Of approximately 600 un-matched records, nearly half had received a newborn screen. Approximately 8% of un-matched records were due to parental refusal of newborn screening. Nine children received an initial screen as a result of this study; one was confirmed as having sickle cell trait. Conclusions We have demonstrated that a freely available record linkage software, Link Plus, can be used to successfully match records of MCH databases thereby providing an opportunity for further research and quality assurance investigations.     

基于医院评审的病历质量缺陷分析与对策建议

依据病历质量评分标准,对2035份缺陷病历进行分析.病历质量缺陷发生率为13.78％.质量意识淡薄,书写不及时,复制及漏缺项严重等是病历质量缺陷的主要原因.提出端正态度,提高医师自觉意识;分类培训,强化病历内涵质量要求;加强监管,健全分层管理联动机制;定期评比,奖优罚劣等对策建议.     

Comparative hospital databases: value for management and quality.

OBJECTIVES--To establish an accurate and reliable comparative database of discharge abstracts and to appraise its value for assessments of quality of care. DESIGN--Retrospective review of case notes by trained research abstractors and comparison with matched information as routinely collected by the hospitals' own information systems. SETTING--Three district general hospitals and two major London teaching hospitals. PATIENTS--The database included 3905 medical and surgical cases and 2082 obstetric cases from 1990 and 1991. MAIN MEASURES--Accessibility of case notes; measures of reliability between reviewers and of validity of case note content; application of high level quality indicators. RESULTS--The existing hospital systems extracted insufficient detail from case notes to conduct clinical comparative analyses for medical and surgical cases. The research abstractors at least doubled the diagnostic codes extracted. Interabstractor agreement of about 70% was obtained for primary diagnosis and assignment to diagnosis related group. These data were sufficient to create a comparative database and apply high level quality indicators designed to flag topics for further study. For obstetric-specific indicators the rates were comparable for abstractors and the hospital information systems, which in each case was a departmentally based system (SMMIS) producing more detailed and accessible data. CONCLUSIONS--Current methods of extracting and coding diagnostic and procedural data from case notes in this sample of hospitals is unsatisfactory: notes were difficult to access and recording is unacceptably incomplete. IMPLICATIONS--Improvements as piloted in this project, are readily available should the NHS, hospital managers, and clinicians see the value of these data in their clinical and managerial activities.     

Patterns of consent in epidemiologic research: evidence from over 25,000 responders

Ethical guidelines in the United Kingdom require written consent from participants in epidemiologic studies for follow-up or review of medical records. This may cause bias in samples used for follow-up or medical record review. The authors analyzed data from seven general population surveys conducted in the United Kingdom (1996-2002), to which over 25,000 people responded. Associations of age, gender, and symptom under investigation with consent to follow-up and consent to review of medical records were examined. Consent to follow-up was approximately 75-95% among survey responders under age 50 years but fell among older people, particularly females. Consent to follow-up was also higher among responders who had the symptom under investigation (pooled odds ratio = 1.61, 95% confidence interval: 1.36, 1.92). Consent to review of medical records followed a similar pattern. Patterns of consent were relatively consistent and represented a high proportion of responders. Males, younger people, and subjects reporting the symptom under investigation were more likely to give consent, and these groups may be overrepresented in follow-up samples or reviews of medical records. Although consent is high among responders, the additive effect of nonresponse and nonconsent can substantially reduce sample size and should be taken into account in epidemiologic study planning.     

日间手术病历质控难点及应对策略

通过积极学习相关政策法规,结合医院自身特点,制定和维护日间手术病历模板,加强日间手术病历书写培训,建立日间手术病历监控平台,多渠道反馈质控信息,多措并举,促进病历质量提升。同时,在工作中应及时发现日间手术病历质控中存在问题,结合实际,分析原因,制定改进措施,把好日间手术病历质量关,为日间手术顺利开展提供了重要保障。     

Assessment of level of care: implications of interrater reliability on health policy.

In Wisconsin, level-of-care assessments are used to set Medicaid reimbursement and determine nursing home eligibility. This study examined three methods of assessing level of care: 1) the Wisconsin quality assurance project (QAP) method, based on observations of patients, patient records, and staff interviews; 2) the Wisconsin standard (STD) method, based primarily on a clinical record review; and, 3) an adaptation of New York's "DMS-I," a checklist with numerical weights used to set level of care. Results address interrater reliability, the agreement between assessments by research teams and actual levels of care set by the State, and the implications that agreement has for reimbursement.     

火神山医院首次出院病案书写质量结果分析

目的对火神山医院出院病案质量检查结果进行分析,为COVID-19收容医院的医疗质量管理提供改进参考。方法收集火神山医院2020年2月4—24日290份出院病案,采用医院自制的《火神山医院病历书写质量检查评定表》对所有出院病案进行检查。对整体质量、条目质量、缺陷聚集特征等进行分析。结果290份病案平均得分为95.87±2.35分,甲级病案率为90.69%。普通病区与重症病区病案的等级分布差别显著(χ^2=85.62,P=0.000)。低于90分的病历27份,占9.31%,分布有一定的科室聚集性。要素检查整体合格率为80.44%±6.21%,内涵检查整体合格率为82.79%±13.52%,两部分合格率无显著性差别。病案检查主要扣分项目前几位分别为要素检查中的病历首页完整性和内涵检查的三级检诊制度落实、与家属沟通告知的相关记录,以及出入院诊断的规范性。医嘱记录、定期对患者病情分析评估与病程内容记录的个性化和准确性属于减分最少的几项。此次检查的290份病历中,未出现可推广借鉴的有益做法的相关记录。结论病案的完整性和准确性是COVID-19专病医院救治效果回顾性分析的重要依据,也是治疗技术后续研究的重要参考,对出院病案的阶段检查、深入分析病案质量指标能及时了解医疗过程中的不足,针对检出结果及时调整质量控制方法,是保证医院改进和提升医疗质量的重要手段。     

Accuracy of healthcare worker recall and medical record review for identifying infectious exposures to hospitalized patients.

OBJECTIVE: To determine the validity of using healthcare worker (HCW) recall of patient interactions and medical record review for contact tracing in a critical care setting. DESIGN: Trained observers recorded the interactions of nurses, respiratory therapists, and service assistants with study patients in a medical-surgical intensive care unit. These observers' records were used as the reference standard to test the criterion validity of using HCW recall data or medical record review data to identify exposure characteristics. We assessed the effects of previous quarantine of the HCW (because of possible exposure) and the availability of patients' medical records for use as memory aids on the accuracy of HCW recall. SETTING: A 10-bed medical-surgical intensive care unit at Mount Sinai Hospital in Toronto, Ontario. PATIENTS: Thirty-six HCWs observed caring for 16 patients, for a total of 55 healthcare worker shifts. RESULTS: Recall accuracy was better among HCWs who were provided with patient medical records as memory aids (P<.01). However, HCWs tended to overestimate exposures when they used patient medical records as memory aids. For 6 of 26 procedures or care activities, this tendency to overestimate was statistically significant (P<.05). Most HCWs with true exposures were identified by means of this technique, despite the overestimations. Documentation of the activities of the 4 service assistants could not be found in any of the patients' medical records. Similarly, the interactions between 6 (19%) of 32 other patient-HCW pairs were not recorded in patients' medical records. CONCLUSIONS: Data collected from follow-up interviews with HCWs in which they are provided with patient medical records as memory aids should be adequate for contact tracing and for determining exposure histories. Neither follow-up interviews nor medical record review alone provide sufficient data for these purposes.     

正确书写病案首页对提高电子病历质量的作用

目的规范病案首页的填写,提高电子病历质量。方法对我院病案首页填写存在的问题进行总结分析,提出改进措施。结果通过对病案首页书写存在的问题及时反馈和整改,为规范医疗文书书写起到了促进作用。结论加强病历首页正确书写,是提高电子病历质量的重要环节。     

某院出院病历质量调查

目的：了解某医院出院病历质量现状,分析缺陷成因,探讨管理对策,为医院病历质量管理提供参考。方法2015年1—12月出院病历共96842份,分层随机抽取15086份进行质控,对质控结果进行汇总分析。结果甲级病案率为96．7％,病历质量总体处于优良水平。存在问题的病历占44．75％,手术科室缺陷率高于非手术科室。缺陷10212项,缺陷项前三位分别为上级医师首次查房记录缺陷、首次病程记录缺陷、入院记录缺陷。结论落实病历质控制度化、常态化、精细化管理,是强化病历质量标准,进一步规范病历书写,提高病历内涵质量的有效措施;通过病历缺陷分析、讲评,强化主干医疗制度的落实,是促进医疗质量持续改进的有力抓手。