A unified framework for evaluating the risk of re-identification of text de-identification tools |
| |
Affiliation: | 1. School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada;2. Privacy Analytics Inc., Ottawa, Canada;3. Department of Pediatrics, University of Ottawa, Ottawa, Canada;4. Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada;5. Michigan Institute for Data Science (MIDAS), University of Michigan Medical School, Office of Research, Ann Arbor, United States;6. Department of Pediatrics, University of Michigan, Ann Arbor, United States |
| |
Abstract: | ObjectivesIt has become regular practice to de-identify unstructured medical text for use in research using automatic methods, the goal of which is to remove patient identifying information to minimize re-identification risk. The metrics commonly used to determine if these systems are performing well do not accurately reflect the risk of a patient being re-identified. We therefore developed a framework for measuring the risk of re-identification associated with textual data releases.MethodsWe apply the proposed evaluation framework to a data set from the University of Michigan Medical School. Our risk assessment results are then compared with those that would be obtained using a typical contemporary micro-average evaluation of recall in order to illustrate the difference between the proposed evaluation framework and the current baseline method.ResultsWe demonstrate how this framework compares against common measures of the re-identification risk associated with an automated text de-identification process. For the probability of re-identification using our evaluation framework we obtained a mean value for direct identifiers of 0.0074 and a mean value for quasi-identifiers of 0.0022. The 95% confidence interval for these estimates were below the relevant thresholds. The threshold for direct identifier risk was based on previously used approaches in the literature. The threshold for quasi-identifiers was determined based on the context of the data release following commonly used de-identification criteria for structured data.DiscussionOur framework attempts to correct for poorly distributed evaluation corpora, accounts for the data release context, and avoids the often optimistic assumptions that are made using the more traditional evaluation approach. It therefore provides a more realistic estimate of the true probability of re-identification.ConclusionsThis framework should be used as a basis for computing re-identification risk in order to more realistically evaluate future text de-identification tools. |
| |
Keywords: | De-identification Re-identification risk Medical text Evaluation framework Natural language processing Data sharing |
本文献已被 ScienceDirect 等数据库收录! |
|