A certified de-identification system for all clinical text documents for information extraction at scale

Abstract Objectives Clinical notes are a veritable treasure trove of information on a patient’s disease progression, medical history, and treatment plans, yet are locked in secured databases accessible for research only after extensive ethics review. Removing personally identifying and protected health information (PII/PHI) from the records can reduce the need for additional Institutional Review Boards (IRB) reviews. In this project, our goals were to: (1) develop a robust and scalable clinical text de-identification pipeline that is compliant with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule for de-identification standards and (2) share routinely updated de-identified clinical notes with researchers. Materials and Methods Building on our open-source de-identification software called Philter, we added features to: (1) make the algorithm and the de-identified data HIPAA compliant, which also implies type 2 error-free redaction, as certified via external audit; (2) reduce over-redaction errors; and (3) normalize and shift date PHI. We also established a streamlined de-identification pipeline using MongoDB to automatically extract clinical notes and provide truly de-identified notes to researchers with periodic monthly refreshes at our institution. Results To the best of our knowledge, the Philter V1.0 pipeline is currently the first and only certified, de-identified redaction pipeline that makes clinical notes available to researchers for nonhuman subjects’ research, without further IRB approval needed. To date, we have made over 130 million certified de-identified clinical notes available to over 600 UCSF researchers. These notes were collected over the past 40 years, and represent data from 2757016 UCSF patients. Lay Summary Clinical notes and reports from routine patient care contain large amounts of clinically relevant information valuable for research like detailed diagnosis and treatment plans, over the counter medication usage, patient’s diet, and physical activity. Patient-level data are imperative to leverage for research, but access is restricted due to personal identifying (PII) and protected health information (PHI) content. Here we demonstrate how to anonymize the textual data by removing all PII/PHI following an externally certified protocol. The steps of the certification process and the detailed methodological enhancements are described. After many iterations of development and validation, the data were certifi