What is patient confidentiality for? The Hippocratic Oath, considered one of the earliest and most widely known medical ethics texts in the world, reads: “I will keep secret whatever I see or hear in the lives of my patients, whether in relation to my professional practice or not, which should not be talked about outside, for I regard all such things as private.”
As privacy becomes increasingly rare in the age of data-hungry algorithms and cyberattacks, medicine is one of the few remaining domains where privacy remains central to practice, helping patients trust their physicians with sensitive information.
But a paper co-authored by MIT researchers examines how artificial intelligence models trained on de-identified electronic health records (EHRs) can remember patient-specific information. The work, which was recently presented at the 2025 Conference on Neural Information Processing Systems (NeurIPS), recommends a rigorous testing setup to ensure that targeted signals cannot reveal information, emphasizing that leakage should be evaluated in the health care context to determine whether it meaningfully compromises patient privacy.
Foundation models trained on EHRs typically must generalize knowledge to make better predictions based on multiple patient records. But in “remembering,” the model uses a single patient record to produce its output, potentially violating patient privacy. In particular, Foundation models are already known to be at risk for data leaks.
“The knowledge in these high-potential models can be a resource for many communities, but adversarial attackers can motivate a model to extract information on the training data,” says Sanna Tonkaboni, a postdoc at the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard and first author of the paper. Given the risk that foundation models may also remember private data, she says, “This work is a step toward ensuring that our community can take practical evaluation steps before releasing models.”
To research the potential risk of the EHR Foundation model in medicine, Tonkaboni contacted MIT Associate Professor Marzeh Ghassemi, a principal investigator at the Abdul Latif Jameel Clinic for Machine Learning in Health (Jameel Clinic), a member of the Computer Science and Artificial Intelligence Lab. Ghasemi, a faculty member in MIT’s Department of Electrical Engineering and Computer Science and the Institute for Medical Engineering and Science, runs the Healthy ML group, which focuses on robust machine learning in health.
How much information does a bad actor need to expose sensitive data, and what are the risks associated with leaked information? To assess this, the research team developed a series of tests that they hope will form the basis for future privacy assessments. These tests are designed to assess their practical risk to patients by measuring a variety of uncertainties and varying levels of attack probability.
“We’ve really tried to emphasize practicality here; if an attacker has to know the date and value of a dozen lab tests from your records to extract information, the risk of loss is very low. If I already have access to that level of protected source data, why would I need to attack the larger Foundation model for more?” Ghassemi says.
With the inevitable digitization of medical records, data breaches have become common. Over the past 24 months, the US Department of Health and Human Services has recorded 747 data breaches of health information affecting more than 500 individuals, the majority of which are classified as hacking/IT incidents.
Patients with specific conditions are particularly vulnerable, given how easy it is to pick them out. “Even with de-identified data, it depends on what type of information you leak about the person,” Tonkaboni says. “Once you recognize them, you know a lot.”
In their structured tests, the researchers found that the more information an attacker had about a particular patient, the more likely the model was to leak information. They demonstrated how to separate model generalization cases from patient-level memorization to appropriately assess privacy risk.
The paper also emphasized that some leaks are more harmful than others. For example, a model disclosing a patient’s age or demographics may be viewed as a more benign leak than a model disclosing more sensitive information such as an HIV diagnosis or alcohol abuse.
Researchers say patients with unique conditions are especially vulnerable because they are so easy to spot, which may require a higher level of protection. “Even with de-identified data, it really depends on what kind of information you leak about the person,” Tonkaboni says. The researchers plan to make the work more interdisciplinary by connecting physicians and privacy experts as well as legal experts.
“There’s a reason our health data is private,” Tonkaboni says. “There’s no reason for others to know about it.”
This work is supported by the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard, Wallenberg AI, the Knut and Alice Wallenberg Foundation, the US National Science Foundation (NSF), a Gordon and Betty Moore Foundation award, a Google Research Scholar award, and the AI2050 program at Schmidt Sciences. The resources used in preparing this research were provided, in part, by the Province of Ontario, Government of Canada, by the companies sponsoring CIFAR and the Vector Institute.
