Ghosts in the Machine: How "Objective" Medical Tech Encodes Human Bias
We naturally want to believe that numbers, clinical devices, and software code are objective. In a medical setting, relying on physical hardware and digital algorithms promises to remove human error and subjective bias from clinical decisions. Yet, technology does not exist in a vacuum. Devices and algorithms are designed, trained, and tested by human beings, meaning they inevitably inherit the structural flaws and blind spots of the societies that built them. When we uncritically trust these tools, we risk automating and amplifying health inequities under the guise of objective science.
A clear example of hardware bias is found in the common pulse oximeter. On every hospital floor, this clip-on device provides a non-invasive reading of a patient’s blood oxygen levels. However, its original design and calibration relied predominantly on light-skinned study participants. Because light absorption varies across different skin pigments, the device performs less accurately on patients with darker skin tones. In a pivotal study published in The New England Journal of Medicine, Sjoding et al. (2020) demonstrated that pulse oximeters were nearly three times as likely to miss occult hypoxemia, which is dangerously low arterial oxygen saturation, in Black patients compared to White patients (Sjoding et al., 2020). When a device overestimates oxygen levels, clinicians may unknowingly delay critical interventions, oxygen therapy, or hospital admissions.
Beyond physical hardware, digital algorithms and predictive software carry similar structural flaws. Hospitals increasingly rely on algorithms to manage patient care, allocate resources, and identify high-risk individuals who need extra support. However, the data fed into these algorithms often reflects historical disparities. In a landmark study published in Science, Obermeyer et al. (2019) evaluated a commercial algorithm applied to millions of patients. To predict future health needs, the algorithm used past healthcare expenditure as a proxy metric (Obermeyer et al., 2019). Because structural barriers cause less money to be spent on Black patients on average, the algorithm misinterpreted lower spending as better health. As a result, Black patients had to be considerably sicker than White patients to receive the same risk score and access to specialized care management. Recent clinical research in JAMA further highlights that even when clinicians are made aware of these algorithmic biases, their diagnostic accuracy can still suffer if the underlying models remain flawed (Ratwani et al., 2024).
The risk of automating disparity has become even more acute with the arrival of generative AI and large language models in clinical practice. A comprehensive 2026 scoping review published in Nature’s Communications Health analyzed 92 studies across clinical specialties and revealed that large AI models exhibit systematic bias across clinical decision support, medical documentation, patient communication, and medical education (Xiao et al., 2026). The review underscored that medical bias in AI is not an isolated technical failure, but an intersectional challenge arising when training data reflects embedded healthcare inequities. Without unified fairness standards and rigorous real-world validation, deploying AI systems risks reinforcing racial, sex, and socioeconomic disparities across fields from cardiology to emergency medicine.
These examples illustrate that health inequities are not only produced by individual clinical choices or systemic policies, but are also coded directly into our technology. Addressing these gaps requires more than technological fixes. It demands a humanistic approach to medicine. Clinicians, researchers, and developers must critically examine how diagnostic tools are tested, what proxy variables software uses, and whose data is missing from the baseline. True equity in healthcare starts when we stop treating medical devices and AI models as neutral authorities and begin questioning the structural assumptions built into the tools we rely on every day.
References
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342
Ratwani, R. M., Sutton, K., & Galarraga, J. E. (2024). Addressing AI algorithmic bias in health care. JAMA, 332(13), 1051–1052. https://doi.org/10.1001/jama.2024.13486
Sjoding, M. W., Dickson, R. P., Iwashyna, T. J., Gay, S. E., & Valley, T. S. (2020). Racial bias in pulse oximetry measurement. The New England Journal of Medicine, 383(25), 2477–2478. https://doi.org/10.1056/NEJMc2029240
By Kiah Nirmal (she/her) | Blog Committee Member
