AI in Healthcare: A Risky Experiment for Low-Income Patients

In southern California, a private company called Akido Labs is operating clinics aimed at serving unhoused individuals and those with low incomes. Patients are seen by medical assistants who utilize artificial intelligence (AI) to listen to conversations and generate potential diagnoses and treatment plans, which are subsequently reviewed by a doctor. According to the company’s chief technology officer, the goal is to “pull the doctor out of the visit.” This trend raises significant concerns, particularly for vulnerable populations who already face barriers to healthcare.

The increasing integration of AI in healthcare is not isolated. A survey conducted by the American Medical Association in 2025 revealed that approximately two-thirds of physicians utilized AI to assist in their practices, including during patient diagnoses. Notably, one AI startup secured $200 million to develop an app likened to “ChatGPT for doctors.” Lawmakers in the United States are even considering legislation that would allow AI to prescribe medication. While these advancements in AI may enhance efficiency for some, they pose a unique threat to low-income patients who are often already disadvantaged in the healthcare system.

Low-income individuals frequently encounter significant obstacles when accessing care. Systematic issues such as overcrowded hospitals, overworked clinicians, and a profit-driven healthcare framework exacerbate these challenges. Economically disadvantaged communities typically have under-resourced healthcare facilities and higher rates of chronic illnesses, often linked to factors like racism and poverty.

Concerns about AI’s accuracy in diagnostics are further compounded for patients from marginalized backgrounds. A study published in Nature Medicine in 2021 assessed AI algorithms trained on extensive chest X-ray datasets. It found that these algorithms inadequately diagnosed Black and Latinx patients, as well as women and those with Medicaid insurance. This bias not only risks misdiagnoses but could worsen existing health inequities.

For instance, a 2024 study discovered that AI misdiagnosed breast cancer screenings more frequently among Black patients, highlighting the disparities in healthcare technology. Such findings underline the reality that AI systems depend on existing data, which may reflect historical biases, leading to detrimental consequences for those already facing health disparities.

Patients are often left uninformed about the extent to which AI influences their healthcare. A medical assistant working with Akido Labs shared with the MIT Technology Review that while patients are aware of AI listening in, they are not informed that it makes diagnostic recommendations. This lack of transparency echoes historical injustices in medicine, where marginalized communities were often subjected to exploitative practices without consent.

While AI may promise efficiency, it comes at the potential cost of diagnostic accuracy and exacerbating health inequities. An advocacy group, TechTonic Justice, recently published a report estimating that over 92 million Americans with low incomes have crucial aspects of their lives influenced by AI, including Medicaid benefits and Social Security disability eligibility.

Legal battles are already highlighting the ramifications of AI decision-making in healthcare. In March 2023, a group of Medicare Advantage customers in Minnesota filed a lawsuit against UnitedHealthcare, claiming their coverage was unjustly denied due to the company’s AI system, nH Predict, which erroneously classified them as ineligible for care. Some plaintiffs are the estates of deceased patients who allegedly died as a result of denied medically necessary treatments. A judge ruled that the case can proceed, signaling a growing scrutiny of AI in health coverage decisions.

Another similar case is unfolding in Kentucky against Humana, where plaintiffs allege that the use of nH Predict led to generic recommendations based on insufficient medical records. Both cases reflect a worrying trend of relying on AI to determine healthcare access for low-income individuals, raising questions about the fairness and accuracy of such systems.

The disparity in healthcare access is stark: individuals with financial means can typically obtain quality care, while those who are unhoused or low-income are at risk of being denied necessary services due to flawed AI systems. This phenomenon exemplifies a form of medical classism that should not be tolerated.

Implementing AI in healthcare settings requires a careful approach that prioritizes patient needs, especially for vulnerable populations. The focus should be on providing patient-centered care delivered by healthcare professionals who are equipped to listen to and address the specific health-related concerns of individuals.

It is critical to avoid creating an environment where AI technologies, developed by private companies, dominate healthcare decision-making. A system that allows AI to dictate medical treatments without rigorous evaluation from the communities affected disempowers patients, stripping them of their ability to influence the technologies applied to their care.

In summary, the drive toward integrating AI into healthcare must be approached with caution, particularly for low-income and unhoused individuals. Their voices and priorities should guide any implementation of AI, ensuring that technology serves to enhance, rather than undermine, their access to quality healthcare.