The healthcare industry is pushing back against proposals to make doctors and hospitals responsible for ensuring the safety of generative AI tools used in medical devices. Instead, provider groups say developers should bear that responsibility, arguing that monitoring these tools would create an unfair burden—especially for smaller practices—and expose providers to greater legal risk.
Why providers reject accountability
In comments submitted to the Food and Drug Administration, the American Hospital Association (AHA), American Medical Association (AMA), and American Telemedicine Association (ATA) argued that developers, not healthcare providers, should oversee the performance of AI-powered medical tools. The groups noted that many of these systems, such as chatbots or predictive analytics, are complex and often lack transparency in how they function.
“Manufacturers should remain responsible for the safety and performance of the devices they sell,” the ATA’s lobbying arm stated in its submission. Providers fear that taking on this role would increase their liability in malpractice cases, while tech companies could face their own negligence claims. Neither side wants to absorb the responsibility for potential errors.
Hospitals and health systems already have governance frameworks in place to evaluate AI-backed devices, but they lack the technical expertise to assess the underlying models. Without clear visibility into how these tools make decisions, providers say they cannot reliably detect flaws or predict failures.
FDA’s risk framework faces pushback
The FDA’s proposed approach to regulating generative AI devices focuses on assessing risk based on the actions a tool performs. For example, a device that provides non-urgent predictions, like a future cardiovascular risk, would be classified as low-risk, while one that makes autonomous decisions could pose severe consequences.
Related Post: Risant Health to Acquire Integris Health, Oklahoma’s Largest System
However, provider groups argue the framework needs refinement. The AHA pointed out that some activities carry multiple risks, such as AI tools recommending emergency care. These tools could either incorrectly flag non-emergency cases or fail to identify true emergencies, leading to harmful outcomes. The AHA suggested expanding risk assessments beyond just the tool’s actions to include factors like error likelihood and potential harm from misuse.
The AMA also questioned the effectiveness of human-in-the-loop safeguards, where clinicians review AI suggestions. The group warned that reliance on these tools could lead to oversight failures, either through clinician error or overdependence on AI. The AMA recommended that risk assessments should also evaluate the human element, not just the technology itself.
Some provider groups, including the Workgroup for Electronic Data Interchange (WEDI), proposed additional safeguards. They urged the FDA to create a structured process for reporting adverse events and to develop “nutrition labels”, similar to those proposed (but later rescinded) by the Biden administration, to help patients and clinicians understand a tool’s functions and risks.
Generative AI tools differ from traditional AI in healthcare because they accept open-ended inputs and produce varied outputs for similar prompts. This unpredictability makes oversight more challenging, and providers say clearer regulations are needed to ensure trust in these systems.
FDA moves toward clearer guidance
Trust in AI-driven medical tools is critical for adoption, and physicians have repeatedly called for stronger oversight. They want devices that are validated by a trusted authority, continuously monitored, and accompanied by a clear liability framework. The FDA appears to be responding: Rick Abramson, director of the agency’s Digital Health Center of Excellence, announced at a recent conference that draft guidance on regulating generative AI medical devices will be released in the coming months.
