How the FDA Is Framing Evidence for Generative AI Medical Devices

A generative AI tool for clinicians can write a visit summary for the physician to edit, or it can tell the nurse which patient to see next. The FDA’s August discussion paper on generative AI-enabled medical devices makes that difference one of two measures that would size a device’s evidence, so the team choosing what version one does is also choosing the size of its evidence plan.

The Digital Health Center of Excellence inside CDRH published the paper on August 18, and public comments closed on October 19. The agency presents it as a discussion paper that proposes no policy, though it shows how reviewers are framing devices built on large generative models.

Two axes of risk

The proposed risk framework asks two questions. The first is what the function does: provide information, direct an action, or take the action itself, with more or less human oversight. The second is how serious the harm would be if a user relied on a wrong output. Risk climbs along both.

The paper defines the first axis broadly. An output becomes more directive as it gets more specific, more personalized and more action-oriented, even if it never says “recommend” or “should.” Adding “talk to your doctor” may not move it back down the scale. The paper also names settings where an output can slide toward directing: patient-facing tools, generalists using a tool built with specialists in mind, and multi-turn conversations that start as information and end as instructions.

Who the product is sold to and what its screen shows decide which of those settings applies. A summary drafted for a physician to edit sits lower on the scale than a ranked patient list handed to a nurse, and the same summary shown directly to a patient may sit higher. Decide which version ships first and write down why, because the evidence plan follows from that choice.

Evidence modeled on a residency

For premarket evaluation, the paper borrows from how physicians are trained and credentialed: structured tests of core skills first, then confirmation in real practice. The device equivalent is a competency assessment of the finished product as it would be deployed, with its model, prompts and guardrails evaluated together.

The first part is benchmarking. It would check whether the device behaves safely, gets the clinical task right, communicates clearly, and holds up on cases unlike the ones it was tuned on. Agentic systems would also be tested on planning and carrying out tasks with several steps. The sponsor would fix the test set, the scoring rubric, the expert graders and the pass criteria before running anything.

The second part is clinical confirmation in real or representative conditions. The paper lists options ranging from retrospective runs on real patient inputs and shadow deployment to clinician review of real cases and a prospective study, and it does not suggest every device needs a prospective trial.

A team can build the benchmark half now. A rubric, a named panel of graders and pass criteria written before the first results come back give the team an evaluation it can rerun every time the product changes.

A foundation model is a supplier

A rerunnable benchmark matters because a generative device keeps changing after launch. The paper lists the parts that move: the model, the prompts, the retrieval strategy, the guardrails. It floats re-benchmarking against the premarket baseline after a change. It also concedes that the FDA’s existing tool for pre-approved changes, the Predetermined Change Control Plan, was built for modifications a sponsor can describe in advance and may not fit model updates nobody can fully predict.

Some of those changes come from a vendor. The paper flags that sponsors building on third-party foundation models may lack visibility into how those models were trained, evaluated or updated. Its proposal is a voluntary Foundation Model Master File, in which a model developer could give the FDA confidential model information that device sponsors then reference with permission. The file would not clear the model for any use, and the sponsor would still owe the agency a safety case for the finished device.

Contract manufacturing already works this way in hardware. When Amprius moves battery production to a rented plant, it has to qualify the new line again. A device company building on someone else’s model carries the same exposure, and its vendor contract should cover it: version pinning, advance notice of updates, audit rights, and a plan for the day the vendor retires the version the company validated.

Dave Saunders is the founder of Base Reality Group and a Fractional CPO for product companies. He was a founder and operator at Galen Robotics, where the surgical-robotics platform earned FDA De Novo authorization in 2023, and he managed a 35-patent portfolio licensed from Johns Hopkins. He wrote Founders Who Finish and publishes The Build. More about Dave →