An Autonomous Eye AI Gave No Result for Over a Third of Patients in Germany. Its US Trial Allowed Retries and Drops.

Researchers at a diabetes hospital in Karlsburg, Germany, told the staff taking retinal photos that retries weren’t allowed. Nobody got dilating drops either. Then they ran IDx-DR, the first autonomous AI diagnostic the FDA authorized, on 875 patients, and it gave a result for 63.4% of them. In the US trial behind that authorization, it gave a result for 96.1%.

A MedCity News column published today cites the German study as a problem of change control, the managing of updates after a device ships. The study also supports an earlier job: writing down, before a pivotal trial, what happens to a patient when the system gives no result.

Where the missing patients went

Examiners couldn’t capture an image for 10.5% of the patients, and the software rejected another 26.1% as unanalyzable. For patients who got a result, IDx-DR did well on severe retinopathy: it caught 94.4% of cases, and it correctly cleared 90.5% of patients without severe disease. Count the patients with no result, and all of those accuracy figures fell.

The patients with no result skewed older. Those who got a result averaged 46 years old. Those the software rejected averaged 61, and those with no image averaged 64. In a subgroup of 141 whose pupils were measured, people with pupils under 3 millimeters got a result just 32% of the time. The authors tested age and pupil size separately, and pupils shrink with age, so I wouldn’t pin the gap on either one.

What the 96% included

The US trial got to 96.1% with help built into the job. Ten primary care offices trained staff with no eye imaging experience for four hours. The software checked each set of photos in seconds and told the operator what to fix. More than a third of participants (35.3%) needed at least one retry, and 23.6% needed dilating drops to reach a result.

Karlsburg removed the retries and the drops on purpose. The authors wanted to mimic a busy office, and they avoided dilation because a screening test should be able to run with no ophthalmologist on hand. They also studied a different group: inpatients at a specialty center, ages 8 to 92, some already treated for eye disease, where the trial took adults with no retinopathy diagnosis. Some of their photos came from a Zeiss camera, and the FDA’s 2018 summary limits the device to one Topcon model. So the two figures aren’t a head-to-head test, and the authors say retakes alone would have raised their rate.

The FDA wrote the job into the device type

The FDA had already tied the device to its operators in 2018. When it created this device type through De Novo, it set special controls, the conditions every device of the type must meet. Those controls require a training program on capturing quality images, human factors testing of what that training does for users, clinical testing under anticipated conditions of use, and data on how results vary with the user and the camera.

The same summary says what to do when there’s no result. The patient should be retested immediately or referred to an eye care provider. A patient who has been dilated and still gets no result may have vision-threatening retinopathy or another abnormality such as cataract. So no result comes with clinical instructions, and someone at the site has to follow them. The autonomy covers the read, and a trained person still takes the picture.

Who’s at the camera six months later

One examiner in the German data, a medical student, shows how fast picture-taking skill comes and goes. The student’s share of rejected images fell from 46.9% on the first day to zero by day four of training. After 196 days away during COVID, it was back at 18.8%. That’s one examiner, so I wouldn’t generalize from it. Still, the US trial trained its operators once, and a commercial site will outlast whoever went to that training.

So before the pivotal protocol is written, put the no-result path on paper: who’s at the camera after your trained operator has left, and what happens to the patient next. That includes whether your sites will use drops and who gives them, how many minutes a retry costs in the customer’s busiest clinic, and whether the customer gets paid for a test that gave no result.

Then pilot in that clinic and report results across everyone who walked in, no-results included. A wound photo or a weld inspection shot needs the same plan, since your customer’s staff takes that picture too.

Dave Saunders is the founder of Base Reality Group and a Fractional CPO for product companies. He was a founder and operator at Galen Robotics, where the surgical-robotics platform earned FDA De Novo authorization in 2023, and he managed a 35-patent portfolio licensed from Johns Hopkins. He wrote Founders Who Finish and publishes The Build. More about Dave →