cross-posted from: https://lemmy.ca/post/64797356

During a provincial procurement process for AI medical systems that transcribe conversations between doctors and patients, government evaluators found serious errors in transcripts generated by 20 programs, she said. [...]

The evaluators ran a pair of test conversations between hypothetical doctors and patients through the AI systems to test their capabilities, and uncovered the problems, the report said.

Evaluators found that nine of the systems had so-called "hallucinations," fabricating information or making suggestions to patient treatment plans not made by the doctors. Those included referring a patient for therapy or ordering blood tests, the report said.

Transcripts created by 12 of the 20 programs contained incorrect information like capturing a different drug than was prescribed by the doctor. Seventeen of the 20 systems missed key details about patients’ mental health issues. [...]

[Ontario Auditor General Shelley Spence] said she personally saw AI scribe systems in use during a recent doctor’s visit.

“I actually went to my doctor because you can hear that my voice isn't exactly what it normally is … and they were using AI scribe,” she said. “So, I kind of mentioned, ‘Please look at the transcript when you're done.’”