but it will have literally zero concept of what it’s talking about.

If LLM answers had zero concept they wouldn't work. We are here talking about it because the answers are correct up to a certain point that makes them good enough.

It’s like taking a Google search for medical information and adding a middleman who occasionally filters in junk data, then wrapping it in a nice package, giving the impression of authority and accuracy, making the process as frictionless as possible.

These middleman are also called teachers, when you go to medical school you have many of these, you don't get handled the medical encyclopedia and left on your own.

You might say “What’s the harm if they end up at the hospital either way”? But the danger is that they never go to the hospital at all because the chatbot told them to sun their perineum or inject bleach with the confident prose of a medical doctor and they just…

that's something you are making up, an LLM telling you to inject bleach would be the same as it telling you that 2+2= coconut, it's the wrong answer, if all the answers where wrong we wouldn't be here talking about this tech.

And who is to blame when someone is injured or killed as a result? Doctors have standards, and they take an oath to first do no harm. If they make a mistake, they are liable for it. How can you hold a machine intelligence accountable for inaccurate diagnoses?

Doctors prescribe you medicines and do surgery on you, an LLM can't do these things, if AI tells you to take a medicine you still have to go to the medic to get these.

The simple answer is you can’t, and so logically they should be strictly prohibited from masquerading as any kind of authority on medicine.

The propaganda meme is talking about a ban on AI answers about medicine. LLMs have no authority on medicine, that's something we will be discussing in 20 years.