disasters that were unintended and didn’t need to happen.

As you say, every "scientifically based" industrial innovation / breakthrough has had the potential to cause new kinds of mayhem, and most of them did to some degree or another. Burning rivers and CO2 overload in the atmosphere spring to mind, but industrial "accidents" are part and parcel of industrial activity, always have been.

That is simply counter-factual. LLMs can absolutely do things that have not been published before, and they can access knowledge that is not in their training data through RAG. Intelligence is debatable, but reasoning models can reliably imitate "thinking" to a degree, and recursive self-improvement might lead to an autonomous advancement of intelligence. That LLMs allegedly "only regurgitate information" is simply wrong.

You're confusing morals, knowledge, intent, and intelligence with capability. An atomic bomb has no will of its own either.

Or if the military put an LLM in charge of nuclear weapons. But surely nobody would be stupid enough to do that!

The only remote risk of extermination is if a company with absolute impunity and endless resources told an LLM to hack the nuclear weapons

Absolutely a huge risk.

and if knowledge of how to do that was already published online and processed as training data.

extremely naive to not understand existing capabilities of 100s of probings and planning based on probings, and multiple strategies, in a swarm working towards a goal.

Still to your first point, the organizations with the budget and impunity, seeking the most powerful tools, are then the ones who will use them for evil, at expense of others, power maximization.