The only remote risk of extermination is if a company with absolute impunity and endless resources told an LLM to hack the nuclear weapons

Absolutely a huge risk.

and if knowledge of how to do that was already published online and processed as training data.

extremely naive to not understand existing capabilities of 100s of probings and planning based on probings, and multiple strategies, in a swarm working towards a goal.

Still to your first point, the organizations with the budget and impunity, seeking the most powerful tools, are then the ones who will use them for evil, at expense of others, power maximization.