I think people made some crazy magical assumptions about AI based on scifi that doesn't apply to real life. Real things to consider and prepare for, but not likely.

Recursive exponential self improvement? Just because something knows how to code doesnt mean it can make the best ultimate next version of itself. Even physical evolution takes millions of years, and it creates mistakes and has setbacks.

We might be seeing logarithmic AI improvement today, like evolution hitting a hill that it can't cross. We might need trillions more gigabytes of clean training data that isn't LLM generated to hit the next level, and it might not even be worth it.

I think what the AI developers are doing by constantly promoting ai fear is linking the idea of exponential ai self improvement to it, because their biggest fear right now might be investors realizing that isn't real, and every dollar they invest is getting less and less back.