the actions of this system cannot be predicted by anyone.

Reminds me of TempleOS where the author thought a random number generator sends messages from god. It was fun when it's just one person experiencing psychosis in a non-violent way. But the mass AI psychosis is the opposite of fun.

Nah I know exactly what that dude is referring to.

It's a property these LLMs have where a large complex network can sometimes do weird things. For example OpenAI had an issue where their model would just not stop talking about goblins. They were having trouble fixing it and figuring out why the model was doing that. They described how with a big network like that it isn't as simple as opening up the network and seeing the node where the offramp meme to goblinmode was stuck. And often it isn't even just one single thing, it's many different places which come together in a complex interaction to get to that end result.

They wrote some blogs and papers about how they tackled this, what the challenges were and some tools they made to help them figure it out. In the end they gave up and retrained the model to try and fix it. The model was simply too big, too much data to analyse and the interactions too complex to just flip a switch and fix it. The media then read that stuff and didn't understand it and ran with it. They wrote stories about how OpenAI has no idea how their mystic models work. These dumb dumbs are just creating super smart AI gods and they don't even have a single idea about how it works or how to fix it when it goes wrong.

This is obviously not true and very dumb, but that pretty much sums up the LLM based AI industry at the moment.

Also I don't really agree with your description in general. Just because the average person doesn't know how something works, most of us know somebody who does know. We understand that just because we only have a high level idea about how something works, we realize there are people who know exactly to the nitty gritty details of how stuff works. And with complex systems that might not be a single person, but a group of people. But we understand this is not fundamentally unknowable. So when somebody says "nobody knows", we would understand that to mean most people don't know but there are still plenty of people out there who do.

The exact same applies to these LLM systems. The general person might not understand these systems or only from a high level point of view, but there are people out there who do. I for one understand a lot about these LLM systems, at least the maths side of things. There's plenty of things I don't know, but like I said I know people who do know and it's not like it's unknowable, I could just go and find out.

These days it seems like people don't have functioning brains anymore. You know we can just go learn things right? There is so much information out there to know and available for learning. For LLMs there are so many articles, videos, papers, books and free open source tools. You can just go and play around, go read up on how these things work. You can go as deep as you like.

In the end an LLM is just a math function, a large and complex one, but still a math function. You put numbers in and numbers come out. Put in the same numbers and the same numbers come out. Although there is a part that can apply randomization using a parameter called temperature, but if you disable the randomization the exact same answer pops out for a given input.