isn't an AI replacing artists evidence it has an experience

I can only speak about the literary world, and I was quite sanguine about ChatGPT in the early days, before I learned about how LLMs actually work. Having experimented with these tools extensively, I am certain that not a single page of good fiction has ever been produced by these statistical models. Their banality is almost uncanny — unless you know how they work, in which case it makes sense.

Now to be fair, fewer than 1 in 100 people can write fiction well, and fewer than 1 in 10,000 can do it at a level I’d consider “art” (as opposed to amateur dabbling).

LLMs are limited by the mathematics of their design. They’re just tracking weighted averages about what word comes next. That’s why they’re so good at corpospeak and technical writing, and so utterly worthless and cringey at writing fiction (or “art”).

If a collection of cells can be creative, then an extremely large mathematical system embodied in a GPU could also, potentially, be creative.

Sure. And a hundred monkeys with typewriters could reproduce the works of Shakespeare. Like you said, the issue is how to do it consistently and not in an infinite sea of garbage, which is what would happen if you increase stochasticity in service of originality. It’s a design limitation.

I have no idea what it's "like" to be an LLM

The same thing that it’s “like” to be a fax machine. They’re not significantly different, and you can literally program an LLM inside a fax machine if you wanted to.

Anyway, leaving you with the thought that you can’t compare “a collection of cells” to digital computers for two reasons.

1. Cellular activity is the domain of biologists, who do not study creativity or art. We have absolutely no idea how the tiny analog machinery of multicellular organisms give rise to consciousness.

2. Comparing digital stuff to analog stuff is a category error.

• “If a collection of cells can be creative, why not a mathematical system in a GPU?”

• “If a collection of cells can be creative, why not cheeseburgers?”

In both cases the answer is potato.