439
Authors Are Furious After Finding Their Works on List of Books Used To Train AI
(www.themarysue.com)
this post was submitted on 29 Sep 2023
439 points (93.5% liked)
Technology
59197 readers
4055 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I certainly hope that none of these authors have ever read a book before or have been inspired by something written by another author.
That would be a much better comparison if it was artificial intelligence, but these are just reinforcement learning models. They do not get inspired.
More to the point: they replicate patterns of words.
So do humans.
...like the naturally occuring neural networks are.
The brain does not work the way you think… (I work in the field, bio-informatics). What you call “neural networks” come from an early misunderstanding of how the brain stores information. It’s a LOT more complicated and frankly, barely understood.
Yeah, accurately simulating a single pyramidal neuron requires an eight-layer deep neural network:
https://www.cell.com/neuron/pdf/S0896-6273(21)00501-8.pdf
that was an interesting read, thank you
Yet you confidently state that the brain doesn't work the way LLMs do?
Obviously it doesn't work exactly the same way that LLMs do, if only because of the completely different substrates. But when you get to more nebulous concepts like "creativity" and "inspiration" it's not so clear.
The part where brain and neural net differ is in the learning via backpropagation, that seem to be done different in the brain, as there is no mechanism to go backwards through the network and jiggle the weights.
That aside, they seem to work very similar once they are trained, as the knowledge they are able to extract from data ends up being basically the same that a human would be able to extract. There is surprisingly little weirdness in AI and a surprising amount of human-like capabilities.
Tell you what, you get a landmark legal decision classifying LLM as people and then we'll talk.
Until then it's software being fed content in a way not permitted by its license i.e. the makers of that software committing copyright infringement.
What exactly was not permitted by the license? Reading?
Using it to (create a tool to) create derivatives of the work on a massive scale.
An AI model is not a derivative work. It does not contain the copyrighted expression, just information about the copyrighted expression.
Wikipedia: In copyright law, a derivative work is an expressive creation that includes major copyrightable elements of a first, previously created original work.
I think you may be off a bit on what a derivative work is. I don't see LLMs spouting out major copyrightable elements of books. They can give a summary sure, but Cliff Notes would like to have a word if you think that's copyright infringement.
Better tell that Google and their search index, book scanning project and knowledge graph.
That's an interesting take, I didn't know software could be inspired by other people's works. And here I thought software just did exactly as it's instructed to do. These are language models. They were given data to train those models. Did they pay for the data that they used to train for it, or did they scrub the internet and steal all these books along with everything everyone else has said?
Well, now you know; software can be inspired by other people's works. That's what AIs are instructed to do during their training phase.
These are machines, though, not human beings.
I guess I'd have to be an author to find out how I'd feel about it, to be fair.
Machines that aren't reproducing or distributing works
If an AI "reproduces" a work it was trained on it is a failure of an AI. Why would anyone want to spend millions of dollars and devote oodles of computing power to build something that just does what a simple copy/paste operation can accomplish?
When an AI spits out something that's too close to one of the original training set that's called "overfitting" and it is considered an error to be corrected. Most overfitting that's been detected has been a result of duplication in the training set - when you hammer an AI image generator in training with thousands of copies of the Mona Lisa it eventually goes "alright, I get it already, when you say 'Mona Lisa' you want that exact pattern!" And will try its best to replicate that pattern when you ask it to later. That's why training sets need to be de-duplicated.
AIs are meant to produce new things.
I don't think anyone is faulting the machines for this, just the people who instruct the machines to do it.
Did you write a comment on Reddit before 2015? If so, your copyrighted content was used without your permission to train today's LLMs, so you absolutely get to feel one way or another about it.
The idea that these authors were somehow the backbone of the models when any individual contribution was like spitting in the ocean and model weights would have considered 100 pages of Twilight fan fiction equivalent to 100 pages from Twilight is honestly one of the negative impacts of the extensive coverage these suits are getting.
Pretty much everyone who has ever written anything indexed online is a tiny part of today's LLMs.
Thank you for your reply.
On a completely separate note, it's funny to think that there exists Twilight fan fiction when ~~Twilight itself started as fan fiction work.~~
Edit: I dun goofed.
Pretty sure it's the other way around.
Fifty Shades of Gray started out as Twilight fanfiction before becoming its own thing.
AFAIK Twilight was always just its own pulp fiction.
Oh true! My memory was fuzzy on the details. Thanks for the correction.
But terminator said neural networks
Damn.
Are you saying the writers of these programs have read all these books, and were inspired by them so much they wrote millions of books? And all this software is doing is outputting the result of someone being inspired by other books?
Generally they probably bought the books they read though.
If George RR Martin torrented Tolkien, wouldn't he be infringing on the copyright no matter how he subsequently incorporated it into future output?
I completely agree that the training as infringement argument is ludicrous.
But OpenAI exposed themselves to IP infringement by sailing the high seas in how they obtained the works in the first place.
I hate that the world we live in is one where so much data is gated behind paywalls, but the law is what it is, and if the government was going to come down hard on Aaron Swartz for trying to bypass paywalls for massive amounts of written text, it's not exactly fair if there's a double standard for OpenAI doing the same thing in an even more closed fashion.
But yes, the degree of entitled focus on the premise of training an AI as equivalent of infringing is weird as heck to see from authors drawing quite clearly from earlier works in their own output.
I have to assume that openAI also paid for the books. if yes then i consider it the same as me reciting passages from memory or coming up with derivative text.
if no, then by all means, go after them and any model trainer for the cost of one book.
Asking an LLM to recite an entire novel isn't even vaguely a thing yet.
Well, here's straight from one of the suits against them:
I'm not even sure how they would have logistically gone about purchasing 294,000 books in bulk in digital form to be fed into training. Using the existing collections seems much more likely, but I suppose we'll see what turns up in litigation.
Also, the penalty for downloading copyrighted material if willful infringement is up to $250,000 per work. So it's quite a bit more than the cost of one book on the line...
God that Aaron/jstor thing makes me see red every time. Swartz was scraping jstor to publish it for the benefit of everyone, openai is doing it to make billions of dollars. Don't forget who the bad guys are (and donate to sci-hub)