I was thinking about this after a discussion at work about large language models (LLMs) - the initial scrape of the internet before Chat GPT become publicly usable was probably the last truly high quality scrape of human-made content any model will get. The second Chat GPT went public, the data pool became tainted with people publishing information from it. Future language models will have increasingly large percentages of their data tainted by AI-generated content, skewing the results away from how humans actually write. To get actual human content, they may need to turn to transcriptions of audio recordings or phone calls for training, and even that wouldn’t be quite correct because people write differently than they speak.
I sort of wonder if eventually people will start being influenced in how they choose to write based on seeing this AI content. If teachers use AI-generated texts in school lessons, especially at lower levels, will that effect how kids end up writing and formatting their work? It’s weird to think about the wider implications of how this AI stuff will ultimately impact society.
What’s your predictions? Is there a future where AI can get a clean, human-made scrape? Are we doomed to start writing like AIs?
As for good human generated data for training and building AIs? It’s like wood from trees. We’ve gone through the “just cut down the nearest tree, it won’t matter, they’re everywhere” period. Soon we’ll enter a data farming period, just like with managed de-forestation, and with the value of task-specific data and LLMs now being obvious, we’re probably already there.
Hmmm … maybe that’s why big social are ratcheting up the prices for their APIs??!!
Honestly, it’s a little creepy how tangible is a Matrix like scenario, without the apocalyptic war part that is. Machines feeding of of our data and thinking (which was, IIRC, the original premise, not energy).
Future AIs might be trained manually over centuries by groups of experts.
I don’t believe this theory 100%, however it is true to some extent. At some point, ai language will plateau out and simply won’t get better. Once it’s at it’s max and has little to learn, they will be so human-like it won’t matter if it’s learning from itself. The percentage of influence would be so infinitesimal it practically won’t matter. At that point it wouldn’t be necessary to learn anymore, anyway.
We aren’t doomed to write like ai, different themes or stories require different nuances. It’s artistic. But it depends on the medium. Sure, resume’s, cover letters, memos, emails and whatever may become robotic (aren’t they already?) But creative stories won’t, to a great extent.
Slightly unrelated, but I was just talking with a friend about how we’re going to have similar issues with young artists trying to copy ai. As is, many young artists will turn to cartoons instead of real life when starting out. Their work is a bastardization of a bastardization, with serious flaws in anatomy, gravity, light, and depth. They go on to call those mistakes their “style” and point to other artists making those same mistakes to normalize them. Since “style” isn’t something they think they need to improve on, they may become good artists overall while having severe, glaring holes in their skillet that any professional can see. You can sometimes even tell when someone started out because “90s anime” or “10s cartoon network” made specific stylistic choices that changed over time.
So I think ai is going to cause similar problems. Newbies will copy what looks pretty to the untrained eye and learn an ai based style. Then when they become more popular they’ll be fed into ai as reference material and perpetuate the problem. Even worse is actual professionals may turn to ai instead of real life references or a desk mannequin. Then their skills may degrade because they rely too much on improper tools. (I’ve already seen this becoming an issue with photoshopped reference photos.)
Anyways, that’s my $0.02
It’s not just art, mass media means we live in the state of hyperreality -where we cannot differentiate between tour chosen representation (signs, symbols) of reality from reality itself-
Most of us have personally experienced far less than what we have consumed through media. Much of our understanding of reality is completely rooted in symbols that we have no grounding for understanding and contextualization.
My take is that "L"LMs are already old news. I think targeted or limited data-set language models are going to be the next wave.
I think this partly because very few people can do LLMs at the scale of Microsoft and Google so I think smaller firms and people in their garage are going to aim their sights on smaller targeted data sets with a eye towards factual accuracy.
And then maybe link them/daisy chain them together. I hope there is this unix philosophy for models where they do one thing well but you can ‘pipe’ data from one to another.
