I was thinking about this after a discussion at work about large language models (LLMs) - the initial scrape of the internet before Chat GPT become publicly usable was probably the last truly high quality scrape of human-made content any model will get. The second Chat GPT went public, the data pool became tainted with people publishing information from it. Future language models will have increasingly large percentages of their data tainted by AI-generated content, skewing the results away from how humans actually write. To get actual human content, they may need to turn to transcriptions of audio recordings or phone calls for training, and even that wouldn’t be quite correct because people write differently than they speak.
I sort of wonder if eventually people will start being influenced in how they choose to write based on seeing this AI content. If teachers use AI-generated texts in school lessons, especially at lower levels, will that effect how kids end up writing and formatting their work? It’s weird to think about the wider implications of how this AI stuff will ultimately impact society.
What’s your predictions? Is there a future where AI can get a clean, human-made scrape? Are we doomed to start writing like AIs?
thanks, it’s terrifying
This sounds like what an ai would write /s
I think that while LLMs are going to get worse, the AI software will get better to the point of strong AI, and it will do a lot of “apple-esque” changes to mass produced speech that will ultimately be for the better… The cynical possibility is that it will further taint human dialogue even though it could provide a better way.
I suspect the quality LLM development teams will pursue the same in-depth data sourcing & cleaning techniques that quality ML researchers are developing today. Or rather, they’ll do something similar in principle to mitigate this issue.
I still agree with your conclusions. It will be a bigger consideration and less scrupulous teams will be more effected.
I’m not sure this is true. They could be trained based on published works prior to a certain date as the formal writing style, eg Project Gutenberg, then layer on the recent internet to better capture modern stylistic trends.
Ultimately, the models will always require fine tuning, and selecting which data set you use for early training has a very large impact on the overall performance of the model. Additional knowledge and trendiness can be learned after the fact.
You have to remember one thing, writing or speaking of a language is not a fixed scientific law or math formula that will stay true through out history. A living language is always moving and evolving in most of its components, be a vocabulary, grammar, or even meaning of words/phrases. We are just entering an era where AI generated content someone might feel appealing and follow that style, compare to copy a contemporary popular writer.
Indeed. As long as the language is still expressive and we understand what is being communicated, I don’t see why it would matter if it “sounds like” AI or not.
If it really becomes a problem then just curate the training data better to exclude the stuff that “sounds like” an AI. Doesn’t matter if it’s actually written by an AI or not, just select the training data that matches what you’d like the AI to learn and go with that. There’s not some kind of magical ghost present in human-written words that’s absent in AI-written words, if the words are the words you want then that’s all that matters.
