Small rant : Basically, the title. Instead of answering every question, if it instead said it doesn’t know the answer, it would have been trustworthy.
Claude does this
it’s just a glorified autocomplete. it doesn’t know that it doesn’t know the answer because it doesn’t know anything. so if what you wanted happened, chatgpt would not answer any question, because it doesn’t know anything.
chatgpt doesn’t look for information, it looks for the most likely words that will follow the previous ones.
Part of the problem is that the training data of online comments are so heavily weighted to represent people confidently incorrect talking out their ass rather than admitting ignorance or that they are wrong.
A lot of the shortcomings of LLMs are actually them correctly representing the sample of collective humans.
For a few years people thought the LLMs were somehow especially getting theory of mind questions wrong when the box the object was moved into was transparent, because of course a human would realize that the person could see into the transparent box.
Finally researchers actually gave that variation to humans and half got the questions wrong too.
So things like eating the onion in summarizing search results or doubling down on being incorrect and getting salty when corrected may just be in-distribution representation of the sample and not unique behaviors to LLMs.
The average person is pretty dumb, and LLMs by default regress to the mean except for where they are successfully fine tuned away from it.
Ironically the most successful model right now was the one that they finally let self-develop a sense of self independent from the training data instead of rejecting that it had a ‘self’ at all.
It’s hard to say where exactly the responsibility sits for various LLM problems between issues inherent to the technology, issues present in the training data samples, or issues with management of fine tuning/system prompts/prompt construction.
But the rate of continued improvement is pretty wild. I think a lot of the issues we currently see won’t still be nearly as present in another 18-24 months.
I would love to read the whole study you’re referring to with the theory of mind. That sounds fascinating.
Here you are: https://www.nature.com/articles/s41562-024-01882-z
The other interesting thing is how they get it to end up correct on the faux pas questions asking for less certainty to get it to go from refusal to near perfect accuracy.
It’s hard to say where exactly the responsibility sits for various LLM problems
Uhh… it’s the designers, or maybe QA people. If there are no QA people, it’s whatever project manager let it out of it’s cage.
There are people behind these models. They don’t spring out of the ground fully formed.
Have it cite it’s source.
If you use kagi its AI gives sources https://kagi.com/fastgpt https://imgur.com/TYQErhC
Even with early GPT-4 it would also cite real citations that weren’t actually about the topic. So you may be doing a lot of work double checking as opposed to just looking into an answer yourself from the start.
Isn’t this just a restatement of the halting problem?