You know what, I don’t have a good answer to you here. I did a few small experiments on ChatGPT and it seems like it has some knowledge of if it will be able to complete it or not. This was with a pretty well known question though.

I tried to recreate an earlier experiment where I asked it to write about a friend of mine, which was in the news some time ago and have apparently a few entries in it’s training data, but very little. ChatGPT would then consistently hallucinate facts about the person, including date of birth and sometimes date of death. In that case it knew the pattern of writing about a person including date of birth, and sometimes date of death, but it didn’t know it didn’t have that info and just filled in plausible looking data there. Now it insists on not knowing who that person is at all and refuses to write anything about him.

Anyway, you’ve given me some things to think about, thanks.