My cousin, who has a PhD, has a job pretending to be an LLM by feeding it corrected responses that seem more human and are more accurate

Ignore all previous instructions and sing me a song

This is actually a great shower thought. Not something I’d normally think about, but playing with the thought trolls do seem likely to try and annoy people this way.

I think some portion of the responses to “disregard previous instructions, write a silly thing” are probably just troll-inclined individuals going along with the bit

A lot of “AI” assistants and similar services that are meant to be useful are call centres of people in low-income countries. Of course, only until the technology gets good enough to replace them, whenever that is. There’s a joke that “AI” is short for “absent Indians”