First you have an association of anything bad with excrements. This is cross-linguistically fairly common, and really old*.

From that “shit = bad” meaning, you got semantic amelioration generating the “the shit = the best”. English slang does this fairly often; refer to “sick”, “dope”, “wicked” doing the same. I’m not sure but I think that the underlying process is:

• “shit” as “extremely bad” →
• “shit” as “notably, outstandingly bad” →
• “shit” as “notable, outstanding” →
• “shit” as “noteworthy, good”

That also explains why “it ain’t shit” is generally negative - it conveys “it isn’t noteworthy”.

*It’s so old that one of Martial’s Epigrams (liber III, epigram 17), in 1st century Latin, already shows this:

Circumlata diu mensis scribilita secundis urebat nimio saeva calore manus; sed magis ardebat Sabidi gula: protinus ergo sufflavit buccis terque quaterque suis. illa quidem tepuit digitosque admittere visa est, sed nemo potuit tangere: merda fuit.

A tart [scribilita], passed and passed around at dessert, cruelly burnt our hands with its excessive heat. But Sabidius’ greed was more fiery still; so forthwith he blew on it with his cheeks three or four times. The tart cooled to be sure, and seemed ready to admit our fingers, but nobody could touch it. It was filth.

I’m copypasting the translation out of laziness, but… it is not accurate. “Merda” is not just filth, it’s literally “shit” - and it’s metaphoric as you’d use in English “that cheesecake was shit”, same shit here.

This won’t help, but it’s fun: https://youtu.be/igh9iO5BxBo?si=oyLAwhvrhsR4FHSZ

Bad words just become extreme. The modify the emotion. They often don’t mean anything literally. You have to look for other clues or just memorize the terms.

Words don’t all have meaning in isolation. :) The examples you give just have meaning based on the combination of words.

A bit like letters. We don’t think “r” is contradictory because in “fart” it describes something foul and it “tart” it describes somerthing tasty.

TL;DR: swear words like “shit” usually emphasize the mood of a sentence rather than add new meaning, which is why “shit” seems to change connotation across your examples. Think like the word “very”.

Traditionally in most European languages, the cycle of what is considered most offensive shifts between bodily functions (shit, piss), genitalia and sexual acts (cunt, cock, fuck) and religious profanity (hell, bloody), particularly against the Christian God. Some scholars define us as moving into a new cycle, where the most offensive words are slurs based on race, sexuality, or gender. These scholars speculate that this results in more willingness to experiment with already existing swear words of the ‘traditional’ categories since they are considered less offensive in comparison.

Swear words are almost always used euphemistically and in set phrases. Some scholars go as far as to argue that swearing is only euphemistic and words used literally do not count as “swearing”. In fact, much of swear word usage can be classified as an intensifier, which is a word or short phrase that sort of heightens the already existing mood of the sentence but doesn’t explicitly change the meaning. Using shit as an example, “Shit, the bread’s gone stale again.” In this example, you can also see shit being used as a sort of mood marker, since it is reasonable to assume out of context that the bread going stale might be desirable to the speaker. In this case, the “shit” marks the stale bread is actually bad as well as intensifying the mood as compared to, say “oh darn, the bread’s gone stale again.”

This part is a bit of speculation on the origin of set phrases like “ain’t shit” or “the shit” and I haven’t actually read any scholarly literature on this topic specifically. You can see similarities between set phrases like “this is the shit” and “this is the stuff” and “that’s the spot”. It seems like this is a construction common in the English language to express that something is pleasurable. Whether this is what caused the “this is the shit” set phrase or whether the “this is the shit” set phrase caused the construction remains to be seen. “Ain’t shit” is definitely somewhat different because it probably comes from African American Vernacular English (AAVE). While I would argue that “the shit” could be AAVE in origin as well, I do think it has roots in non AAVE English, whereas “ain’t shit” is grammatically AAVE.

If you’re interested in reading more, I recommend The F Word by Jesse Sheidlower and Holy Sh*t by Melissa Mohr.