And that’s completely your right to do. However, that is not what the tolerance contract covers. It goes beyond what most people would tolerate normally. Also, people cannot both break the social contract, and then insist you hold up the other end.
By example, I’ve previously had long debates over nazi Germany and Hitler’s economic recovery. I would even tolerate Nazis, if they followed the social contract from their side. Unfortunately, the various Nazis groups regularly break that contract. They then try and hide behind it, when others take offence.
Conversely, I also disagree with the “tankies”. They tend not to break the social contract however. This gives them the right to reasonable tolerance of them, and their views. They respect others, despite disagreeing with them. They, in turn, gain a level of respect in discussions.
Don’t get me wrong, I am tolerant of a lot, from purely moralistic reasoning. The social contract is a larger entity however. It formalises what many of us feel. It also shows us where the lines are, beyond which people are abusing our tolerance. It’s the larger social version of our internal morals.
I don’t find social contract arguments all that convincing, but we can just pretend my social contract is “no violence or you get fucked” and ignore that. Tankies are way easier to talk to than Nazis, though I don’t really find myself talking to nazis often - just run of the mill bigots. Anyone with consistent standards or ethics is fairly easy to talk to, even if we disagree.
In my personal life I tend to take on more than half of the social costs in some friendships and I probably do the same when arguing with certain types of people. I’m more tolerant than I strictly need to be, but I feel like treating people like that is necessary for me.
The social contract concept is over-used by people who try to make it cover too much. It becomes a one-sided contract of adhesion which you’re assumed to have agreed to simply by existing. This, however, is simple reciprocation—it’s more like a truce than a contract. It would be unreasonable to expect tolerance from others while refusing to grant the same tolerance to them.
Of course there is no obligation to be intolerant just because the other person is; you are free to make a better choice.