I don’t find social contract arguments all that convincing, but we can just pretend my social contract is “no violence or you get fucked” and ignore that. Tankies are way easier to talk to than Nazis, though I don’t really find myself talking to nazis often - just run of the mill bigots. Anyone with consistent standards or ethics is fairly easy to talk to, even if we disagree.
In my personal life I tend to take on more than half of the social costs in some friendships and I probably do the same when arguing with certain types of people. I’m more tolerant than I strictly need to be, but I feel like treating people like that is necessary for me.
The social contract concept is over-used by people who try to make it cover too much. It becomes a one-sided contract of adhesion which you’re assumed to have agreed to simply by existing. This, however, is simple reciprocation—it’s more like a truce than a contract. It would be unreasonable to expect tolerance from others while refusing to grant the same tolerance to them.
Of course there is no obligation to be intolerant just because the other person is; you are free to make a better choice.