Ill give you the term for it at the end because everyone hates the term. Its the same reason those people hate vegetarians, and gay men, and renewable energy, and small cars. Wearing a mask is consideration for others, its compassion for the vulnerable, its protecting yourself. Its weak, effeminate, unmanly. The whole of conservatism boils down to: toxic masculinity

A lot of them got sucked into the whole “the government is forcing it on you to control the population”, and they simply can’t comprehend that anyone would voluntarily wear what they now consider being the symbol of submission to the government. In their mind it doesn’t work and never worked and you’re just virtue signalling your support of the government. It’s wild and a lost cause.

I’d expect it to get much worse now.

Just tell them you have cancer and are immunocompromised. That usually shuts them up. Irrational people are not worth arguing with and sometimes this makes them think twice about harassing other people.

I live in Taiwan, people here wear masks for whatever reason and no one bats an eye. You can wear a mask if you are sick, have too much acne and want to hide it, or because you are a medical worker and need to protect yourself and patients like OP, etc.
It also helps that we all went through the SARS pandemic, which could be seen as the previous version of COVID. Unlike COVID, SARS transmits via respiratory droplets after symptoms like fever arise, so everyone learned to wear masks during that time, and it worked. When COVID came, we knew what to do.

Just say you are protecting them whether they like it or not