It’s not built by Nazis for Nazis.
Some opinions follow, because nothing is black/white.

I’m pretty sure the Devs are Communists and are very anti-nazi, anti-racist, anti-facist. They have controversial opinions, so I think some people think/say they are Nazis.
You can read more about their opinions here: https://github.com/dessalines/essays and draw your own conclusions.

Additionally, I’m fairly confident that they have separated those beliefs between lemmy.ml and lemmygrad, and I’m fairly confident they do a good job of moderating Lemmy so it doesn’t become lemmygrad2.
Some people who post thinking Lemmy is a bastion of free speech may get a shock when they get moderated. And they might equate such moderation to more extreme political beliefs or false equivalents.

As for the Lemmy software, it is Free Open Source Software. It’s built for everyone. Literally the most socialist/communist thing there can be - advocating for public ownership.
I think there is enough vested interests in Lemmy, with enough skills behind it, that - if the main software somehow became an issue (unlikely) - that a dominant fork would emerge and everyone could migrate (there would be a period of turmoil during the transition).

If you want my opinion:
You have nothing to worry about.
If you want a right/left/commy/socialist/safe-space/wholesome/techy/nazi community, there will be an instance that tailors towards it.
If you aren’t sure, you can pick an instance for now and get a feel of the fediverse, then create a new account on another instance when you find one that suits your sensibilities (I’m sure account migration is in the works).