I think you’re very right about a misunderstanding of what upvotes and downvotes do on lemmy, the system that serves posts to users is very simple in lemmy, and some semblance of an algorithm would help a lot.
I also think the downvote issue wasn’t as big of an issue on Reddit because they’ve always fudged their numbers post scores, even moreso when the platform was smaller. They would intentionally make the number of votes fluctuate to prevent downvotes brigading and they would even limit how many votes could be applied to a post at one time. That’s why the top posts of subreddits used to stay the same for literal years, if that many peoples votes made it through, that meant the post got REALLY popular. Maybe that’s something else we could look into.