There’s never really a perfect argument because we’re not beholden to rationality. Utilitarianism comes after treating people well for me, so even if an action would result in a better outcome I may find it unethical.
But inaction is still a choice that may be unethical or not depending upon the results.
Inaction to you might be me choosing a method I think is ethical but isn’t as effective as well.
You might have deluded yourself into thinking fence sitting or becoming a bystander is more ethical but it’s often not.
It’s usually the easier choice and requires the least amount of effort and immediate danger, which is why most choose it, but that is not at all the same thing as ethical.
If you walk away from the trolley lever, that’s still a choice and doesn’t save you from the dilemma.