I’ve had this problem too, on both mobile and desktop. If you made a manual entry to your Contacts list and added that address, you’ll need to delete the address there separately. That isn’t necessarily the cause though. I have places that stubbornly remain starred.