They do that for easier discovery of existing contacts. Say someone switches from WA to Signal and they want to contact someone. How’d they know their username? Their contact list would be empty and they would uninstall it. Signal would only be for nerds.

so go use matrix and get all your friends on it I guess