Simple steps to take before hitting the streets
With the right intel you could piece back some of the pieces, especially with some pieces from other sources, with just that metadata. With metadata, it’s about putting together lots of sources to see the picture clearly which is why Facebook bought WhatsApp for just the metadata (& address book). The thing is that you, can skip Signal & you will still have several free software messaging alternativ where nothing is on a US-based server where they can subpoena.
But that’d already entail control over the whole Signal AWS in- and egress as well as any VPN you may be using and/or your local ISP. And then you still have to prove the actual link to the natural person. At that point we’re speaking of a threat level assuming the US DoD as adversary. While not impossible, I think if you’re willing to pick that kind of fight, you’re clever enough not to rely on Signal (or most digital communication).
Signal is not WhatsApp, there aren’t a lot of data points linking your communications to end points in the same way Meta does link them.