The difference between whitened data and encrypted data is that you can decode the whitened data if you have the correct receiver. Encryption uses a key so that you also need the correct key.
You may not have a receiver for say FT8 that I may be transmitting, but you're also transmitting in Morse code which I never learned. Nothing is stopping either of us from receiving the other's transmission if we want to.
I'm told there was been an explosion of digital modes and it is a real concern that if there are so many undocumented or hard to identify ones, it's effectively encryption. Perhaps a future rule change will address this.
Proprietary modes have never been illegal though. You can buy a receiver just like I can study morse code.