Your account number is made up of two parts - one part lets you view and deposit, the other lets you withdraw. You can give the first part of your account number to anyone (if you don't mind seeing how much money you have), but you should keep the second part you keep very safe.
In fact, once people hear that the second part (the private key) allows you to withdraw, they don't even need to hear the rest.
For many people though, they don't like the idea that people who can deposit can also view amounts and transactions. I guess that's why cryptos like Monero are worth looking into.