It's one thing to talk casually, in which case I agree with you. But as soon as hard numbers are on the table, it's no longer casual, and if you do not provide sources then the assumption has to be that you pulled the numbers out of your ass and you are not to be trusted.
To get around that, just don't provide numbers and don't speak authoritatively. It's very easy, I don't know why people speak authoritatively if they know they can't back it up.