You're probably experiencing the placebo effect.
If you care there's software that will let you double-blind test yourself to see what bitrate is transparent for you.
LAME above 165 should be transparent (i.e. sound the same as the source) for most people even with good equipment.