On one hand you avoid frustration, on the other hand you miss out on greater enjoyment.
If you haven't gotten into high-end audio already, I'm guessing you're not the type who "listens to equipment" as opposed to listening to music. So long as you're paying more attention to the music than the minute differences between exponentially more expensive pairs of headphones (or DACs and amps), you're probably not going to have the problem you're afraid of. I can rarely hear the difference between my midrange Sony wireless headphones and my friend's super fancy open-back headphones when there's an equalizer profile applied to each to make them sound similar.