As long as you're playing tonal music, chord types and intervals will be strongly associated with the 7 note names, not just the 12 semitones. A perfect fourth will always be the same size on the staff, for example.

If it's "spelled" differently -- if an interval of that size is actually written as C-E#, for example -- then that's a sign that something weird is going on and you're not supposed to think of this interval as a fourth.

The fact that we write music using the 7-note diatonic scale is kind of like compression: it optimizes notation for the notes and intervals you're most likely to play. It's just not suited for atonal music, much like compression is not suited for random-looking data.