i've never seen mp4 containing anything else than video so it's reasonable assume they would contain video most of the time and thus have a video player as the default app.
Peertube can create audio-only mp4 files by setting the video resolution to 0 [1]. I use this to create audio-only versions of lecture videos for use on the road etc - which is why I made this feature and submitted the PR.