It also introduces a lot of fragility into the ecosystem. If your TV fails (which does happen sometimes), you're suddenly without access to almost all features of the hardware? Unacceptable.
Aside my guess is the Apple TV does usually work “headless” in OP use case with music playing controlled from his phone. One only needs a tv for streaming video (obviously) and I think for initial setup.