It is, and you're misunderstanding some of the basics of supersonic flight.
"As speeds approach the speed of sound, the additional phenomenon of wave drag appears. This is a powerful form of drag that begins at transonic speeds (around Mach 0.88). Around Mach 1, the peak coefficient of drag is four times that of subsonic drag. Above the transonic range, the coefficient drops drastically again, although remains 20% higher by Mach 2.5 than at subsonic speeds. Supersonic aircraft must have considerably more power than subsonic aircraft require to overcome this wave drag, and although cruising performance above transonic speed is more efficient, it is still less efficient than flying subsonically."[1]
Factoring in that supersonic airplanes are significantly heavier, have lower L/D, must spend more fuel getting to higher altitudes, still have to fly subsonic a considerable amount of the flight time/path, etc. means that it's a matter of physics.