I'm surprised the 40mm f2.8 isn't more popular, it takes great pictures, it's cheap, and physically using it is great (it really transforms how the camera feels because it's so small). Strongly prefer it to the 50mm f1.8, unless it's super dark.
That said. It's a decent lens. If I could only find what gear pocket mine got shoved into :-)
However, I think one of the biggest reasons the 50mm gets recommended so much is because it is such a huge difference for someone to go from a f/3.5-5.6 kit lens to an f/1.4 or an f/1.8 in terms of DoF, and for beginners trying to move up this can be a huge eye opener.
Personally I think the 40mm is much easier to use, and it's certainly much sharper wide open.
Nikon has an APS-C format normal lens and that is the standard recommendation for that system. Pentax has a 35/2.4 lens and that is the standard recommendation for that system. Canon's lack of an affordable normal prime is a glaring weakness in the lineup and one of the reasons I eventually moved on to a NEX - you could get a 35mm and a 19mm prime for less than I could get a single 40/2.8 in the Canon system.
The 40/2.8 is the closest thing to a normal lens in Canon's lineup. But 40mm is still significantly longer than 35mm, and since Canon uses smaller 1.6x crop sensors instead of the industry-standard 1.5x crop sensors we are talking about going from a 50mm equivalent on other systems to a 64mm equivalent on Canon. That's still too long for a normal lens, it's not particularly fast at f/2.8, and it's more expensive than a Nikon 35/1.8 DX or a Pentax DA 35/2.4, but it's the least-bad option in the Canon stable.
My personal opinion is that Canon's lineup is heavily geared towards professional users, particularly full-frame users. They have nice L-series glass and their primes are well-placed for full frame users, but they just don't have many low-cost options for entry level users. There's no EF-S 12mm, 14mm 17mm, 22mm, or 32mm, all crucial focal lengths. Nor are there any cheap APS-C long primes. And their electronic aperture impairs the ability to use manual lenses like the excellent Samyang series, which could otherwise fill in some of these gaps.
While you bring up very good points I'm personally inclined to disagree on the "cost" argument when comparing 50mm f/1.8 and 40mm f/2.8, simply because by the time the 40mm pancake was released the 50mm f/1.8 went up in price from $80 to $125. Very shortly after it was released the price for 40mm went from $199 down to $149 for a while and at that price the difference between $125 and $149 is negligible. At this price point the 40mm is arguably of a higher "value" than the 50mm unless you really need f/1.8.
In the first sentence of your post you used present tense and I'd like to point out that the 50mm f/1.8 has been discontinued, so anyone continuing to recommend this lens is giving dated advice (not saying you are).
Currently the 3 cheapest Canon primes are the 50mm f/1.8 STM at $125, the EF-S 24mm f/2.8 STM at $149, and the 40mm f/2.8 at $199.
I currently recommend people the 40mm if asked about what to get for a full frame or the 24mm f/2.8 STM for APS-C. The 24mm on APS-C gives you a nice 38.4mm equivalent and I don't know why you've dismissed this fine lens (I hope that's no too harsh).
Interestingly Yongnuo released the YN 50mm f/1.8 for $50 that looks almost identical to the Canon 50mm f/1.8 II that's been discontinued.