If Firefox is the only browser that implements asm.js, it'll have about the same issues as Dart or NaCL. That is, Dart also compiles to JS, but if Dart2Js were 6x slower on Firefox than Chrome, no one would be cheering, and in fact, this is a complaint Mozilla themselves raised in the beginning.
If I'm writing a game in C and compiling it to asm.js, and it's gonna run 6x slower everywhere else, I'm effectively developing it for Firefox only no differently than if I had compiled C to NaCL/PNaCL.
For this to be effective, it has to be cross browser. At least, Mozilla and WebKit/Chrome.