Firefox 36 will have issues, yes. Nightly, Aurora or Beta will do better. You don't need gstreamer for MSE/MP4 support. Just have set to true in about:config the following:

media.fragmented-mp4.ffmpeg.enabled media.fragmented-mp4.exposed media.mediasource.mp4.enabled media.mediasource.enabled

This will use the system install of library libavformat.

If you receive hangs on this configuration with FF 37 or higher, a bug report would be much appreciated. Or contact me directly if you don't want to do the bugzilla dance.