I have noticed that the browser becomes more unresponsive with videos in tabs compared to single process Firefox. I enabled it to make my multi window, lots of tabs workflow smoother but it seems to have the opposite effect.
There was a known problem with video performance with e10s, though it was fixed two months ago (now Firefox 45 Dev Edition). The e10s sandbox inadvertently blocked hardware video decoding.