Have you tried with addons disabled? Your experience with E10s largely revolves around which addons you're using.

Some addons use compatibility shims with multi-process and this can significantly harm performance in some cases (even potentially making it worse than e10s disabled).

This site is useful regarding the addon compatibility: http://arewee10syet.com/