- Stylo, the new CSS styling engine currently enabled for a percentage of Nightly users
- WebRender, a new GPU-based rendering engine that's currently in much earlier stages
- Quantum, an initiative to improve Firefox's real and perceived performance (there's a bunch of smaller initiatives that compose this, like Quantum Flow and Quantum DOM)
- Photon, a project to redesign and modernise the Firefox UI
Most (but not all) of these projects are aiming to ship with Firefox 57 later this year, but you can check them out now in Nightly. I've been using it for a few weeks and the difference is really night and day, huge props to all the Firefox contributors.