You can do more than pfsense with a plain old Linux box but it takes some dense reading to learn to manipulate the traffic control and routing tables yourself.
The reason why most open source router distributions with a UI are pretty basic is because everyone doing things sufficiently advanced knows how to manipulate the network stack directly and only needs command line