You should really try out Yocto. There is a lot to learn but you can configure it from super-minimal tiny distros to mega kitchen-sink-included distros.

Everything is customizable, every file, every compiler flag and option of every package, with easy overrides in your own layer which only 'appends' to the original instructions, or adding and removing packages as you see fit in the base install.