Of course, you may still want to tweak configuration, and then you have to "eject", which means that all configuration files are dumped in your directory for you to edit, and you basically don't use create_react_app anymore. You now have to get webpack and babel configuration right to make your changes, but you have a solid basis.
Nowadays, I try to never eject, and see create_react_app as a lib providing everything I need. It sometimes prevents me to try new staged js features, but on the other hand I often have new features on update that I didn't bother enough about to eject in previous version.
Such wrappers around build system are probably the way to go. I would love to see something like create_react_app but allowing to provide buildpacks, so we can find in the wild the perfect configuration for our taste.