Can you specify what exactly your problem with that is? So far you've handwaved at "I don't trust the optimizer to produce good code", but loc is not inherently tied to how many user-facing abstraction layers there are. In fact, you'll find that there's maybe one additional layer of data abstraction in the actual std::vector itself. There's not much to cut through for the optimizer, and in my experience it has zero trouble doing so.
In terms of compilation speed, you pay the frontend cost for those 20 kloc exactly once if you use precompiled headers.
> should either be split into several much more specialized classes
Could you elaborate what "specialized" (or runtime-configurable) versions of std::vector you are thinking of? Just the fact that some people care about exception guarantees, and some people care about move construction, and some people care about custom allocators, and some people care about emplace semantics, and that not all of these are the same people, doesn't mean that it's inherently good to have separate classes for these aspects.