No, it's not a fair suggestion, because it's not a real suggestion at all. Over 2000 skilled maintainers work on the kernel, there's hierarchy, processes, culture, community, the linux kernel project both adapts to and shapes computing globally, companies are in talks with the linux foundation in regards to drivers etc, distros are tightly coupled and tested on it, there are north-star people like Linus... It's very much a big ship.
So firstly, yes, there IS something "standing in the way of anyone forking a kernel and building their own version" - for example that it's absolutely useless, unless you can do the job of 2000 maintainers, your "fork" will be the upstream base + 3 little patches on top and you'll spend a month getting it stable on your system.
Secondly, this isn't something you solve by saying "screw it, I'll do it myself" and abandoning the ship. Even if you did build your own linux kernel to your liking, this doesn't address the concern about the global linux kernel project that other 99.9% of the planet and distros and hardware uses.
This "don't like it? Fork it" argumentation fallacy has been present ever since open source became a thing, and it's quite frankly insufferable, but applying it to the goddamn Linux kernel, that's a new level.