Is it just because there's so much going on in every chunk, and the rendering engine has to load entire chunks all the way from the bedrock to the sky?
Check out Cuberite for comparison. It's a Minecraft server built from the ground up in C++ and it's waaaaaaaay faster than the official server. Advantages of a rewrite I suppose...
I think the end of the line for Java Edition will come at some point, but I also think they know perfectly well that they cannot do that prematurely, and we're very much still in "premature" territory.
Basically any part of the code that reads/writes chunks in the world or just references some kind of chunk-related data-type would have to be altered
https://www.minecraftforum.net/forums/mapping-and-modding-ja...
There might be issues caused by the raw number of chunk loads, I'm not sure there. But I can say that the majority of my lag issues when hosting minecraft can be squarely blamed on doing world generation in the main thread.
Quite fast.
You'll also want to look into making sure the save data is persisted to durable disk across reboots/power loss.
Of course you will probably want to incorporate something that copies the ramdisk contents to somewhere a little more permanent ever so often, set an interval you're comfortable with losing data within! (but given the huge improvement i'd say it is worth it!)