I think Android is a pretty good example of how "fracturing" is just FUD and not really that big of a deal.
No, you won't have literal one-to-one code, at least outside of WebVR. But I highly doubt porting is going to be that hard.
If anything, because you have to do the graphics on your own, there is an even greater chance of successfully building cross-platform systems. At the end of the day, it's all just GLSL.