People used to make similar arguments about 35mm film. Now that that's firmly surpassed I guess 4x5 will do. 4x5 is only 16x the area of a 35mm sensor so it's not too far away, you will have to soon move on to 8x10 to make the same argument :)
Digital has beaten 135 film, but it still hasn't beaten 120 / medium format in several key areas. I would be surprised if digital beat 4x5 in the next 10 years because there's no substantial market for such a product.
Basically, DSLRs require perfect conditions in the field to actually achieve the sharpness stated by the camera manufacturer. A 4x5 sensor allows much more slop.
Nanoparticle films are coming out. Gigapixel resolution in 35mm format.
The laws of physics most certainly are THE fundamental thing in the way. Electron leakage, current ripples at nanoscale, capacitive interference. There's a lot more involved when it comes to semiconductor technology.