Some backends are fairly sophisticated and require extensive domain knowledge from designing databases that can be scaled, caching systems, security issues and so on.

On the client side you may need someone who is an expert in C#, in 3d rendering and geometry and in audio. Finding someone great at both is going to be very hard, and frankly not worth the benefits unless you have a very small team.

That said I believe front and backend should work closely together. There is no reason that the frontend should be blocked because the API design can be figured out before the backend is written and the frontend can mock out the missing features until they are available.