I agree that there are pretty big advantages to start with maximal set of sensors, but that is true only when you are a company with the sole goal of trying to make autonomy stack with shit ton of money and a product that you can modify because you own it.

That's not true in the case of Tesla. They started as a EV company that offered autonomy later and since they were selling the product, they had to decide a minimal set of sensors that would, in theory, still work to keep the product cost reasonable.

Tesla does have a sizeable 3D ground truth; they collected tons of data mounting LiDAR on their test vehicles.