Still trying to understand the ToR case. They're pretty basic with enough ports, BGP, and mgmt agents to do just what you need. That said, these things are _stupid_ cheap now. The Quanta switches that AWS used/use couldn't have cost more than $200/each direct from ODM. Maybe it makes sense at scale if you only need 38 ports in a rack to build your own so that you save on power costs from cutting out the extra 16 ports or so? Improving PUE and opex might be the big driver here?
I mean, these ToR switches really are super cheap if you buy direct from ODM. For designing your own ToR, you'll need at least a team of 5-6 people each making $400-500k annually at a company like Google or FB. How much is the cost savings per ToR switch in capex? $10/switch? In opex? I think it might make opex sense.
You'll build and run your own mgmt plane OS for sure, but that's tangential to building your own hardware.