1. Caltrain is dog slow, infrequent and doesn't tend to go to places you want. Santa Clara County actually runs what seems to be a fairly decent VTA light rail that will get you from, say, downtown Mountain View to tech companies east of Moffett Airfield (Lockheed Martin, Yahoo, some of Microsoft's buildings, etc) but sadly not to the Google campus (west of Moffett Airfield).
To take public transport from SF would require you going to South SF (say half an hour), 1h 20m to MTV and 15-20 minutes on a shuttle if one is running. It's 2 hours best case scenario or one hour (or slightly less) on the 101 on the shuttle busses.
2. Zoning is a disaster for commerce in the valley. Most land is reserved for ("zoned") single family residences on large blocks of land such that density is very low. Public transit just doesn't work at low population density;
3. New development is virtually nonexistent in SF. Rent control is, at this point, an unmitigated disaster to a city that's still clinging to some 60s idyll.
Personally I think NYC is far more attractive than the Bay Area in all ways but two: the weather and the inertia of existing tech companies. You can live 5 minutes walk from work in NYC. There are very few places you could do that (or would want to) in SF.