Just a few of the realities that you will face if and when you do so:
1) A 12+ hour time difference. I recall nightly conference calls that started at 10pm. If you are somebody who values your time, you will find this untenable. Just use Slack or Basecamp to touch base? See #2:
2) Complete inconsistency in skill and communication ability. It was almost the norm to wake up in the morning and check for the changes we expected to be in place while the outsourced team worked overnight only to be shocked at the lack of progress. One question about a particular detail of the assigned task would bring the work to a screeching halt and we wouldn't find out about it until the next day. We would answer the question (usually inconsequential) and the progress would resume (far behind schedule). In many other cases, work was just done incorrectly and we had to spend the day course-correcting.
3) Lack of talent. After all, the workers who qualify for H1-B's are a self-selecting group who are, by definition, identified as the most talented employees. There is no guarantee that their counterparts who stayed home are able to do the job properly. In my experience most employees were sub-par, as one would expect when they are willing to work for pennies on the dollar compared to a U.S. citizen or somebody on an H1-B.
I think many of your points could be summarized as "we hired a bunch of idiots and they couldn't do their job very well". Timezones made some difference in that you weren't able to closely supervise them, but the fundamental problem was that your management shouldn't have outsourced the work to people not capable of doing the job.
With a small remote team you have the luxury of carefully choosing remote candidates and interviewing them individually. Enterprise-level work (read, bringing on teams of 10-50 developers in a short time frame) does not afford you that opportunity.
I'm not sure how large your company is, but it's pretty apparent you haven't worked with outsourced employees on an enterprise-level (7-8 figure) project before.
This is false. Many (at least 50% of H1B Indian workers) do not fall under this category.
To give you an example (in software): a company like Infosys would send you average or below average consultants who are good at communicating in English. And then Infosys would charge you higher than the premium developers located in the US, while paying the worker just about average (or even minimum pay by market standards). Over a longer period of time, there are quality issues, reworks, extensions and dependability created. Now, even if a client decided to leave Infosys and hire a premium developer - it is hard and costly. Who wants to take responsibility of a bad quality thing without being given time to improve quality.
The rest of the 50% who are talented, they get paid well, they contribute positively to the US economy and indirectly hurt the Indian economy.