Without going into whether this belief is well founded (personally I don't think it is), its a combination of two factors: an assumption that there is a pool of potential business owners already in the state who are deterred by higher taxes, and a desire to be like Delaware (or internationally, Ireland), where low taxes and favorable regulatory structure are such that many large, inter-state or multinational corporations headquarter themselves there.

The first point, as you suggested, is pretty questionable. As you pointed out, it's worth noting that the current wave of entrepreneurship has headquartered itself in California, New York, and Massachusetts, some of the highest tax areas in the country.