Yep, you're within your rights to panic and flee.
And, yep, you may have overreacted. Companies subcontract work. That's what they do. Among other things, this is how firms train and audition new people -- by hiring them as subcontractors.
If you pay Company X for the work, you get work with Company X's name on it. If something goes wrong, you get to blame the whole company -- they don't get to dodge the blame by passing it on to their subcontractor. But, in return, you shouldn't expect the right to micromanage the internal affairs of the company.
Of course, it's also possible that you dodged a bullet. Who knows? If the contractor couldn't convince you to stay on as a client, using arguments like the ones above, that might be a sign that you weren't likely to get along well together. Client-contractor relationships are a tricky business. Often you are required to make gut-level decisions.