Yes I would, if it seemed to me that an awful lot of people who had studied carpentry were mediocre carpenters, while many who hadn't studied carpentry were excellent carpenters. I would start to question the usefulness of carpentry study.
I don't think we should accept that the study of philosophy improves critical thinking just because that is "one of the aims" of philosophy as a discipline. (One of the aims of homeopathic medicine is to cure disease, after all.) I certainly accept that you'd have a very hard time going through a philosophy program without ever thinking critically, but I'd say the same thing about a student in a comparative literature program.
I also think it's very hard to teach philosophy well, and that a poorly taught philosophy course can easily detract from critical thinking rather than bolster it. I took my share of philosophy courses as an undergrad, and a good half of them were glorified history classes with strong implications that certain viewpoints were "wrong" and others were "right." I suppose you could say those courses were not really philosophy courses if they were taught that way, but then I would say it's unfair to say that philosophy education improves critical thinking if you will only admit examples where it does so.