I've already had some success with Llava although existing image classifiers are still going to outperform it.