That seems like a reasonable approach if you have the right skills for it. My problem with such an approach would be that I'd not feel comfortable to make an overall evaluation of a person based on a short interaction. Having a clear cut process with well defined metrics provides me with more confidence that I'm evaluating candidates fairly. Without it I'd be worried that my rating would be too biased by superficial stuff like how they dress etc. Something that at least in psychology research seem to mbe a lot more common than many people think. In general, looking too much at cultural fit would feel I'd risk "hiring my friends" rather than based on what value they could contribute.