Definitely a hard problem. I'd say spend more than 1-3 hours of time with the person. Definitely figure out the culture, or the culture you want to create. Culture fit is not just do I like this person (you need to at the very least respect them- if you don't like or respect them, it's hard to imagine things working out).

Culture fit is supposed to turn away qualified people that otherwise would work well. If you value culture, then it's worth turning away these skilled, fine people. Valuing culture is supposed to do that. Lastly, you shouldn't hire people that are a culture fit but unskilled to grow into or do the tasks necessary. They're a bad option.

What matters to your company? When other people are not in the room guiding them, do they decide things based on what's important to the company/mission/vision? That's part of culture.

New media that might help give some guidance:

http://startupclass.samaltman.com/courses/lec10/

http://startupclass.samaltman.com/courses/lec11/

Read the readings below these videos, especially this one: http://blog.samaltman.com/how-to-hire

Have more than one interview. Try hiring them on a short side-project and see how it goes. It's hard to hide your true self after 2 weeks or so. Ask yourself this: Would you trust them with the life of your company? Would you trust them and 10 of their friends you never met with the life of your company?

AirBnB (or some other company) has the culture interview by someone not in the same role- so an engineer never decides if a prospective engineer is a culture fit for AirBnB. This abstracts from judging fit based on their specific skils.