Most of the research on this comes from the field of educational psychology. The specific term that you would search is 'self-efficacy'.

http://en.wikipedia.org/wiki/Self-efficacy