The Rayleigh criterion above is exactly what is used to answer the question you're asking :) it tells you what size objects you can resolve with a given wavelength of light, which if you think about it in a different manner also tells you what the smallest useful pixel size you can have is. Two smaller pixels next to one another would be able to distinguish no more information than a combined one.