Yup, this is the case for all neural nets.