I assume people didn't have mills at home for the same reason most people of today don't have full blown data centers at home. People did have mortars though like today's people have PCs and smartphones.
I think you're wrong to assume that of two things that co-evolved, one thing had to cause the other.