The rhetoric around the means of production was initially in reference to slaves. Communism was an offshoot of the abolition movement not in reference to at the time minimal number of factories. And in that context people did abandon slavery.
Even today factories aren’t the means of production, they don’t make houses that’s mostly people with hand tools. Nor do they make food or mine resources etc.