Just because Luddites were sympathetic doesn't change the fact that they were wrong. They didn't care for the welfare of workers in general so much as they were looking out for themselves. It did not matter to them that hundreds of thousands of peasants chose to move to find work in the cities because it was better than life in the countryside and by destroying machinery, Luddites were taking away their livelihoods while the factories were shut down.
It's usually at this time that someone will bring up the enclosure of common land as the reason why people were forced out of the countryside, but that doesn't change that factory life was still offered a better alternative for a lot of people. The privatization of fertile land was inevitable with a growing population. Without the new factory jobs, even more people would be stuck as serfs like they were in China or Russia at the time.