During industrial revolution, the clothes (and other fabrics) were getting dramatically cheaper. A family that could only afford cheapest clothes could now get a higher quality stuff. A family that could not afford any clothes at all, could now get cheap stuff.
This is what the luddites wanted to stop. It's not "luddites starving to death" vs "factory owner get no profit", it was "luddites starving to death" vs "many many more people can not afford clothes"