That could give you an starting point. But I believe the main issue is that fashion products have, by definition, short shelf life, so you can't run the algos on SKU data. Then you can use deep learning on product images + user categorical data to try to predict preferences, maybe simple binary classification?
I guess using images as input should give better features than textual description.