It's a deep learning model..hence sometimes illogical as well :). Thanks for trying...it will learn and improve over time...
I think the later makes more sense. Some kind of input from the users like "I actually liked the website visually" when the algorithm gives a low score and vice-versa would help the model be more accurate?