Or perhaps, "They are attracted to him because he is infamous and attractive"