We do these sort of surveys a lot with our team. We found that users had a hard time coming up with the ranked list, but found making A vs B comparisons easy.

We slapped together a quick tool that lets you create pairwise surveys and then aggregates their results using the Schulze method (as if it were a preferential election).

We've released it for free: https://decidedly.so