Honestly one of the coolest things you can do for this sort of problem offer a version of the Talos Image Factory, where people can select the combination of plugins and the architecture and get a build.

There can be a lot of permutations to create of course but each one only needs to be built once per version you release.

To limit permutations you can always create pre-selected bundles of related plugins (like package groups on Linux distros) and let people select those bundles. They can always fall back to building it themselves if they really need to be picky about the plugin selection.