There are multi armed bandit algorithms for this. I don’t know the names of the public tools.

This is especially useful for something where the value of the choice is front loaded, like headlines.

We've used this Python package to do this: https://github.com/bayesianbandits/bayesianbandits