Sure, but all analogies are flawed. Yours is not as offensive to me as you might have hoped because I’m agnostic. But I can also flip it on its head:

The problem with religion comes when it’s used as a justification for horrible acts—just like one might use the lack of free will to justify horrible acts.