Pytorch might be suboptimal but it's what's already there e.g. for Pyro. (PyMC3 uses Theano; can Theano drive JAX?)
OTOH I'm not sure most people know what tasks are GPU-worthy or not. I haven't the slightest idea of why MCMC/Variational Bayes is amenable to GPU speedups and Persistent Homology isn't.