I'm not an expert on MCMC for BNNs by any means, but even with small networks I think it's a little tricky to get right. If my memory serves me right, this paper[1] focuses on small networks and goes over the issues and how to get around them.

[1]: https://arxiv.org/abs/2402.01484