I have similar problems in Python with OpenCV. It should be so simple, but generally is not (or at least last I tried)
To be honest, it's always this dice roll with most things on pypi. If there are wheels for you, golden. If not, you better hope it's a pure python package or it doesn't pull any outside libs. Things are getting better though with solutions like cibuildwheel.