It is already a good thing, but it currently requires a lot of engineering effort to actually get it to work with acceptable quality. It's not something that works out of the box like half precision, or for some models, int8. And to your point, for many production scenarios the ratio of engineering work vs performance gains is maybe not worth it. But for models that are going to handle massive load in inference it is worth it in my experience.

I expect that this will be made much easier in future with better hardware support and smarter sparsification libraries.