Hacker News Reader

Comment by throwaway129305 | Hacker News Reader

[dead]

Shameless plug, this is why I built Mighty Inference Server [0]. It wraps ONNX Runtime in a production ready rust binary, and can be installed quickly and painlessly on most machines.

The sentence transformer models in the example notebook from the OP will infer queries in <10ms, which is fast enough for a production customer facing service…and lets you run on commodity instances without expensive GPUs

[0] https://max.io

Some of this might be difficult to set up but I don't think that's the reason why people don't use more pretrained models. If you have a working ML environment adding a pretrained model to it is trivial most of the time.

That's a big if. The initial set up is a pain.

And even once the initial set up is done, ML software is constantly changing/breaking. You're forced to reinstall a new proprietary Nvidia driver, which is then incompatible with the old version of PyTorch the model uses... But that other model requires a newer PyTorch... It's a pain... The fact that there's a dependency on GPU driver versions is ridiculous.

This should apparently improve, as at least CUDA no longer depends on specific versions (thank god). I was pleasantly surprised last time I installed it.

I think that's a good step but it's still a very brittle ecosystem IMO. Needs time to mature.

Completely agreed, but it's probably the Python part that's causing the problems, tbh.

Uhh, conda create a new environment first. Conda literally does everything else for you (including the actual hard part , getting your GPUs to play nice with CUDA and to be recognized by pytorch or tensorflow)

Just make a new conda environment, and throw it away if it gets clobbered.

But don't listen to me, I only work for a billion dollar company that does all of its cloud data science using conda, so I must not know what I'm talking about...

I think the point is that conda is great until suddenly it's not because of various compatibility issues. Especially when the errors you get will sound very cryptic to someone who's not used to ML and libraries that use CUDA.

That's at least my experience, I've personally always preferred to not use it at all (N=1).

Also, you sound very insecure, not sure what the company drop at the end is for.

Reply on news.ycombinator.com