Nvidia's banking on TensorRT to expand its generative AI dominance | Hacker News Reader

Nvidia's banking on TensorRT to expand its generative AI dominance

theverge.com

19 pointswebmaven21 comments

TensorRT, like many other frameworks, has always been really cool in GenAI demos and essentially useless and unused because no one integrates those demos into the more fleshed out projects.

And TensorRT in particular is a pain for end-users to install correctly, especially when they gate parts of the stack behind sign-up walls.

That's what this article is about, they're doing the integration work.

An integration with a very popular tool is already available for use. I've personally tested it and did get a 2x speed up. I didn't have to sign-up for anything.

https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT

Ugh... I know everyone uses automatic, but I don't really like automatic. Is janky as heck, especially with SDXL, especially if you want to dig into the code and make some manual hacks.

Still, you are right, a native a1111 extensions is quite impressive and exactly the kind of thing Nvidia should pursue.

What do you use instead? confyUI ?

ComfyUI or fooocus handle SDXL way better than A1111 with 10gb VRAM or under. While I really like A1111, it just isnt usable with XL for me.

I run them on a 3090, and fooocus is still way better.

Fooocus MRE for SDXL. The quality is just so much better than a1111 or vlad's fork. And its stable, with a good UI:

https://github.com/MoonRide303/Fooocus-MRE/commits/moonride-...

I use VoltaML for SD 1.5: https://github.com/VoltaML/voltaML-fast-stable-diffusion/tre...

Next on my todo list is trying ComfyUI with AIT: https://github.com/FizzleDorf/ComfyUI-AIT

so many projects... its overwhelming

FYI I just learned that Fooocus-MRE has been depreciated, but many of the same options are now in the debug menu upstream.

...Yes, it is impossible to keep up, lol

diffusers also has example usage for TensorRT in the Stable Diffusion context: https://github.com/huggingface/diffusers/blob/85dccab7fd280c...

Seems those particular examples were submitted by NVIDIA employee too: https://github.com/huggingface/diffusers/commit/9d44e2fb6600...

Shouldn't be too hard to adopt to other similar pipelines if you've used diffusers before.

That's a good example of a "hit and run" demo.

None of the community Diffusers pipelines I know of have successfully integrated this. VoltaML made a valiant effort and gave up in favor of Facebooks AITemplate (for now).

Worked for me and was easy to integrate, but I use diffusers directly. Not sure what the others are doing, but if you can manage to implement your own pipeline with diffusers, you should be able to integrate the TensorRT pipeline with relative ease.

https://github.com/VoltaML/voltaML seems to have successfully integrated it as far as can tell, it says TensorRT is one of the compatible backends.

VoltaML abandoned it (but may revisit it) and replaced it with AIT.

The problem with a vanilla diffusers pipe is that you miss out on all the cool community augmentations that really make sd shine. Crazy img2img stuff aside, some random examples include:

- Fooocus's implementation preserves 'momentum' when switching from the sdxl base model to the refiner.

- freeu augmentation

- tomesd

- fooocus image style presets, with known styles from the dataset strait from SAI and others.

- auto prompt expansion

- performance hacks not in vanilla diffusers

I'm still don't understand. The opening of the article seems discussing about consumer end use cases. But then H100 comes to the discussion. So TensorRT SDK is targeted to the end user usecases integration or not?

That is TheVerge being TheVerge and largely missing the original post: https://blogs.nvidia.com/blog/2023/10/17/tensorrt-llm-window...

Did they also release new drivers for linux that carry these features?

I'm going to really not be happy if you have to use Winblows for these...

I don't think you have to use any new drivers to take advantage of it? At least the TensorRT example from diffusers run on my 535.113.01 drivers (Linux).

TensorRT was originally linux (or WSL) only. Windows support is one of the new revelations.

Its not like ROCM where official consumer gpu support is only on Windows.

> Its not like ROCM where official consumer gpu support is only on Windows.

I don't follow AMD's GPUs, but lolwat. Is this really their master plan to finally catch up to NVIDIA? Those antitrust investigators in France should be given this as Exhibit A for why the rest of the industry is so far behind NVIDIA, not NVIDIA misbehavior.

AMD has lots of blame for underbaking rocm, but the problem isnt that simple either.

Nvidia essentially shares their architecture across gaming GPUs and servers. Whatever work is done for A100s is "inherited" by the 3000 series and such.

AMD, on the other hand, split their architecture between desktop (rdna) and server (cdna).

These work completely fine on linux.

Reply on news.ycombinator.com