Hacker News Reader

It may be a semantic thing but I don't think bandwidth pooling works like that? Unless your blocks are distributed perfectly between cards there's going to be data passed between each card and from VRAM to compute. Correct me if I'm wrong though

View on HN

I think the way it works is that the weights have to get streamed through the compute and the weights can be split very easily. With multiple cards you do have a coordination/communication phase after each layer of weights but that's usually much less significant than the compute phase.

It depends on how you slice the weights. You can split along layers which means each GPU waits for the previous one to finish or you can split the tensors across gpus.

Splitting the tensors is closer to n times the effective vram bandwidth but there are still overhead, sync waits, and reduce steps that eat away at the overall performance.

Reply on news.ycombinator.com