Hacker News Reader

DistributedDataParallel (potentially) does both model and data parallelism. Data parallelism is also absolutely used when training large models, it has its downsides, but I don't think there's any way around it if you're training with a large amount of gpus.

View on HN

How does DDP do model parallel?

I phrased that wrongly, DDP itself doesn't of course. I meant that using it in the way GP does is also doing model parallelism.

I don’t see any mention of model parallel in GP post. How could you possibly use DDP to enable it?

GP contrasts DP and DDP by saying that DP is "where you clone your model over each GPU" and DDP is "'proper' multi-GPU training - you can now train big models and put a little bit of data on each GPU". That's simply not what DP or DDP is. What could this possibly mean if it's not misunderstanding DP as data parallelism and DDP as model parallelism? I'm fairly certain that what they're describing is using DDP (which only does data parallelism) in addition to model parallelism.

Reply on news.ycombinator.com