The catch is that the benefits of open vs non-open don't translate neatly from software to models. If software is binary-only, is it exceedingly difficult to change it in any kind of substantial way (you can change the machine code directly, of course, but the very nature of the format makes this very limited). OTOH with a large language model with open weights but without open training data - the closest equivalent to open source for software - you can still change its behavior very substantially with finetuning or remixing layers (from different models even!).