This is precisely the point that is contention. The entire argument for prohibiting marijuana is that this is, in fact, not the case, and that, through a number of indirect channels, people "smoking weed in private" harms others throughout society in a variety of ways.
Obviously, as I said, there is considerable disagreement about whether this is, in fact, the case, (and, also, as to whether, even if it is, prohibition mitigates or exacerbates these harms, and as to whether, in any case, prohibition is an ethically-acceptable response even if the harms exist and prohibition mitigates them.)
But it certainly is not the case that those who support prohibition generally agree that the crimes are "victimless" that those opposed to prohibition describe that way.