Certainly something is lost by removing scarcity... at the same time, once that scarcity is gone, it can't be restored short of civilizational collapse. There's no way to put the streaming genie back in the bottle, nor the Stable Diffusion genie.
I don't see many people saying that we need to "un-release" anything.
What I do see is a want for ethical considerations for vulnerable parties be part of the discussion. I don't think consent is too large a barrier, especially since the music-focused variant of these tools currently has the developers walking on egg shells to avoid aggravating people with deeper pockets.
Instead what I see is the borderline criminalization of the people who were forced into the role of gatekeeping to be able to support themselves in the span of a few months because someone said "We can!" and apparently didn't watch Jurassic Park.
Napster was a peer-to-peer file sharing application. It originally launched on June 1, 1999, with an emphasis on digital audio file distribution. Audio songs shared on the service were typically encoded in the MP3 format. It was founded by Shawn Fanning, Sean Parker, and Hugo Sáez Contreras. As the software became popular, the company ran into legal difficulties over copyright infringement. It ceased operations in 2001 after losing a wave of lawsuits and filed for bankruptcy in June 2002.
Use of the output of systems like Copilot or Stable Diffusion becomes a violation of copyright. The weight tensors are illegal to possess, just like it's illegal to possess leaked Intel source code.
If you use the art in your product, on your website, etc., you risk legal action.
The companies that train these systems can't distribute them without risking legal action. So they won't do it. It's expensive to train these models.
It will always exist in the black-market underground, but the civilized world makes it illegal.
That's where this is going, I hope. Best case scenario.
This is obviously something that we're going to have to collectively figure out - the technology is here and the technology is still being developed. Either we adapt our thinking or consider it taboo. Anything else (such as restricting usage to a "trusted subset") is just delaying the inevitable.