what is the regulation that mandates proprietary baseband?
While (AFAIK) there isn't a regulation stopping someone from selling radios that have completely free software basebands, you can bet that the manufacturer will be prosecuted if users suddenly start outputting radio waves that don't follow regulations (suing users is harder than suing a manufacturer). As a result, there's a disincentive for manufacturers to ever sell free software radios (because by definition they would have to allow modification).
[1] https://www.infoq.com/news/2015/07/FCC-Blocks-Open-Source