And like a sibling comment said, the ADSP firmware is not being reverse engineered, and that doe a lot of the heavy lifting for cell tower connections.
In the original goal of the Librem 5, that is why they advertised an IP only phone tied to matrix. The idea is you don't need to connect to a mobile tower.
However, one avenue that may be interesting to explore is the use of WiFi calling as a means to bypass a lot of cell tower insecurity. In WiFi calling, somehow you connect all of your telephony/SMS/MMS through your WiFi (I assume it's through a VPN). However, when you sign up for it, you have to provide an address for e911 (presumably since they cannot figure out your location via cell towers). I would be curious to see if you can make a WiFi calling only phone, and in the process, you can completely turn off your modem.