The baseband could be exploited for targeted surveillance. First they would need to identify your device. If your device is anonymous and you carefully use your SIM for data only, identification is difficult.
Brave and patient folks are working on Linux handsets such as Librem and Pinephone that have fewer attack vectors. I hope to switch to one of those when my Pixel stops receiving updates.