Yes this is the option striking the best balance of function and security which I have been using for over three years. I have tried other custom operating systems and debloated carrier provided devices, but Graphene tops all the options IMO.
The baseband could be exploited for targeted surveillance. First they would need to identify your device. If your device is anonymous and you carefully use your SIM for data only, identification is difficult.
Brave and patient folks are working on Linux handsets such as Librem and Pinephone that have fewer attack vectors. I hope to switch to one of those when my Pixel stops receiving updates.