Have you looked into any of the Jolla devices?
You can get a Sony Experia phone with Linux SailfishOS preinstalled on it for less than $500:
https://buy.jolla-devices.com/
Or even installing CalyxOS? You can get a Pixel 5a 5G phone (selling on ebay for under $200) and then install CalyxOS on it.
I'm going to grab one of the Jolla devices and give it a test drive next month. I think there's decent alternative out there where you can still maintain your privacy without having to step all the way down to a "dumb phone".