It's tricky, because most phones can't be flashed with an alternative aftermarket OS. I know that LineageOS etc. exist, but most phones don't have a stable build and a lot of SoCs will never work reliably with an aftermarket OS.

In that case whoever will exploit your phone first is the one who owns it, not you.

Also an interesting question in context of the regulation discussion: whoever sold you the phone is responsible for addressing significant product defects according to consumer protection rules in most (?) countries in EU. So it could become a problem if the irreplaceable phone batteries didn't die about the same time as the security updates end.

I also have an old etrex with AA batteries. It's a wonderful device. It supports OSM map data, so it's easy to update, the batteries last a week and you can buy replacements literally everywhere.

Find a battery of the same size or a little smaller and put it in. batteryjunction.com has a big selection. You might have to solder some wires on or similar. Adafruit.com has some with JST connectors which have a good chance of being what your GPS uses inside, but they don't have that wide a selection of cell sizes, so it's iffy. Anyway the first thing to do is open up your GPS and look at the cell. In the old days you could usually find a phone battery the right size on places like dx.com, but replacement phone batteries aren't that common any more.