An Apple phone is a good bet. They last quite long, have a fantastic battery life (especially for your use cases), do everything you're asking for (including having SSH clients / terminal emulators) and some of them aren't too big.

For the most part you can definitely find what you want with an older-gen iphone, or the new SE.

Unfortunately mobile systems have now consolidated to only iOS and Android, and many android phones don't get regular updates / get much slower within 1 year / aren't supported long and even not expected to have a long life by their manufacturers. OTOH lots of people hold on to their iphones for a really long time because they just work, and for many years. Notice how many people still hold the original SE or the 7.