gpg is widely agreed to be a train wreck, but age is not generally suitable for encrypted backups because it doesn't offer any form of authentication (so an attacker can replace a backup with a malicious one).
If you sign the backups with some distinct key of the backup server, why wouldn't the attacker have access to those keys too (in the above scenario they already have access to the keys that the backup server is using for encryption).
I know there's an open issue in age for adding authentication[1], so there clearly is some threat this would protect against but I can't figure it out.