There's a lot of complexity in gpg that's unnecessary for this usecase, I'd use age instead.
If you sign the backups with some distinct key of the backup server, why wouldn't the attacker have access to those keys too (in the above scenario they already have access to the keys that the backup server is using for encryption).
I know there's an open issue in age for adding authentication[1], so there clearly is some threat this would protect against but I can't figure it out.