How often do you think you would see them if you lived in the same city?
If you see them twice a year, for a few days straight, I think you are doing really well and I am sure they appreciate the effort you make.
Can you cut the difference and go camping once a year or every second year for a week or two as a family? So their kids and your kids get to know each other, and maybe your wife and theirs too.