0. Or a homelab, or a cheap 256MB VM, or a NAS that can run docker containers, or an old Chromebook: anything that can execute a Go binary, x64, arm6 or even mips
The local clones should evolve incrementally due to "git fetch", and then the restic or rclone task should figure out how to make incremental updates to the S3 content.
A periodic fetch into a persistent cloned repo will be incremental unless the upstream is doing something crazy with frequent branch deletions and repacks. In practice, most upstream repos I encounter behave relatively monotonically. They accumulate new commits and branch/tag heads but do not often create garbage or need repacking.
A periodic backup of the cloned repo will also be incremental if using an appropriate tool like restic or rclone-copy. Also, since the clone only changes during the fetch, you can serialize these in one periodic job and be confident that you are making a consistent snapshot of the repo.
The advantage of this approach is its simplicity. It is easy to reason about and easy to work with the backups to restore a repo without having to learn about other tools. It's the kind of thing I could feel comfortable setting up and running for years on end with little supervision.
A more sophisticated approach that integrates with git hooks, e.g. to do event-driven rather than periodic backup, is plausible but I think could quickly get in the way of itself. And if working with a hosted upstream, you would need to integrate with their proprietary hooks, e.g. GitHub actions, and deal with other restrictions of the hosting environment. Such a solution likely brings new failure modes and may not be a worthwhile tradeoff...
I don't understand where you are stuck, tbh.
yencabulator has provided a good tip I think, as you could store the previous set of refs and use that to build an incremental Git bundle (one with only the objects that were not in the previous bundle). I don't know if you can do that with the existing Git client though.
Very similar to yours, but also does forks, which I’m not sure yours does by a quick glance at your readme. Although, mine doesn’t automatically mirror from another forge, just clones everything locally. I’ll have to add a TODO to add mirroring.
Please have a look at how I handle them and consider adding a link to my project in your “similar tools” section, and I’ll do the same for you!
I just looked over your project, and it seems pretty cool. A little bit different than mine, since mine is specifically github to a gitea mirror repository. I don't mind adding a link to your project in mine.
Thanks for sharing!