Hacker News Reader

Comment by beagle3 | Hacker News Reader

This is a nice little project, but I think rsync is better suited for any serious use. It can do ssh, it's own protocol or any connection program that would tunnel bidirectionally. It does very efficient transfer, can create off-line patch files, and basically do any one-way file/dir sync function you can think of.

Edit: this one is continuous and rsync is one-shot. Personally, I've been using inotify+rsync+timeout (intofy+rsync has a race condition, so limiting the inotify wait and syncing anyway is essential). Does sshync avoid the race?

View on HN

Yes it combines also amazingly well with inotifywait[1], perhaps there is a port for other OS-es too.

Here is a tip on how it can be used ( not tested )

    #!/bin/bash

    WATCH_PATH="/some/path"
    DEST_PATH="/some/path"
    REMOTE="root@example.com"

    while read LINE; do
        FILE=$(echo ${LINE} | cut -d' ' -f2 )
        /usr/bin/rsync -r ${FILE} ${REMOTE}:${DEST_PATH}
    done < <(/usr/bin/inotifywait -mq -e CLOSE_WRITE --format "%w %f" ${WATCH_PATH})

1 : http://linux.die.net/man/1/inotifywait

For most uses, rsync -a (preserve all possible metadata, like cp -a) is preferable to rsync -r ;

Also, unless you have a very large (breadth or depth) tree, you can just let rsync synchronize the entire path - it is quite efficient, will pick up additional diretories (you need inotifywait -r for that, and AFAIK you still have an unavoidable race).

If you're considering replacing your current (assumedly homegrown) solution; I'd recommend using lsyncd instead.

You might also check out Syncany if you're interested in client-side encryption or Syncthing if you're looking for something more Bittorrent Sync like.

I use rsync. Question for you (or any expert here). Is there any possible way that rsync can alter the source file (unintentionally I mean, by accident or bug)?

And what happens to an rsync file that is modified while in the middle of a transfer? [1](I haven't been able to find satisfactory answers to these questions hence the question here).

[1] Say you have a 1,000,000 line file that you are rsynch-ing and you delete line 50000 and insert at line 750000?

Rsync opens source file in read only mode - so it has the same probability to modify a file that regular copying does. That said, the --remove-source-files option will delete source files after copying; --delete may remove destination files; and used together they drop the idempotence guarantee that rsync gives. So you have to be careful.

Re changing file - IIRC it will keep Resending until file size+mtime at end of transfer is same as in the beginning.

I use similar solution: my own shell script with inotify + timeout + unison over ssh. Unison is much better suited for two-way synchronization than rsync.

I love unison, but how do you use it with inotify?

Newer versions of Unison (since around 2012) can watch directories with inotify: http://blog.nguyenvq.com/blog/2011/07/18/real-time-file-sync...

For http://www.cis.upenn.edu/~bcpierce/unison/ your thoughts on why that is better than using rsync ?

Edit: I am now noticing this tool isn't even maintained anymore.

A new version has been released this year. unison works remarkably well, it's understandable there is not much to add.

From https://alliance.seas.upenn.edu/~bcpierce/wiki/index.php?n=M...

"Rsync is a mirroring tool; Unison is a synchronizer. That is, rsync needs to be told "this replica contains the true versions of all the files; please make the other replica look exactly the same." Unison is capable of recognizing updates in both replicas and deciding which way they should be propagated."

Larry - would you please email us at info@rsync.net ? We have a few items to discuss with you ...

(your profile mentions to respond to one of your comments if someone needs to reach you ...)

Thanks - I will but not just yet (a few reasons for that actually).

Don't know if this is still the case, but in the past, unison was extremely picky about the versions at both ends; I stopped using it when I upgraded my Ubuntu box and as a result I needed to either recompile an older version for the new box or upgrade the version on all other boxes (across windows and various distros).

I now use rsync for bidirectional mirroring (any two versions from the last 15 years or so can talk to each other), git for synchronization (getting history for the price of needing to add/commit). Can't beat the network efficiency and compatibility of these two.

Reply on news.ycombinator.com