Hacker News Reader

Comment by prashanthellina | Hacker News Reader

Thank you! I tried hard to keep the code very simple sacrificing some 20%-scenario requirements (such as automatic conflict resolution).

I am using this to synchronize my Markdown based notes files across machines. There are under a hundred files now so I haven't hit any issues in that department yet.

However, there is a gradual memory leak which persists even when I kill and restart my process. I observed this only on a KVM based Linux guest - not sure if this is because of inotify based listening but I'm going to have to dig deeper to find out.

View on HN

Forking might be easier to debug.. but I think I might have missed some of the functionality in your script.. something like (totally untested):

   (while true
        do
            inotifywait \
                --exclude '.*\.sw.' \
                --recursive  \
                --event close_write  \
                --event delete \
                --event moved_to \
                --event moved_from \
                   dir1/
             git push
             sleep 0.1
        done
    ) 2>&1 >/var/log/push_log.log &
   
   (while true
        do
          cat << 'EOF' | ssh remotebox
            inotifywait \
                --exclude '.*\.sw.' \
                --recursive  \
                --event close_write  \
                --event delete \
                --event moved_to \
                --event moved_from \
                   dir1/
             git push #??
             sleep 0.1

EOF

        # (second EOF cannot be indented)
        done
    ) 2>&1 >/var/log/pull_log.log &

    echo "Kicked off pull and push processes."
    wait

That is Gold! Implementing the functionality in shell itself was what I considered first however I needed two things

1. Make it work on OSX 2. Use a lock file to prevent multiple instances from running when I put this in crontab

I found watchman[0]. Got to see if I can use that in place of "inotifywait".

[0]: https://facebook.github.io/watchman/docs/install.html

Yeah that should work!

Nice clean code. A couple of places to start looking are probably in keeprunning(). Any file handle issues on repeated runs?

The threads in lines 245-247 feel like they could deadlock or have a race, esp if changes happen nearly simultaneously on both ends, or if a change on one end is received and sync'ed to the other. I'm not sure how git handles multiple writes? Almost wonder if forking might cure the memory leak too or at least make it easier to see (see my other comment of some shell code).

The memory "leak" persists across restarts of my process. That is what beats me. If memory were leaking while the process was alive and was reclaimed by the OS upon killing the process, then I could debug the process itself.

I read that running multiple git commands at the same time on the same repo should be fine (I can't remember where I read this). It is probably better to synchronize regardless!

Reply on news.ycombinator.com